Making Sense of “PixelSense”: the Technology Behind Microsoft Surface 2.0

Making Sense of “PixelSense”: the Technology Behind Microsoft Surface 2.0

(Originally posted October 20, 2011)

So we were all totally “wowed” when we got our first peek at Surface 2.0 at CES 2011: …however, for so long, there hasn’t been much information published on how “PixelSense” works…(believe me, I tried, as I’m sure many of you have as well!)

Microsoft’s innovation is breaking all barriers lately; this excites me tremendously both from a developer perspective and from a “Softie” perspective. From Surface 2.0 to Microsoft Kinect to Windows Phone 7 to one of the most amazing breakthroughs for “Microsoft” – receiving 7 Awards at the “International Design Excellence Awards”! Tthank you, Bill Buxton, among others, for emphasizing UI/UX: “We Developers/Designers/Devigners” have done it: (link no longer active).

Getting back on topic, Microsoft Surface 2.0 has provided endless opportunities for developers to utilize their creativity to create futuristic applications, just has Kinect has provided us. What an exciting time to be a “Developer, Developer, Developer”!

Microsoft created the first version of Surface knowing the first release was only the beginning, a prototype if you will, to what was “really” envisioned for it to eventually become…

The visionaries include (but not limited to) Steven Bathiche – “StevieB” – whose Microsoft Research profile has the slogan “Thinker. Tinker. And so forth and so on” (I mean, how cool is that?), and Andy Wilson (see the YouTube link at the end of this article)…

So now, let me introduce an overview of Microsoft Surface 2.0’s ground-breaking “PixelSense” technology (using only a 4 inch display)!

Partnering with Samsung, Microsoft was able to create an LCD screen (Samsung SUR40 for Microsoft Surface) sense infrared input across the entire gorilla glass panel via incorporating transistors sensitive to the IR spectrum. Each RGB (red, green, blue) trio of 2 million pixels in a Surface 2.0 gorilla glass screen – you read that right – is joined by a fourth pixel that can “see” infrared light! It’s a “vision-based” interaction without the use of cameras! Talk about leveraging the power of the Windows 7 Operating System to the max! Think rather than using electrical conductivity to detect objects on the screen, “PixelSense” “sees”…(hear whatever theme movie song you want here…) This was proven by Microsoft placing a piece of paper onto the new Surface display, and Surface “read the writing” on the paper…ooohhhh!

This means that across the entire LCD panel, there are input and output pixels in the display. The pixels are laid out across the LCD, and when IR light is shot through the LCD, the “input pixels” passes information to the “sensor pixels” which objects are reflected. These sensed objects are passed to the software of the Surface machine and the software – written by you or me – are immediately processed and interpreted what these objects look like.

These “images” don’t do anything on their own…patterns are evaluated. Sixty (60) times a second, everything on the “surface of Surface” is passed to the hardware & everything is assigned a new tracking number. This allows the software that interacts with the images if “object 3”, say for instance a finger, has moved, or it’s a new “object 3” that it’s never been seen before, for every frame. It’s up to the software application made by us to decide what to do with those objects – as usual.

So let us Developers take this amazing Surface 2.0 “PixelSense” technology – just like we are doing with the Kinect hardware – and create (to quote a favorite developer/friend of mine) “very sexy, hot new software”!


  1. […] who cut a hole in a table and made Surface 1.0, for those of you who haven’t read my article “Making Sense of PixelSense: the Technology Behind Surface 2.0” , I’d suggest you do so 😉 ), Arduino/Netduino gives you a head start on ideas to take to the […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: