Wearable Depth-Sensing Projection System Makes Any Surface Capable of Multitouch Interaction

OmniTouch, a wearable projection system developed by researchers at Microsoft Research and Carnegie Mellon University, enables users to turn pads of paper, walls or even their own hands, arms and legs into graphical, interactive surfaces.

OmniTouch employs a depth-sensing camera, similar to the Microsoft Kinect, to track the user’s fingers on everyday surfaces. This allows users to control interactive applications by tapping or dragging their fingers, much as they would with touchscreens found on smartphones or tablet computers. The projector can superimpose keyboards, keypads and other controls onto any surface, automatically adjusting for the surface’s shape and orientation to minimize distortion of the projected images.

OmniTouch: Wearable Multitouch Interaction Everywhere (10 pages)

“It’s conceivable that anything you can do on today’s mobile devices, you will be able to do on your hand using OmniTouch,” said Chris Harrison, a Ph.D. student in Carnegie Mellon’s Human-Computer Interaction Institute. The palm of the hand could be used as a phone keypad, or as a tablet for jotting down brief notes. Maps projected onto a wall could be panned and zoomed with the same finger motions that work with a conventional multitouch screen.

The OmniTouch device includes a short-range depth camera and laser pico-projector and is mounted on a user’s shoulder. But Harrison said the device ultimately could be the size of a deck of cards, or even a matchbox, so that it could fit in a pocket, be easily wearable, or be integrated into future handheld devices.

“With OmniTouch, we wanted to capitalize on the tremendous surface area the real world provides,” said Benko, a researcher in Microsoft Research’s Adaptive Systems and Interaction group. “We see this work as an evolutionary step in a larger effort at Microsoft Research to investigate the unconventional use of touch and gesture in devices to extend our vision of ubiquitous computing even further. Being able to collaborate openly with academics and researchers like Chris on such work is critical to our organization’s ability to do great research — and to advancing the state of the art of computer user interfaces in general.”

Harrison previously worked with Microsoft Research to develop Skinput, a technology that used bioacoustic sensors to detect finger taps on a person’s hands or forearm. Skinput thus enabled users to control smartphones or other compact computing devices.

The optical sensing used in OmniTouch, by contrast, allows a wide range of interactions, similar to the capabilities of a computer mouse or touchscreen. It can track three-dimensional motion on the hand or other commonplace surfaces, and can sense whether fingers are “clicked” or hovering. What’s more, OmniTouch does not require calibration — users can simply wear the device and immediately use its features. No instrumentation of the environment is needed; only the wearable device is needed.

Chris Harrison’s Omnitouch project page

Different but similar gadget

Karl Willis and colleagues at the Disney Research labs in Pittsburgh, Pennsylvania have come up with a multiplayer gaming system called SideBySide that uses handheld projectors to let you play on any nearby surface.

SideBySide uses modified picoprojectors that emit both infrared and visible light. Invisible infrared markers that track the interactions between players are projected on top of the game’s graphics, which are currently limited to a single green colour since the other two colours of the red, green and blue projector are replaced by infrared.

Each player has a prototype device that contains a modified projector along with an infrared camera to detect the markers and a button for interactions, while processing is handled by a Macbook Pro. Games include competitive boxing and cooperative brick destruction. The system can also be used for more mundane tasks, such as transferring files.

If you liked this article, please give it a quick review on ycombinator or StumbleUpon. Thanks