Augmented multitouch with Microsoft Kinect

By Christopher, January 15, 2013

I am glad to present something I worked on at my current workplace Tandemic. Much thanks for giving me the possibility!

Get some impressions in these videos:

The awesome video by Lena:

The less awesome recorded video but with some explanations from myself:

This human computer interfaces uses an imaginary augmented touch screen in the air in front of the user. So if the user is touching through this plane with his hands, this is considered as a click/touch, if his hands are in front of  he can see moving cursors. The plane of touch in the air can be configured to be in a relative distance to the users body or in absolute distance to the Kinect sensor. The video shows only the configuration with the relative imaginary touch plane, because we experienced this is working better in general. The distance and mode is configurable.

In contrast to existing solutions I do not try to recognize the click by gestures like opening or closing the palm or something similar, because for me this seems not to work reliable with the current available Kinect sensor. I think the resolution is not high enough to detect this reliable in a certain distance from the sensor. Moreover it needs a lot of processing power to do the image processing on each frame. So my solution does not require much processing power and will run on nearly all today available hardware. E.g. I am running it on my Acer1825 with just 1,3GHZ low power dual core without any problems or heavy load.

For emulating the touch interface and injecting the touch commands to windows I have based my solution on the multitouchvista framework. That is why you can use all windows integrated touch gestures and touch enabled windows applications. The interface is capable of tracking several users at the same time and every user has two cursors/hands for interaction.

The interface will even work on Windows 8 but I have not figured out yet how to inject nice additional cursors which are working nicely on the new startscreen.

For reference here are other available solutions(, which are using also the multitouchvista framework):

Unfortunately I did not find the code from the following one:

http://www.barbagallogiuseppe.com/Prof/MultiTouchDriverKinect.pdf

The next solution requires a Nvidia CUDA enabled graphics card as long as you are not recompiling the used Emgu CV. I was a able to try it on a colleagues computer.

http://junedmunshi.wordpress.com/2011/08/01/kinect-multi-touchvista-windows7/

http://www.kinecthacks.com/kinect-multitouch-for-windows-7/

This frameworks enables further hand and finger detection with the kinect and includes a demo for controlling the mouse cursor by hand movement and clicking with several different available gesture. But this is also not reliable enough in my opinion, but worth a try.

http://candescentnui.codeplex.com/

 

 

My code and further instructions and details will be available soon on github.