Recognizing Hand Gestures

The following tutorial describes one way to use a vision system to recognize hand gestures from an overhead webcam looking down at hand gestures over a solid background. The setup allows for various gestures (the digits 1 to 10) from one hand to be recognized by the vision system.

The basic principal to this recognition is to compare the shape of the hand once extracted from the background to a database of existing images that can be used to determine what digit is being indicated. Note that the digits 1 to 10 can be shown on a single hand with the hand in frontal view. We first start out this recognition process with a example image of what the camera sees.

Original Image

The hand can easily be extracted from the background using a AutoThreshold module.

Auto Threshold

The problem now is that the length of the wrist/arm can vary depending on how far in the image the hand appears. This is enough to throw a shape detection system off from the real shape of the hand. To remove the wrist from the shape we replace the hand blob with the largest inscribed circle.

Inscribed Circle

By dilating and subtracting this from the blob image we can separate out the individual parts of the hand.

Split Blob

The result can then be processed to remove any blobs that are touching the border of the image. Since the wrist should be the only part touching the border of the image it can easily be removed using the Blob Filter to result with just the fingers.

Wrist Removed

By merging the remaining fingers with the part of the blob that was removed by the inscribed circle we remain with just the palm and the fingers of the hand. As the inscribed circle will be relative to the size and location of the hand this process is quite repeatable as long as the hand is frontal.

Isolated Hand

After smoothing this final shape to remove any irregularities the Shape Match module which is trained on the following images. Note that this database is trained on images previously saved from this point in the pipeline to the file system.

one two three four five six seven eight nine ten

This module will generate a SHAPE_LABEL variable which can be used to indicate what shape graphic the module matched against.

The following  robofile can be used to replicate these results along with the following video. The database to train the Shape Module on can be downloaded as a Zip File.

Note that the gesture for 10 is very inconsistent due to the failure of the inscribed circle method. Once we move the hand to a profile view the inscribed circle will move on each frame since the hand profile does not have a flat inner palm area that dictates the placement of the inscribed circle. Once this moves sporadically the rest of the algorithm will not function well.


The End

That's all folks. We hope you've enjoyed this little adventure into an application of machine vision processing. If you have any questions or comments about this tutorial please feel free to contact us.

Have a nice day!

 New Post 

Hand Gestures Related Forum PostsLast postPostsViews
Getting started
Please ignore that I have selected "hand gestures". There is no option available for the simple problem I cannot solve.
9 year 3 2731
sign language
hi all, I'm working on project at college. it's robot, my part in project is vision. I want say "hello" by sig...
11 year 3 2544
hi guys i am trying to detect the gestures of my hand, using the blob shapes,, oper...
12 year 2 3339