|
Virtual Keyboard
Anonymous |
16 year
|
Hi! I have played with this software for 30 min and was wondering if someone could help me? What I need to do:
To set up a webcam looking at an A4 piece of paper with maybe nine 3cmx3cm boxes printed on it. Each box would have a letter or a picture inside it. I need roborealm to either
1. move the mouse to x,y and click
2. simulate a keyboard stroke
3. open a file
whenever someone puts their hand over a particular picture
The idea is to have kids with disabilities to have a customizable keyboard that is simpler than a regular one and can be designed/reprinted by their teacher according to their needs. Can anyone help me get started? I've looked at the some of the tutorials but can't seem to find out how to work with variables and blobs, or even what blobs are. Any advice would be appreciated.
Thanks
|
|
|
|
Anonymous |
16 year
|
how about putting some coloured finger-caps,you can track it accordingly WRT the motion over the perticular pic.
But i doubt how the sensation of touch can be established;guess you need another webcam for that.
Good luck :)
|
|
|
|
Saikumar. from India [20 posts] |
16 year
|
and to that anony post of mine wd like to add-
you can play around with objects also of diff colours,unlike just going for 2d,3d too must be very much possible
reg,
saikumar
good luck :)
|
|
|
|
Anonymous |
16 year
|
Thanks for the ideas guys! I thought a little bit about using coloured dots on the finger nails to track but realised that they won't always be visible and this needs to be a very robust system. I'm leaning towards using the skin module and activating tasks when something other than black pixels (i.e. skin) are in a certain area of the screen.
It all has to take in shadows, jerky hand movements etc so would be pretty complicated in the end. ATM I'm using autohotkey and a hacked keyboard to make a hardware version until I get something workable.
Again, thanks for the input!
|
|
 |
John Cabrer from United States [13 posts] |
16 year
|
Assuming that the simple keyboard can consist of a single horizontal row of keys, you could implement a binary system using the various colors to track which key is hidden by a finger or hand. Assign the following values to the colors for example:
Red=1
Green=2
Blue=4
Yellow=8
Cyan=16
Magenta=32
If you add these values up in a variable as they are obscured...
3=Red and Green pressed
20=Blue and Cyan pressed
23=Red, Green, Blue, and Cyan pressed.
This give 64-Bits worth of combinations (not including zero, of course).
|
|
|
|
Anonymous |
16 year
|
Hi John, that looks like an awesome idea!
The keyboard would ideally have 2-3 rows so I was thinking of having a timer that activates the command only if a key above it hasn't been activated so that it gives a user time to reach over the bottom row, but I'll take what I can get.
Also, I'd like anything to be on each button square such as pictures or letters and have it changeable. I tried "training" roborealm with some of the pictures so it can be flexible but don't know how to tell it to do something when that picture can no longer be detected or if there's skin tones on it.
Maybe if I had a coloured box around each key?
Could you give some guidance on how I would get roborealm to accept a square as being red? If a hand is only partially covering the button how do I average out the pixels of a section of the display and then round up/down to red? Averaging would be good cos it could take care of shadows if I have the button with the greatest difference to it's original be the pressed button...
Thanks for the ideas mate!
|
|
|
|
Saikumar. from India [20 posts] |
16 year
|
You can do this:
-find the "area"{a1<=area} of your decided colour squares using-"geometric statistics" module.
-when the finger is moved over you get another "area"{a2<=area} ,using "geometric statistics"
-compare the areas.
-do it for all the coloured squares
-find the key presses based on max of [a2/a1]
Hope it helps.
Saikumar
|
|
|
|
Anonymous |
16 year
|
Anyone care to upload a couple images that give us something to work with? Perhaps a sample image with no hands in the image and then one with a hand pressing a particular "button".
This should be possible using RoboRealm and we're currently looking into a keyboard stuffing routine that can do your final #3 step ... but before we get there understanding the environment that would be best for these kids is first.
The images will help us understand the expected lighting situations, perspective of the virtual keyboard from the camera, the initial layout of the squares, how large they are and which colors are coming out the best to be used for this application. Don't commit yet to one type of virtual keyboard but a single prototype would be great to be able to play around with.
STeven.
|
|
|
|
Anonymous |
16 year
|
Thanks for the input! This community is really helpful!
Hopefully the keyboard could work with any image whether through training roborealm with each image or through assigning an e.g. 40/40 pixel space that activates something when it is nolonger black (if using the skin module) so that the images themselves do nothing.
I'ved used Autohotkey to make a gui that can open any app/make any keypress or mousemovement/action that classroom teachers can use to modify their board's actions with to make it actually useful. The only probem is that I don't think it can 'capture' simulated keystrokes and simulate new ones... not yet anyway.
The orientation of the virtual keyboard (piece of paper) would have to be flexible, so I was thinking of having corner detection of the paper and warping the pixel activation spaces off that.
I can't really describe a constant lighting condition because I don't think there would be one except that there would be fluorescent lights and indoors away from bright shafts of sunlight. So imagine a desk in most offices.
Here are some of the pictures that would probably be used for various reasons. One is a virtualkeyboard that could open word, increase/decrease the font size and type 12 different letters (note the name shannon is mixed in there).
Sorry for the long post...
|
|
|
|
Anonymous |
16 year
|
Here's the same keyboard but distorted that would probably reflect it in the real world. It would be good to be able to use a webcam sitting on the monitor peering down at the virtualkeyboard.
|
|
|
|
Anonymous |
16 year
|
Regarding the lack of constant light source:
Newer webcams such as Logitech Fusion do a great job self-adjusting to varying lighting conditions. This should almost fully eliminate that problem. On top of that, you can always have an LED array around or next to the camera pointing downwards providing a little bit more uniform lighting conditions.
Also, I would personally use to webcams, one on the top left of the monitor and another on the top right. That way you can view the same area (keyboard) from two opposite angles, minimizing ambiguity which shadows form.
Cheers,
Nemanya
|
|
|
|
Anonymous |
16 year
|
Well here's my first attempt at making a simple webcam keyboard. Lighting isn't a problem since the skin module gets rid of any shadows.
All it does atm is find the COG of a filtered hand and then activate a red dot in the quadrant of the screen where the hand "mainly" is.
1. Anyone have any ideas on how to stabilise the COG so that it has fluid movement? I plan to be able to move the mouse with it and it's too jerky as it is.
2. Also, I want each dot or action that occurs in each activated quadrant to activate only once and then deactivate itself so the user has to move their hand out-then-into the quadrant to re-activate it again. Any ideas?
3. Lastly, any ideas on how to deactivate any accidental lingering keypresses when the hand moves off the screen? It takes a few seconds to realise there's nothing there.
Thanks
 program.robo
|
|
|
|
Saikumar. from India [20 posts] |
16 year
|
1.stabalizing the COG
-put the web cam little far away from your virtual keyboard.
Its working smoothly on my system here <:)
2.Use the time dependent variable,which would be activated only when you have finger on it for time duration=T.Similar kind of variable to sustain its effect for time duration=T'
3.not much that I can advice,but try to accurately filter out the colour/size/shape of key.POINT-2 may also help you here :)
one more suGGestion: try to have another[2nd] camera which is "seeing" the side view.
BTW nice progress! ;>
Reg,
Saikumar.
|
|
This Thread
