Microsoft Kinect

The Microsoft Kinect module provides an interface to the Microsoft Kinect XBOX 360 sensor. The sensor is remarkably useful for robotics and is a great depth sensor at a reasonable price. While the Kinect was created as an addition to the XBOX it can be used just in conjunction with a PC to gain access to the depth information that it provides.

The main feature of the Kinect is that it provides a 640x480 depth map in realtime (30 fps) that indicates which objects are near versus far in the given scene. This is invaluable for use in obstacle avoidance and navigation. In addition to the depth sensor the Kinect also provides a traditional 640x480 RGB image, a 3 axis accelerometer, the ability to tilt its head and a nifty LED that you can change the colors on!

The Kinect RoboRealm module provides access to the depth map, RGB image, accelerometer information and provides the ability to control the head tilt angle all from within RoboRealm. Note that once the depth image is within RoboRealm all the other traditional filters can now be applied to process the depthmap in new ways. If you would prefer to just have the image as a webcam you can use the RoboRealm virtual camera to pipe the image into other webcam compatible application. Or if you just want to work on the image yourself you can use the various ways to interface with RoboRealm including our popular and flexible API.

While the Kinect is a great addition to robotics there are some limitations. First, the depth map is only valid for objects more than 2.5 feet away from the sensing device. If an object is closer than 2 feet to the Kinect its depth will NOT be detected correctly and be presented as a zero depth. If you have a large robot you can place the sensor at the back such that the depth is generated at the border of the robot. For smaller bots you'll just have to avoid getting too close to objects otherwise they will fall too close to the sensor and come up as unknown (black) areas.

Second, the Kinect performs its magic by using an IR projector with an IR camera (a form of stereo vision) which means that our favorite bright yellow friend, the sun, will cause some issues with its functionality on a sunny day outside. If you use the Kinect outside it will NOT work in direct sunlight but will in shady areas. However, a nice aspect of having IR being projected is that it provides high resolution images in total darkness. In this case your robot can navigate around without needing any daylight!

Third, as the device is essentially a projector it draws about 2Amps of power in order to run. This may not be an issue on a large robot that has large batteries but may be an issue on mid sized robots that are power conservative.

Regardless of its shortcomings the Kinect offers a much richer data set than most advanced laser systems and at a significantly lower cost! It produces one of the best stereo depth maps that we've ever seen ... and in realtime!

Win XP Kinect PC Installation

The installation of the Kinect on a PC is not as straightforward as connecting other plug and play USB devices. Because official drivers for the Kinect on a PC have not yet been created/released we have to use an alternative driver that provides sufficient flexibility to get things working. For that reason you will need to perform the following steps to get your Kinect functioning with a Windows PC. Follow these steps BEFORE plugging it in.

1. Download - Download the open source libusb-win32 application from sourceforge and unzip it on your computer. You will need the bin version as apposed to the src version unless you plan to compile the library yourself. This system provides a generic windows driver that can be used to communicate with the Kinect if you know the right commands. The main file you need within this download is called libusb0.sys.

2. Download - Download the Kinect .inf and .cat files. These files help Windows understand what driver to use with the Kinect. You will have to also unzip this into a location that you know as you will need to specify this path later. These files were generated by using the inf-wizard.exe program within the libusb package but are included here as downloads for speed and ease of use.

3. Kinect - Plug in the Kinect into the PC. The PC should respond with a driver install dialog. Select the radio button that allows you to tell Windows where the drivers are "Install from a list or specific location (Advanced)". On the next interface using the Browse button you can specify the path where you saved the .inf and .cat files. That should make Windows happy!

4. libusb - Once the .inf files are used Windows may complain about not knowing where the libusb0.sys file is. You will have to find that file in the c:\???\libusb-win32-bin-1.2.2.0\bin\x86 folder where you downloaded and installed the libusb files. Note that the ??? might be "Program Files" but it will depend on where you chose to install the libusb application.

5. Repeat the above steps of specifying where the .inf files are and the usblib0.sys file is for the Motor, Camera and Audio devices of the Kinect (3x). Note that the Motor driver includes the accelerometer and led drivers.

6. RoboRealm - Once this is complete you can now fire up RoboRealm and insert the Kinect module (easiest way is to select the Search tab and type in Kinect). Once this module has been inserted the default configuration should show you the depth map and the RGB color image as a single large image within the main RoboRealm GUI.

Win 7 Kinect PC Installation

1. Uninstall - You MUST uninstall KinectSDK for the following to work with Windows 7. Windows 7 will automatically download the KinectSDK but due to commercial licensing restrictions we CANNOT support those drivers. You can uninstall the drivers by going to your Device Manager, right clicking on the Kinect drivers and select uninstall. If you fail to uninstall these drivers the following drivers will not be able to receive images or depth information from the Kinect and appear as a black image.

2. Download - Download the open source libusb-win32 application from sourceforge and unzip it on your computer. You will need the bin version as apposed to the src version unless you plan to compile the library yourself. This system provides a generic windows driver that can be used to communicate with the Kinect if you know the right commands.

3. Download - Download the Kinect .inf and .cat files. These files help Windows understand what driver to use with the Kinect. You MUST unzip this into the c:\???\libusb-win32-bin-1.2.2.0\bin\ location of the previously download and unzipped file. The reason this location is required is that Win7 will NOT permit specifying where the .sys file is if it is in a different location than the .inf file. Placing those files in this folder will allow Win7 to find all the appropriate files in one location.

4. Kinect - Plug in the Kinect into the PC. The PC should respond with a FAILED driver install. Ignore this but let things settle (i.e. no more popups) before continuing.

5. hdwwiz - Click on Windows button and in the Run dialog box type in hdwwiz and press OK. This will run the hardware wizard. Click Next to get started.

6. hdwwiz - Select "Install the hardware that I manually select from a list (Advanced)". Click Next.

7. hdwwiz - Keep the "Show All Devices" selection and press Next.

8. hdwwiz - Click on the Have Disk button and select that libusb bin folder where you also unzipped the .inf (Driver) files into. Click Next.

9. hdwwiz - Select one of the 3 items (Camera, Audio, Motor) to install. Click next and allow driver to be installed when security message pops up.

10. hdwwiz - Repeat running hdwwiz for the next two items such that all 3 devices have been installed.

11. RoboRealm - Once this is complete you can now fire up RoboRealm and insert the Kinect module (easiest way is to select the Search tab and type in Kinect). Once this module has been inserted the default configuration should show you the depth map and the RGB color image as a single large image within the main RoboRealm GUI.

12. Problems - If this does not work try first rebooting and re-run RoboRealm.

Vista Kinect PC Installation

2. Download - Download the Kinect .inf and .cat files. These files help Windows understand what driver to use with the Kinect. You MUST unzip this into the c:\???\libusb-win32-bin-1.2.2.0\bin\ location of the previously download and unzipped file. The reason this location is required is that Vista will NOT permit specifying where the .sys file is if it is in a different location than the .inf file. Placing those files in this folder will allow Vista to find all the appropriate files in one location.

4. Kinect - Plug in the Kinect and wait for the cannot locate driver software popup.

5. Select "Locate and install driver software". Click next.

6. Press 'Continue/Accept' when security confirmation request appears.

7. Select 'I don't have the disk, show other options' when Vista complains about not finding driver.

8. Click on 'Browse my computer for driver software (Advanced)'

9. Specify the libusb bin folder where you also unzipped the .inf (Driver) files into. Click Next.

10. Select "Install this driver software anyway." when Vista complains about the driver not being signed.

11. Repeat the process for the next two devices.

12. RoboRealm - Once this is complete you can now fire up RoboRealm and insert the Kinect module (easiest way is to select the Search tab and type in Kinect). Once this module has been inserted the default configuration should show you the depth map and the RGB color image as a single large image within the main RoboRealm GUI.

13. Problems - If this does not work try first rebooting and re-run RoboRealm.

Interface

Instructions

1. Images - select which image you'd like to focus on. This will display that image within the main RoboRealm GUI window. By selecting the different radio buttons you can chose between the depth image (the image that shows the distance to an object based on pixel intensity/color) and/or the RGB image (the typical colored image you see from webcams) and/or the IR image. Because you may be using the images in different ways you can chose to have images saved in memory as marker images or be show in the main RoboRealm interface. Note that if you are not using both images you can chose the Show Depth or Show RGB only as they help to reduce USB traffic in case you are working with a slower USB bus. Also note that the IR and RGB image CANNOT be used at the same time.

2. Motor - The Kinect has the ability to tilt or bow its head. You can select the current value from 0 to 128 (with 64 being about level) to reposition its head. To automate this movement you can instead select a variable that contains or will contain a numeric value from 0 to 128 which will set the tilt angle. Note that once you select or type in a variable the manual number will be disabled to indicate that the tilt angle is now under programmatic control.

3. Depth Options - The depth image that the Kinect products is a 11 bit (0 - 2047 value) number that can be displayed in several ways. The most common depth map display uses intensity (black to white) to indicate the distance to any particular pixel within the image. This can be useful for successive processing (for example threshold) but may not best indicate the different depth levels. Thus you can use the different color schemes to provide more insight into what depth levels are present in the image.

Grayscale - reduces the depth information from 12 bits to 8 for each RGB pixel

Color Scheme X - uses the depth value as an index into a color palette. Each palette uses a different color transition.

Line - produces a line that indicates the closest depth to an object. This is similar to what a SICK or LIDAR laser would produce. Note this option enables the Top and Bottom settings which allow you to crop the depth map to a specific height. Reducing the height allows objects that are above or below the needed detection level to be ignored when creating the distance line.

Raw Depth - the 12 bit number is encoded with 8 bits in the red channel and the remaining 4 in the green channel. This allows you to access the full depth resolution in external applications by decoding the R and G channel. (Note the B channel is set to 0).

4. Gamma - While the Kinect can produce up to 2048 depth levels we've not found it to go much beyond 1600. Because of this not all the colors will be displayed or used. You may also want to focus more on closer depths and lose some resolution at more distant depths (since gray images are 0-255 you lose 3 bits). Using the gamma correction can help to bunch up values to better highlight the different depth values.

5. Threshold - You may not want objects that are far away from the Kinect to appear in the depth map. Using the threshold value you can remove far away objects and also compress the color space of the resulting image to better show depth differences. Using the threshold you can eliminate all but nearby objects that can then be further processed by other modules.

6. Accelerometers - The 3 axis accelerometer provides position information for the Kinect. The X and Y values indicate roll and tilt with the Z axis being a check to determine if the device is upside down. Keep in mind that the Accelerometer only measures tilt and NOT orientation (Z axis).

7. Smooth - Increasing the smooth value will smooth the unwanted jitter due to noise that the accelerometer numbers will typically display. By decreasing the smooth value you make the device much more responsive to movement, increasing the value will delay the sharpness of the movement but also make it much more stable and change gradually.

8. LED - Use the radio buttons to select what state you want the led to be in. To programmatically control that state select a variable that contains or will contain a value from 0 to 6 that will cause the led to change state.

9. Brightness - refers to an intensity or luminosity scale of the RGB image that ranges from totally black to totally white and has no effect on color values.

10. Contrast - refers to how far pixel values can deviate from gray in the RGB image. The higher the contrast the more black and white and image appears. The lower the contrast the more gray an image appears.

11. Auto Exposure - Allow the Kinect to adjust exposure time for the RGB image.

12. Exposure - refers to how long your camera takes to record an RGB image. In a well-lit scene, exposure times can be very short because plenty of light is available stimulate the CCD pixels with enough energy to record an image. At nighttime, exposure time will increase dramatically due to the near absence of light. A quick exposure time will also reduce motion blur, whereas a slow will introduce more blur if the object is moving.

13. Saturation - refers to the intensity of the RGB colors, i.e. how red a red color is. By decreasing the saturation of an image you remove color and produce a monochrome grayscale picture that represents only darkness and brightness, or luminance. Increasing saturation in an image produces artificially intense colors.

14. Auto WhiteBalance - Allows auto adjustment of the RGB image colors such that white pixel are adjusted to a white color.

15. IR Properties - Specifies the intensity the projector that is transmitting the IR dots. Note that the 1280x1024 mode has the highest IR projector value whist the smaller modes allow for some adjustment.

Example

Microsoft Kinect Depth Map with RGB image

Variables

KINECT_DEPTH - when saved as a marker this variable contains the depth image
KINECT_RGB - when saved as a marker this variable contains the RGB image
KINECT_X - the X axis accelerometer value
KINECT_Y - the Y axis accelerometer value
KINECT_Z - the Z axis accelerometer value

For more information

Hector Martin - Open source Kinect driver
Kinect driver for Windows prototype!
Kinect drivers on YouTube

New Post

Microsoft_Kinect Related Forum Posts	Last post	Posts	Views
Unable to find the Microsoft Kinect Sensor! Hey guys, I've been trying to use my Kinect with Roborealm and I can't seem to figure out how to get the program to recognize th...	7 year	2	1629
Kinect Depth Image I have an original Kinect that I'm using with a Dell Latitude i5 Windows 7 64 bit laptop. I've followed the instructions for ins...	8 year	3	2502
Failing to pause pipeline after waiting 10 seconds... Hi, I have written a c# application that is based on the c# sample app that uses sockets to communi...	9 year	7	3564
Kinect Hi all, I'm working the kinect for my FRC team this year. The tutorial at ...	9 year	2	2829
Depth marker data format Hello, using only Depth (RawDepth) with my Kinect I get 640x480 pixel on the display, origin: lower left corner. <...	10 year	2	2868
No RGB-data with Kinect Hardware: PC Windows 7(64bit) and Xbox 360-Kinect (early version) ...	10 year	3	2877
Problems connecting to xbox 360 Kinect I am getting an "Unable to find the Microsoft Kinect Camera Sensor!" error when trying to load the Microsoft Kinect module.&nb...	10 year	3	3189
Conect Microsoft Kinect to RoboRealm? Hi, I'm new using RoboRealm. I try to conect Microsoft Kinect to RoboRealm, but I have ...	10 year	5	3201
Xtion shows artifact in line mode I have an Xtion Pro Live running in roborealm 320x240, line mode with the max min set at 61, 59. No matter where t...	10 year	5	3464
Kinect RGB image anomalies Kinect module working well in RR 2.49.13 with Microsoft drivers on Windows 7, but a couple of odd effects:	11 year	3	3115
object recognition with Kinect Kinect provides both depth information and rgb information. (1) What is the best module to analyze...	11 year	2	3248
Use of Kinect Module in FRC Hello, We are using the a Kinect on our robot for this year's FRC game. It has been tracking fine, however, every...	11 year	4	3323
set intensity of Kinect-IR-Projector hi, i have found the GUI for Kinect-Interface-Setting under ...	11 year	2	3533
Kinect IR 320x240 I've been using Roborealm with the Kinect for object recognition. I've been using the IR function fine, but I have it hooked u...	11 year	6	3707
kinect color proportional to depth > I need a software/code that allows kinect to display > black and white contrast levels proportional	11 year	2	3917