Lego NXT Vision Ball Picker

(1.1 MB) Video from the point of view of the Lego NXT as it sees the ball and cone. This is the original unprocessed video.

(1.4 MB) The processed video as seen from the Tribot's perspective. Notice the switch from blue ball to red cone tracking!

Problems

It is worth pointing out some of the failures that we experienced during the creating of this tutorial. As one can learn much from mistakes we chose to share some of the issues this current robot configuration has.

1. Perhaps the most difficult issue is keeping the robot together. Grabbing, bumping, and jerking can reduce the solidity of the Lego robot and it tends to start becoming lose which causes the camera mount to drop or raise slightly. This changes the field of view of the camera. Giving a quick squeeze to the camera mount once in a while ensured that the mounting of the gripper claw was stable and that the camera could receive the best field of view.

2. The wireless camera+receiver can be very noisy. In fact many trials were terminated due to the camera image suddenly becoming violently colorized with jagged lines everyone (the kind of screen one sees in a highly noisy electrical environment). We will even go as far as creating a module around this issue to stop the robot when it detects this scenario. It only effects a couple frames at a time but often the ball is only seen for a couple frames and thus the robot can easily miss the detection.

3. Camera battery life is an issue. Once the battery becomes less powerful the image tends to become darker and darker. It is bad enough to have to deal with the specula lighting that is produced on shiny surfaces but we also had to deal with the camera light being reduced. Luckily using the flicker reduction and tweaking the threshold values the robot could still see the ball in an image where a human would have great difficulty in seeing it. In this case software was able to compensate.

4. The Lego NXT bluetooth is great for going wireless but we found that the delay in motor control can become too great for near real-time control. From the video you can see how the ball is almost overshot but then corrects itself. This is in part due to the induced delays inherent in the system of which Bluetooth is a factor. Regardless, reducing speed is the best way to solve this issue.

Your turn ...

Want to try this yourself? Here is the .robo file that you can download and run yourself. You will also need the VBScript code to be placed into a ball_picker.vbs file next to the .robo file. Note that you need RoboRealm installed to run this file. Clicking on any of the modules will bring up that interface and allow you to customize it for yourself. Try getting the robot to track a green ball instead of a blue one ...

That's all folks. We hope you've enjoyed this little adventure into an application of using machine vision to pick up and drop balls using a Lego NXT TriBot. We hope we have inspired you to download our software.

If you have any questions or comments about this tutorial please feel free to contact us.

Please Support Us! Our software is free but our costs are not. Please take the time to visit our sponsor's links on the right of each page. They work hard too!

Link To Us! Please visit our link to us page and add a link to our website to help us get the word out!

Have a nice day!