Machine Vision
The RoboRealm application provides a simple and quick interface for experimentation into Machine Vision using a simple WebCam. Machine Vision is typically a complex technology involving insight into how images are represented within digital computers. A deep understanding of lines, edges, colors, textures, etc. is often needed with supported background in linear algebra, differential equations, eigenvalues, etc. This requirement often dissuades people from getting involved in machine vision technologies. RoboRealm's task is to provide a fun environment into exploring machine vision and related techniques without requiring significant technical and mathematical backgrounds.Machine Vision is the process of interpreting pixels that are captured using a digital device such as a webcam or digital camera. Typically, machine vision can be broken into the following parts:
1. Image acquisition - how you get the image
2. Image conversion - converting the image into a usable format
3. Image processing - tweaking pixels
4. Statistical Analysis - understanding the image through numbers
5. Machine control - moving, rotating, etc. a hardware device based on the image
Image Acquisition
Image Acquisition is how you get the image into your computer. While machine vision does not strickly indicate live or saved images it is often more useful to have video capture devices provide images for a machine vision application due to the need for realtime controlling of machines. During development or prototyping one can use saved images for processing which removes much of the difficultly of acquiring images until the final deployment stage.
For experimentation image formats such as gif, jpeg, etc that can be easily acquired via the web can provide a large test bed of images surrounding a specific topic.
For realtime acquisition of images, Windows does provide a more standardized technique called "Video For Windows" or VFW. This technique allows you to create a single program that can then interface with most if not all video capture devices that are supported under windows. RoboRealm supports this technique and we have successfully interfaced to many video capture devices such as webcams, TV capture devices and digitizers.
Realtime acquisition of images in the Unix environment is somewhat more arcane and requires understanding of a specific device's interface. For many capture cards the device exposes its interface via a file based handle, as much of other Unix based device drivers are. However, the specific commands used to set capture images require specific commands that are usually not transferable to different devices. Nevertheless, many machine vision applications are built custom towards a specific purpose and therefore do not need to interoperable between a large number of devices.
The result of the image acquisition process is to have in memory or on disk a sequence of bytes that represent the image in some format.
Image conversion
The image conversion process takes the results of the image acquisition and converts the image into a format that is easier to use during the next couple of machine vision phases. The conversion process focuses on how the image is formatted (i.e. is it a compressed image file like GIF or JPEG) and how the pixel values are stored or packed (i.e. what are the color value ranges, is the color value an integer or floating point number?).
Many hundreds of image formats exist but some are more popular and in wide use than others. For web related applications the typical image formats of GIF, JPEG or PNG comprise the majority of the image accessible on the web. For other machine vision applications TIFF or PPM are more widely used. If your image acquisition process leaves you with an encoded or compressed file you will need to decode the file in order to gain access to the raw pixel color values before proceeding to the next machine vision phase.
If you have chosen to skip the image acquisition phase you will most likely be dealing with images that are encoded as JPEG. To access the raw pixel values you will need a package such as FreeImage to load and provide the pixel values to your application. Note that Windows also has built in JPEG decoding routines but FreeImage supports more formats and works in multiple environments.
Traditionally, image acquisition devices will provide you with a sequence of bytes that are packed in a specific way. If you're lucky you will have a sequence of bytes that are stored as 0-255 values in sets of 3. The first byte will traditionally be the red value, followed by the blue and then green. The fourth byte will then be the next pixel's red value, and so on. This is known as an 8 bit RGB packed pixel. The 8 bit refers to how many bits are allocated for each component of the pixel. This format is also referred to as RGB 24, the 24 being the sum of 8+8+8 which represents the 3 values that comprise the pixel. This format is usually the easiest from a conceptual point of view to process and thus is usually the desired format at the end of the image conversion process.
There are many other pixel packing formats such as I420 or RGB 555, etc. All of the formats just represent the image pixel values in a different way. For example, RGB 555 uses just 5 bits (instead of the previously mentioned 8) to store color values in an RGB sequence. The main different is that the RGB 555 format is less precise than the RGB 24 format since fewer bits are used. The benefit is that an image is much smaller and can be transmitted much faster through smaller bandwidth pipes (note that RGB 24 is 3 bytes per image whereas RGB 555 is 2 bytes with one bit being unused). In addition, the loss of image color while perceptible is not significant and will usually not alter and machine vision processing results. If image size and transmission is not and issue RGB 24 is normally preferred since it is easier to work with byte aligned (8 bit values) rather than have to split bytes into bits to extract the RGB triplet.
Image Processing
Once you have the image in a usable format (RGB 24 is in use in RoboRealm) you can start the image processing phase of the machine vision application. Image processing is perhaps the most time consuming and difficult phase of a machine vision application. This stage requires you to convert the image using any number of many many techniques to change the image into a desired view.
For example, if you have a need to track an object then you first need to decide what characteristic about that object can be used to detect the object as robustly as possible. These characteristics are often referred to as image featured. Note that more than one feature and sometimes thousands can be used to detect or track and object. These features can range from simple color or intensity (object brightness) to more complex features such as edges or shapes. Some features to use when determining your approach can be
- color - does the object have a unique color (i.e. neon green, bright purple, etc)
- intensity - is the object brighter or darker than other objects in the image
- object location - is the object always in the top of image, right corner of image, etc.
- movement - does the object move in a specific way, i.e. does it wiggle, sway, move slowly, stationary
- texture/pattern - does the object have a unique texture (i.e. tree bark, red bricks, pebble stones)
- edges - does the object have well defined edges that are straight or circular
- structure - given simpler blobs or parts of the image can the object be composed of simpler objects arranged in a specific manner?
Statistical Analysis
Processing an image for specific features can enhance and extract aspects of the image that can be used to focus on the final target. However, feature extraction through image processing can result it many MANY numbers that need to be reduced substantially to be useful. For example, when driving a robot based on an image thousands of image pixels will reveal hundreds of pixel edges. Statistical Analysis of those resulting pixel edges can reduce the hundreds of pixel edges to a single Center of Gravity number that can be used to turn the robot to the left or to the right. There are a number of statistics that can be deduced from the image (some even without any image processing) that can be used to tie an image into a usable few or single numbers that represent the image.
Several statistical analysis numbers can include
- center of gravity - determines where the resulting pixels could be balanced on a single point. Also referred to as center of mass.
- pixel count - count the resulting pixels that are left. High numbers can mean an object is present, a low number means it is absent.
- blob - an area of connected pixels - size, location, number, etc can be computed
- max, min, mean, mode - basic pixel value calculations
Machine Control
Once the appropriate numbers have been calculated they need to be translated into motor or servo movements in order for the robot or machine to react to what it sees. Controlling a machine from a PC computer typically requires a board that translates serial or parallel commands into PWM (Pulse Width Modulation) or actual electrical current via an H-bridge. Many servo boards meant to control traditional servos exist. Have a look at Parallax for examples of such boards. Sending commands to such a board requires that the statistical numbers calculated from an image processed image be translated into left and right motor commands in the case of steering a robot. This translation can be in the form of simple condition statements that turn a motor on or off based on some thresholded value. For example, if the COG (Center of Gravity) of an image is left of the image center send the servo board a 255 for the left motor. Likewise send a 255 position command to the servo controller board for the right servo to move the robot right.
Often, controlling several motors based on an image may require inverse kinematics to determine the angular values for each of the servos required to position an arm or other complex robotic device to its desired position. For example, if a robotic arm has two servos that function like a human arm to position the hand at a certain point requires the two servos to have specific angles that will position the hand at a certain X, Y point. Calculating angular values for these servos is dependent on the resulting X, Y coordinate and on the length of the arm parts involved. Refer to the subject of Inverse Kinematics to find out more on how to calculate the appropriate angular values for the servos.