This tutorial shows one possible solution for tracking insect paths. The tutorial surrounds the Blob_Tracking
module which provides various parameters for connecting blobs on successive images in order to determine
movement. The goal of the tutorial is to identify the individual paths of the insects in order to further
analyze their interactions and quantify their movement. The path coordinates are provided for external
applications to process the results in the desired way.
Movement tracking requires successive images which are best represented in movie files. For this tutorial
we use the following raw video file provided by Juan Pablo Busso from the University of Zürich.
(click to download/view)
As we are working with multiple frames in a video we use the Media Reader module
to play the video frame by frame. Note that there are many other modules that can be used to play videos or stream
network images. Mutliple modules ensure that many formats can be used within RoboRealm to analyze frames.
(Media Reader, VLC_Player, Read_AVI, HTTP_READ, RTSP_Player)
From this video we pick an individual frame to test processing. Prior to tracking blobs we need to prepare the image
for best results. While this is not a huge issue, it can cause the moving blobs to be broken into smaller blobs
that could not be tracked. By zooming into one part of a single image we see our first problem.
The video signal sampled is from an NTSC source that has interlaced the image. Left and right rows are successively
updated with the current live image which causes a broken object in areas of high movement. Interlacing images
was common when video transmission capabilities were more limited and while still present today are slowly
being replaced by high resolution single image frames.
The solution to this is to use the Deinterlace module which uses various
techniques to eliminate the jaggies caused by movement. Using this module our zoomed single image now appears as:
As the insects are contained within a particular area we can Crop the image to just
focus on that area the insects will be contained within. Cropping the image to focus on just the area you want
to investigate is always recommended since cropping is quick and it reduces the amount of data to be processed in
In addition to cropping, we also use the Normalize module to correct the lighting
to increase the contrast between the insects and the white background. Given these two additions the image appears
We are now ready to extract just the insects from the white background. As we have such a high contrast between
the insect and the back ground we can use the Auto Threshold module
to separate foreground and background. Because RoboRealm processes blobs as white objects we select the Black Mask
to invert intensities, i.e. black objects become white. While its easy to switch from one thresholding method
to another, care should be taken to watch the entire video using any particular thresholding technique. A particular method
may work for the current frame but fail in previous or successive frames.
As we are working with relatively small objects we use the Dilate module to help consolidate
those insect objects that may be broken due to the thresholding. This ensures that bug parts are combined into a single
We still have some parts that are not actual insects. In the upper right corner there are two black dots that are
not insects and stationary. We can remove those using the Blob_Size module since they are smaller
than the actual insects. While these would remain still and would be ignored during tacking, we chose to remove
them prior to tracking.
As we now have a black and white image, we can finally use the Blob Tracking
module to connect the blobs in each frame with the frames before. There are many options to chose from in terms of
how to associate a blob with its previous image blob. In our case, the insects move quite a bit from image to image
so checking for overlapping blobs will not work for the fast moving insects. Instead we use the Location or Proximity
and Size attribute to connect blobs. The location assumes that blobs that are closest to previous blobs are the
same blob. The Blob tracking module is able to determine the correct association by analyzing all blobs in the image
for their best match. Thus if a fast moving insect moves close to a stable insect, the global solution will determine
that the stationary insect will not match to the fast moving one. We use the size attribute at a diminished weight (25%)
to ensure that in situations where two fast moving insects cross each others path the actual size of each insect will
help resolve the ambiguity.
Your specific project may require additional attributes to be considered in order to resolve ambiguities. A simpler
solution for this tutorial would be to increase the frame rate of the camera to capture smaller movements. The location/proximity measure will
then become more effective due to the smaller movement and the overlap attribute might become valid.
While experimenting, be sure to have the module set to label colors. This will color each blob with a distinct
color that if suddenly changes indicates that the tracking was lost for some reason. Slowing down the Media Reader
(or VLC or AVI_Reader or ...) to play frames slower can help provide some insight as to why tracking may have
A couple frames from the playback show the growing paths of the insects:
The module monitors the VIDEO_FRAME variable (selected in the interface) to check when the video loops. Once it
does, the tracking information is cleared and restarted. This ensures looping video is processed correctly
by restarting the tracking on each loop.
Once the information is generated exporting the path point data for each object we can use the
Write Variables module. By inserting the BLOB_TRACKING
variable into that module a CSV file will be generated that contains the movement in each image frame. The format of that variable
is BLOB_ID, BLOB_X, and BLOB_Y which means that after the full run we will have all path points
of each blob and also in what frame they moved.
Once tracking is complete, we may want additional statistics about the tracked blobs beyond what the
Blob_Tracking module provides. In this case, we add the Geometric_Statistics
module that generates an AREA array of the number of pixels of each blobs. Note, we add this module just
AFTER the Blob_Tracking module (with Annotation enabled) to ensure that we see the same image as the
Blob Tracking module. This is
necessary since the two modules need to see the same blobs in the same order otherwise we cannot correlate
the Blob_Tracking ids with the Geometric_Statistics array. As the Blob_Tracking module can remove blobs
(Remove Un-Tracked option) its important to process the same image in order to get the blob order correct.
To perform the annotation we use a VBScript module to create an array of X, Y coordinates for each blob
followed by a Text string of "Area: Z" which can be used in the Display_Text module to display an area
value above each blob.
The key to this array is that
BLOB_TRACKING_IDS - array created from blob tracking with zero entries for non-tracked blobs
AREA - array created from Geo Stats module of each blob
are two arrays generated by different modules but are in the same order. Most modules in RoboRealm analyse
the image from the left to right, bottom to top order. As long as the same image is used in both modules,
the ordering of blobs WILL be the same. Given these two arrays, we know have enough information to
correlate a blob id with its area.
ids = GetArrayVariable("BLOB_TRACKING_IDS") ' from blob tracking'
area = GetArrayVariable("AREA") ' from Geo Stats module'
if isArray(ids) then
if ubound(ids) > 0 then
j = 0
for i = 0 to ubound(ids)
if ids(i) <> 0 then
areaList(j) = GetVariable("BLOB_TRACKING_" & ids(i) & "_X")
areaList(j+1) = 20 + GetVariable("BLOB_TRACKING_" & ids(i) & "_Y")
areaList(j+2) = "Area: "&area(i)
SetArrayVariable "AREA_LIST", areaList
Note that with any solution it is very important to test enough images to validate correctness. Testing on only a
couple seconds of video is not sufficient for a reliable solution.
Download the Blob Tracking robofile to see the steps
and more about the modules used. You can download the original video
used in this tutorial here.
That's all folks. We hope you've enjoyed this little adventure into an application of machine vision processing
and have inspired you to download a free trial of our software.
If you have any questions or comments about this tutorial please feel free to