Many objects can be segmented from the background just using color. Working with multiple colored
objects can be tricky to keep track of them and to create a summary of the detected objects. This
tutorial shows two possible solutions for segmented M&M candy from the background and
provides a summary count of what is found in the image.
The first technique is to use the circular nature of the M&Ms to first segment them from the background. We do this by
first running the Canny edge detector.
We then use the Circle module to identify circle. We annotate the location of each circle
using a white disk 5 pixels smaller than the originally found circle. This ensure that circles are
NOT overlapping. This ensures successive analysis will consider each M&M individually.
Now that we have a mask for each M&M we replace each disk by the color nearest to the mean color of all the pixels
that exist in the same location in the original image. The Blob_Colorize module allows us to do this.
Instead of simply replacing each blob with the mean average color of the pixels the white discs are over, it checks first checks
within its list of colors for the nearest color and replaces the blob/disc with that color. This performs a crucial step. When
the replacement happens, the module also generates an array of the X,Y location of the blob AND includes a label that we create
along with those coordinates. The label is important since each green M&M can have very different colors. In order to
count all the 'green' M&M's we need to force green colored M&M's into a single label. The nearest color search does
this. In other words a blob with mean of 0,245,33 and 4,246,2 which are both green in color will be labeled with 'green'.
Because we now have an array of blobs (BLOB_COLORIZE_LABELS) and their labels we can use a basic VBScript to loop through all the
elements in the array in order to count the occurrence of each color.
list = GetArrayVariable("BLOB_COLORIZE_LABELS")
if isArray(list) then
if ubound(list)>0 then
for i = 0 to ubound(list) step 6
if list(i+2) = "green" then green=green+1
if list(i+2) = "blue" then blue=blue+1
if list(i+2) = "red" then red=red+1
if list(i+2) = "yellow" then yellow=yellow+1
if list(i+2) = "orange" then orange=orange+1
if list(i+2) = "black" then black=black+1
SetVariable "Orange", orange
SetVariable "Blue", blue
SetVariable "Yellow", yellow
SetVariable "Red", red
SetVariable "Green", green
SetVariable "Black", black
The main part of the VBScript code is
if list(i+2) = "green" then green=green+1
for each color which simply increments a variable (with the same name as the color we are counting) each
time that label is seen in the array. By processing the full array we can get a count for each color. These
are then set into RR variables so that we can use the Display_Variables
module to show the results. By using the Math module we can merge the full color
results back into the original image to verify that we didn't miss anything.
The first technique works well by utilizing the circle module to find the initial M&M's. This
works well for circular objects but suppose we had other objects that could not be so easily discovered
by their shape. In the second technique we assume the objects are not circular but instead JUST rely
The first step is to reduce the actual number of colors in the image. Any image made up by 1000s of pixels
will have very subtle color changes which are not exactly the same. By using the Reduce_Colors
module we effectively reduce the image into blobs of similar colors. These blobs will never be perfect in terms of circular
shapes but we have to start somewhere.
Now that the image has been reduced to a few colors, we want to remove those parts of the image that are not relevant to
our analysis. This includes the tan background (originally the wooden table) and parts of the image that make up the
shadows of the M&M's. The Circle detection module in Technique #1 avoided this problem.
The best module for removing unwanted blobs is the Blob_Filter module. This module is
perhaps the largest module within RoboRealm because it encompasses so many characteristics that can be filtered. Keep in
mind that the blob filter will see each different colored group of pixels as an individual blob. The module
reports that 202 blobs are found before the module runs. As we only want around 60 actual blobs we have to do a bit
of work to remove the 140 that we don't want.
At this point we could also go back and use a circular shape as an attribute to remove all non-circular blobs, but
we will stick with color and size to show more generic filtering. In our case we remove objects smaller than 215
pixels since our M&Ms are larger than that, remove objects larger than 5000 pixels (which eliminates the background
table blob) and then remove any blob that doesn't have a strong color. Objects such as shadows will not have a strong
hue and thus are removed. The problem, however, is that same filter ALSO removes the black M&M's ... since after all
black is not a very strong color!
To remedy this issue we have a second filter within the Blob_Filter module that selects
blobs that are much darker than most in the image. This combined with a size filter successfully selects JUST the dark
M&Ms. This result is then merged into the primary or Main filter to create the final image which includes both colored
AND dark blobs.
The problem now is that many of the blobs are still touching each other. In order to count them correctly we need to split
them into individual blobs. Before we start splitting we need to fill in the empty holes in the blobs. This is done
using the Fill module.
To split the blobs we separate them using the Erode module. By eroding the blobs by a 10
pixel perimeter which will break them into individual blobs.
Because we want to preserve the approximate size of each M&Ms we then use the Dilate module
to dilate each of the remaining blobs by 10 pixels BUT we ensure that no blobs will reconnect. This will then result in each
blob being separated from each other.
We can now follow a similar process as Technique #1 by utilizing the Blob Colorize
to color in each of the remaining blobs with their source mean color.
and then use the same VBScript to sum up the results.
While the results are the same for both techniques, the pixels that determines a final M&M is a little different. You can
see that the color technique does use more of the pixels to define an M&M (i.e. the final area is a bit larger) but
as that was not a final measurement we were interested in either technique will work.
In terms of performance, the color technique #2 (while including more modules) is 40% faster than the circle detection technique #1
just for this specific image.