loading
 
Using "Eliminate Global Flow" in the Optical_Flow Module
from United States  [214 posts]
7 years
Hello,

I would like to use the Optical Flow module to track a moving object even when the camera itself moves.  The camera is mounted on a pan and tilt platform using Dynamixel AX-12+ servos and the motion is fairly smooth.  I am hoping that the "Eliminate Global Flow" option in the Optical_Flow module can be used to solve my problem.  However, after experimenting a bit with the module, it is not entirely clear to me how to proceed.  What I had hoped was that the flow due to a smooth panning motion of the camera would be essentially eliminated leaving just the motion of a test object.  However, it seems that there is still significant global flow left over even after checking the Eliminate checkbox.

I have created a few sample videos to show what I mean. The first member of each video pair shows just my hand moving back and forth.  The second video then adds a panning motion of the camera.  In the first pair, I display the optical flow blocks in yellow.  In the second pair, the flow needles are displayed in red.  And the third pair is just the raw video without any overlay.  My question is whether or not anyone can figure out how to extract the moving hand pixels from the videos that include panning of the camera?

First Video Pair: Optical Flow Blocks in Yellow
http://www.pirobot.org/videos/optical-flow/test.avi
http://www.pirobot.org/videos/optical-flow/test2.avi

Second Video Pair: Optical Flow Needles in Red
http://www.pirobot.org/videos/optical-flow/test3.avi
http://www.pirobot.org/videos/optical-flow/test4.avi

Third Video Pair: Raw Video without Overlay
http://www.pirobot.org/videos/optical-flow/test5.avi
http://www.pirobot.org/videos/optical-flow/test6.avi

Any ideas would be greatly appreciated!

--patrick


Anonymous from United Kingdom  [99 posts] 7 years
Working on the same problem.  No great success.  Here is what I am trying so far.   First thing to do is try to pull the object of interest out from the background. (In this case your hand out from the "room")  
This can be done using  a large search window size and then filter on min length to just one less then the size of the search window.  This will mean that the displayed flow vectors are only those of the object moving the "most" which should also be the "closest" object to the camera.  

What I actually do to isolate the largest needles is take the needle array, calculate the magnitudes of each vector and then sort the array in order of magnitude.  I then pick off the top 20 highest magnitude needles and use those(This should just be that largest object again)  I then calculate the center of this object using a COM calculation.

Next, the AX12s give you position data which is fairy accurate.(The velocities are garbage.)   (How are you controlling the AX12s?)     I have them report their positions on the serial bus and with each frame I request the current position of each servo.   Do the forward kinematics for your pan and tilt to figure out the actual position of the camera in each frame. Calculate the planar velocity for the camera by projecting those two points onto some imaginary plane in the front of the camera (I use 40 cm, and yes this is bogus)  Then subtract that vector from your global flow vector.

Now in my application, I am just worrying about pan.  This makes the math much easier!  

Keep me updated on what you come up with!

 
Anonymous from United Kingdom  [99 posts] 7 years
Meant to mention this previously, also look at using a visual anchor.  This can help to isolate the global motion from the local motion.  Haven't had time to pursue this yet.  Again let me know how it goes!
from United States  [214 posts] 7 years
Hey profmason,

Thanks for the great ideas!  I'll try to give them a go this weekend and will let you know how it turns out.  I am controlling the AX-12's with the free Dynamixel library from http://www.forestmoon.com/Software/DynaCommander/.  This works with the CM-5 over a serial link in "Toss" mode with the CM-5 set to "Manage".  My robot control program is written in C# and I use RoboRealm via the API.

--patrick

from United States  [60 posts] 7 years
Hi P & PM,

Just stumbled on this thread. I started working with optical flow the last couple of days after finding some papers on using it for obstacle avoidance.

Here is the approach I've taken which seems promising:

Stabilize, Optical Flow, Geometric Statistics (for COG)

The Stabilize cuts down on some of the noise. The COG tells when a single obstacle is coming near and you can turn away from the COG.

The problems are when there are multiple obstacles and when you directly approach an obstacle.

I've recorded AVIs so I see what is happening in slow motion and this all seems to work. I playback the AVI to try different approaches.

What would be nice on the Optical Flow is being able to get / display only  the top/bottom N percent of the needles based on size. That should eliminate the background needles if there is something in the foreground.

I'll post here and on my blog when if I get something more substantial working. Please do the same. I post my robo file but you'll have to get your own AVIs.
program.robo
from United States  [214 posts] 7 years
Hi Rud, Prof Mason and STeven,

I finally had a chance to come back to this problem and I think I found something.  
As I mentioned earlier, I was hoping that the "Eliminate Global Flow" option in the Optical Flow module would do the trick but it seems to have minimal effect on the block intensities.  However, I just discovered that the same option in the Movement module (there called "Reduce global movement") seems to work very well!  Using the test video I posted at:

http://www.pirobot.org/videos/optical-flow/test6.avi

I can easily isolate the moving hand even when the camera is panning by using the Movement module followed by the Population Threshold module.  I've attached the .robo file I found to work.

STeven, I'm wondering if the Eliminate Global Flow option in the Optical Flow Module is working correctly.  When I toggle this check box on and off, I see the effect in the overall movement vector but not in the block intensities.  In contrast, the Reduce global movement check box in the Movement module has a clear effect on pixel intensities when the display is set to Grayscale.

--patrick

program.robo
Anonymous from United Kingdom  [99 posts] 7 years
Great Solution.  I have had good luck with the movement module for tracking moving objects in the frame. There might be something to this idea of using the movement module to determine the global flow and then subtract that from the flow vectors.  I really want to do optical flow balancing for obstacle avoidance.  This means I really need to correct the optical flow vectors for the global motion.  I still haven't figured out any answers for this.   Unfortunately, I am currently distracted with trying to get gcc to compile using theNEON fpu on the ARM8 board I am using for the embedded vision processing.
from United States  [214 posts] 7 years
Hi Prof Mason,

You mentioned you are using AX-12+ Dynamixels as am I.  I was wondering if you experience the same problem I do which is driving me nuts.  I have 10 servos connected to a CM-5 which in turn is connected to my PC by way of a Serial-USB converter.  I power the CM-5 with the supplied 12V brick.  As I said in an earlier post, I am controlling the Dynamixels using the .NET library from Forest Moon Productions.  Everything works extremely well EXCEPT that at random intervals I get an "Unhandled I/O Timeout Exception" on the serial port connected to the CM-5.  I am polling the Dynamixels in Synchronized mode using a dedicated thread with 15 ms between each query.  When the timeout occurs, I have to power-cycle the CM-5 (which also power-cycles the servos) to regain control.  The time it takes before I get a timeout can be anywhere between 10 seconds and 20 minutes but it's a show stopper if trying to do a demonstration.

Other forums have suggested a cable problem but I have replaced and/or tweaked all the servo cables, the Serial-USB converter and even the CM-5.  I have also tried the USB2Dynamixel controller in RS-232 mode (the Forest Moon library does not support the direct TTL connection).

Any thoughts would be greatly appreciated!

--patrick
Anonymous from United Kingdom  [99 posts] 7 years
I am writing code to run directly on the CM5 using winAVR.  I started with the sample AX12 code from robotis and went from there.  I do have an RS232 protocol that I use to talk to the CM5 that I wrote.   I have not seen your exact problem but I have had problems (undiagnosed really) with synchronous mode timeouts.  Switching over to asynchronous seems to solve the problem on those particular robots.    It seems like some of the servos will randomly or due to thermal shutdown not reply within the specified time frame.    I don't think it is a CM5 problem.  (I have swapped CM5s on the same robot and it didn't fix it)    Disabling the AX-12 reply seems to fix the problem and swapping servos also seems to fix the problem.  (In the case of the humanoid, the knee servos often overheat and go into thermal shutdown which takes them off the bus and causes  the problems you describe)  The servos will start responding again on a power cycle.  

The robosavvy bioloid group is pretty active and might be a good place for extra help.  

Roborealm related bit.  Folks should try the new python script module!  This lets you do all the things you would have been doing in vbscript but in python which has numerous examples and is well documented on the web!  I think vbscript has always been frustrating and this is a GREAT alternative.  

good luck!
mmason
from United States  [214 posts] 7 years
Hi Prof Mason,

This is very helpful information!  I don't think I can use asynchronous mode since I get timeouts immediately as soon as I try that.  The problem is that I have more than one thread setting goal positions and movement speeds, so to keep them from clobbering each other I use the synchronized mode on the servos and send a sync write periodically in its own thread.  Also, I can't turn off the AX-12 reply since I am reading back position information from the servos.

Fortunately, I have the source code for the Forest Moon library and so I can now at least catch the timeout exception.  However, I haven't figured out how I can then recover the serial port without a power cycle.   I will also check out the robosavvy group.

Thanks again,
patrick
from United States  [214 posts] 7 years
Hi Prof Mason,

Thanks to your tip about the robosavvy group, I finally solved my long standing timeout issue regarding the AX-12's.   A group member there suggested I catch the timeout, flush the serial buffer, then resend the instruction.   It took me awhile to figure out where to do this in the Forest Moon Dynamixel source code but I finally made it work.  I can even unplug a cable and have the robot recover gracefully when I plug it back in.  In case anyone else is using the Forest Moon library, here is the modification I made to the ReadData routine:

protected byte[] ReadData(int id, Register startAddress, int count)
{
   List<byte> cmd = new List<byte>();
   byte[] returnPacket = null;
   bool retry = true;

   // the start address and count form the parameters for the command
   // packet
   cmd.Add((byte)startAddress);
   cmd.Add((byte)count);
   while (retry)
   {
       WriteInstruction(id, Instruction.ReadData, cmd);
       try
      {
         returnPacket = ReadPacket(id, count);
         retry = false;
      }
      catch (System.TimeoutException)
      {
          Console.WriteLine("TIME OUT accessing ID: " + id);
          Stream.Flush();
      }
   }
   return returnPacket;
}

--patrick

http://www.pirobot.org

This forum thread has been closed due to inactivity (more than 4 months) or number of replies (more than 50 messages). Please start a New Post and enter a new forum thread with the appropriate title.

 New Post   Forum Index