Robot Vision and Image Recognition

• Will Girtern - ajaxdude
• Matt Buhler - 'Ferris'

“Vision is our most powerful sense…[and] provides us with an enormous amount of information.”1 With this in mind, the objective of this project is to integrate machine vision into a mobile robot, in order to achieve autonomy. In particular, the ultimate goal is to develop a low-cost robot that will recognize a particular human in front of itself, and intelligently follow this person, assisting him or her as needed.

Scroll down to the 'Behind the Scenes' section to get a better look at the various design stages of our project!

This project is comprised of six main components:

• Platform - a Lynxmotion 'Terminator' robot platform
• Motor Controller - RobotPower 'Scorpion XL'
• Pan Tilt Mechanism - controlled by Wii Nunchuck
• Network Camera w/ Processing Unit - Surveyor 'Blackfin' Camera
• Central Processing Unit - Arduino Duemilanove
• Image Processing Software

In order to manage the short five weeks of the 2010 Winter Session, this project has been divided into four logical phases:

1. Phase I - Platform:
   • Pan Tilt controlled with Wii Nunchuck
   • Robot Platform - Lynxmotion Sumo Platform + Scorpian Motor Controller
2. Phase II - Network Camera Streaming Live Video
   • D-Link DCS 900 Network camera
   • CMUcam3 Robot Vision System / Sensor
3. Phase III - Code Development
   • Image filters
   • Image algorithms
   • Navigation algorithms from image info.
4. Phase IV - Integration
   • Combine platform + camera to track familiarized human

The Arduino Duemilanove is a very powerful and very open-source development board, featuring Atmel's Atmega 328 micro-controller at its heart. I chose the Arduino based on its robust, flexible, and overall easy-to-use hardware and software. From receiving inputs from a variety of sensors to controlling lights, motors, and other actuators, the Arduino can do just about anything. Projects can be stand-alone or they can communicate with software running on a separate computer - the basis for our project. The Arduino will be responsible for controlling the Pan Tilt servo-motors, as well as passing up-to-date robot trajectories to the motor controller.

The 'Scorpion XL' is a great motor controller to provide the necessary current for the four 12vdc motors of our robot. Due to the scarce documentation, i.e. User's Manual, for the 'Scorpion XL' I've decided to post a tutorial on how to incorporate this motor controller into your future robot project. The tutorial can be accessed by clicking here: Click here.

One of the more interesting components of this project is the Pan Tilt mechanism. Currently, the servo-motors of the Pan Tilt are directed by the 3-axis ADXL330 accelerometer found in a Wii Nunchuck remote. Hopefully, and with enough funds, a XBee wireless communication module will free up the wire-tether between the Nunchuck and the Arduino Board. Provided there is enough time, we hope to use other controls on the Wii Nunchuck to allow the user to guide the robot wirelessly as well.

• Here is a snippet of our main function library. I wrote these functions in order to deliver basic trajectory movement for the robot, as well as the ability to accept infrared-range readings:

 void goForward() {
    for (int thisCount=1; thisCount < 80; thisCount++) {
    myservo_left.write(50);    //move forward
    myservo_right.write(50);
    delay(20);        //wait 20 milliSecs
    }
 }

 void goBackward() {
    for (int thisCount=1; thisCount < 80; thisCount++) {
       myservo_left.write(130);    //move forward
       myservo_right.write(130);
       delay(20);        //wait 20 milliSecs
     }
 }

 void turnLeft() {              //turn 90 degrees left
    for (int thisCount=1; thisCount < 80; thisCount++) {
       myservo_left.write(120);    //move forward
       myservo_right.write(60);
       delay(20);        //wait 20 milliSecs
    }
 }

 void turnRight() {             //turn 90 degrees right
    for (int thisCount=1; thisCount < 80; thisCount++) {
       myservo_left.write(60);    //move forward
       myservo_right.write(120);
       delay(20);        //wait 20 milliSecs
    }
 }

 void scrapeIR() {
    IRValue = analogRead(0);          //store analog input on pin 0:
    Serial.println(IRValue, DEC);  //send to monitor as ASCII-encoded decimal  
    delay(200);                  //wait 10 milliSecs 'til next reading
 }

void exploreRight() {           //look 90 degrees right
 for (int thisCount=1; thisCount < 30; thisCount++) {
    pan(120);                   //move camera right
    tilt(120);                  //look up at 30 degrees
    myservo_left.write(60);     //move right
    myservo_right.write(120);
    delay(20);                  //wait 20 milliSecs
 }
 pan(90);
 tilt(90);
 myservo_left.write(90);        //stop
 myservo_right.write(90);
}

void exploreLeft() {           //look 90 degrees left
 for (int thisCount=1; thisCount < 30; thisCount++) {
    pan(30);                   //turn camera left
    tilt(120);                 //look up at 30 degrees
    myservo_left.write(120);   //move forward
    myservo_right.write(60);
    delay(20);                 //wait 20 milliSecs
 }
 pan(90);
 tilt(90);
 myservo_left.write(90);       //stop
 myservo_right.write(90);
}

The network camera that we ordered for the winter session was out of stock. Therefore, to press on, we hacked a 'Blackfin Camera' produced by 'The Surveyor Corporation', which was used in conjunction with their SRV-1 mobile robot. The Blackfin uses RoboRealm Machine Vision Software to process and manipulate captured image frames and then uses this information to 'enlighten' the robot about its surroundings, i.e. Turn left NOW!

The real heart of this project is the image-processing and image-recognition element of the robot. With 'RoboRealm' I was able to apply several pre-defined filters to extract only the specific pixels of the image-stream that I wanted the robot to 'recognize'. To begin, I drew a bright red circle on a piece of paper and wrote a script for the robot to detect only the red circle:

Above is the image after the red-color filter has been applied. Notice I was even able to determine where in the frame the red circle appeared with 'RoboRealm'!

The 'pipeline' is the combination of image filters and scripts sent to the 'Blackfin' processor in order to 'recognize' and track an individual. Here is the completer order of the pipeline for the project:

RGB_Filter Red: Find what's red in the image
Erode 1: Remove some of the surrounding noise
Blob Filter: Remove all blobs that are too small and not circular enough
COG: Determine position of red circle in frame
VBScript: Tell 'Blackfin' how to get to red circle
Surveyor_SRV1b: Interface for the 'Blackfin'

After drawing several conclusions, I then turned to a trickier subject - the image recognition of a human. Below, the resulting image stream of the robot after it has been processed with a RBG Filter to extract only the red in my sweatshirt.

Here is the image stream again after the RBG filter has been momentarily paused. The image may appear blurry, but this is actually after a Gaussian Blur Filter has been applied to soften sharp spikes in the pixel contrasts, ultimately removing extraneous noise as well. In fact, the Gaussian Blur happens to be a detrimental component in the human's image perception.

Now that I was able to 'recognize' a particular human with the robot's image processing unit, I had to determine how to move the robot to the object. 'RoboRealm' provides a module called the Center of Gravity (COG), which will encapsulate an specified object with a box. The module also keeps track of the size and x- and y-coordinates as the object moves within the screen. Therefore, if the object moved left, I needed to pass a digital signal to my MCU which would inform the motor-controller to turn the robot left.

Here is a snippet from the VBScript being passed to the 'Blackfin' processor:

'Global vars
speed = 60
width = GetVariable("IMAGE_WIDTH")
halfWidth = width / 2
cogArray = GetArrayVariable("BLOBS")

if isArray(cogArray) then
  if ubound(cogArray) >= 1 then
    cogx = cogArray(0)

  if cogx < halfWidth then
    'Turn left
    lmotor = 128 + ((speed*2)*(cogx/halfWidth)) - speed
    rmotor = 128 + speed
  else
    'Turn right
    lmotor = 128 + speed
    rmotor = 128 + ((speed*2)*((width-cogx)/halfWidth)) - speed
  end if

  'Set motor speed
  SetVariable "left_motor", lmotor
  SetVariable "right_motor", rmotor
  else

  'Stop if no ball seen
  SetVariable "left_motor", 128
  SetVariable "right_motor", 128

  end if
end if

Unfortunately, 'RoboRealm' only interfaces with 6 general purpose input/output pins (GPIO) on the Blackfin camera. This means that at any point in time I am limited to sending a whopping 6-bits of info to the MCU regarding an image - thats a number ranging from 0-63! We were forced to narrow the scope of the image processing to only the x-plane(floor of the environment). In other words, it was most logical to tell the bot to turn left or right when the object moves rather than up or down. The images that we were streaming from the robot camera where approximately 320 pixels wide. Therefore, we scaled the 320 pixels down to a number ranging from 0-63 - thereby meeting the 6-bit threshold.

To Illustrate this tactic, below is a diagram dividing the video frame into three main sections. Basically, if 6-bit binary number being passed to the Arduino falls outside of a 'Tolerance Range' the robot will move left or right:

Here is the revised VBScript:

'Global vars to hold image dims
width = GetVariable("IMAGE_WIDTH") 
cogArray = GetArrayVariable("BLOBS") 

if isArray(cogArray) then  
  if ubound(cogArray) >= 1 then          
    'Scale 320 pixels to only 6-bits
    cogx = (cogArray(0)*63)/width 
   
    'Pass digital info to MCU
    SetVariable "cog_x_1", cogx & 1 
    SetVariable "cog_x_2", cogx & 2 
    SetVariable "cog_x_3", cogx & 4 
    SetVariable "cog_x_4", cogx & 8 
    SetVariable "cog_x_5", cogx & 16 
    SetVariable "cog_x_6", cogx & 32 
  end if 
end if 

For an in-depth look into the numerous design processes and the evolution of this project, follow this link: Click here.

The application of this robot could be directly tied into the military; the robot will follow a foot soldier into battle and assist this soldier with war fighting needs. In addition, the scope of this project could expedite the solution for an autonomous cave-mapping robot, particularly with the combination of a laser rangefinder.

1 “Introduction to Autonomous Mobile Robots”. Roland Siegwart and Illah R. Nourbakhsh. Page 117.