Lejos Vision System

Overview
This a a Java Vision System for the Lego MindStorms Robotics Invention System. It provides motion detection, color detection, light detection and interfacing with robots that use the Lego Mindstorms RCX brick, to allow Mindstorms robots to respond to what they see. It supports most of the functions of the Lego Vision Command software, but in an open extensible way that will allow much more sophisticated functionality to be added. Some of the ideas come from the Robotics SDK produced by the UK Sun Technology Evangelist, Simon Ritter - see Image Capture From Webcams using the Java[tm] Media Framework API and Java[tm] Technology Robotics Developers Kit. Some of the code and ideas come from the projects of Konrad Rzeszutek - see Konrad Rzeszutek's Projects Although it is designed for use for the Lego Cam that comes with the Lego Mindstorms Vision Command product, it will work with most PC Web cameras. I have tested it with the X10 XCam2 wireless camera. This camera can be mounted on a Lego Robot to avoid the trailing USB wire, which is a problem with the Lego Cam.

Functions supported
Functions supported include:

A moving video display on your PC of what the robot sees Definition of regions in the field of view Flipping the image horizontally to fit the type of example best Detecting motion Detection of specific colors Light detection Taking snapshots

Recording videos, including sound Playing sound files Sending commands to the RCX GUI control of color and motion sensitivity, region display, etc. Extensibility of the GUI to allow remote control of the robot, etc. The remote control class can also support speech control of the robot

Regions
All recognition occurs within a region. Currently only rectangular regions are supported. Regions are numbered from 1, and are their outlines and numbers are overlayed on the moving video display.

Listeners
Three types of listeners are currently supported:

Motion Listeners - detect motion in a region Color Listeners - detect a specific color in a region Light lsteners - detect bright light in a region

There are three interfaces corresponding to these: MotionListener, ColorListener, and LightListener.

Responding to what is seen

The various types of listener can respond to what they see in specific regions. Examples of possible responses, supported by the API, are:

Taking a snapshot, e.g. of an Intruder Recoding a video of what is seen Playing sound effects or musical instruments on the PC Sending commands to the RCX to move the camera, turn towards light, avoid motion etc.

Remote Command Execution

Sending remote commands to the PC is done by the josx.rcxcomm remote method execution classes. Client stubs and a server implementation of functions needed for camera and robot movement are provided as part of the vision API.

Example

An example program that detects motion, sounds an alarm and takes a snapshot of an Intruder:
import josx.vision.*; public class Motion implements MotionListener { private static int image = 1; long lastPlay = 0; public static void main(String [] args) { (new Motion()).run(); } private void run() { Vision.setImageSize(320, 240); Vision.addRectRegion(1, 0, 0, 320, 240); Vision.addMotionListener(1, this); Vision.startViewer("Intruder Detector"); } public void motionDetected(int region) { if ((System.currentTimeMillis() - lastPlay) > 1000) { lastPlay = System.currentTimeMillis(); Vision.snapshot("Intruder" + image++ + ".jpg"); Vision.playSound("../../Effects/Alarm.wav"); } } }

Download
The java classes, source and documentation are provided in a jar, and two zip files: vision.jar contains the josx.vision package, visionsrc.zip contains the Java source code, and visiondocs.zip contains the html api documentation

Installation
Step 1 - Download and unzip
Download vision.jar, e.g to c:\vision\lib and put it on the CLASSPATH. Download visionsrc.zip and unzip into a directory such as c:\vision. Download the API documentation and unzip into its own directory, e.g c:\vision\apidocs

Step 2 - Download JMF

This is available from JMF It is best that your camera is plugged in when you install JMF, as this will ensure that the capture devices for it are put in the JMF registry.

Ensure that JMF is on the CLASSPATH by following the installation instructions.

Step 3 - Determine camera and sound devices

Run the JMStudio application to determine what video and sound capture devices are available on your system. Look at properties to determine the video and sound capture devices. If the camera was not on when JMF was installed try doing "Detect capture devices". If this does not work, try calling jmfinit from the JMF bin directory.

Step 4 - Edit Video Properties files

Edit the video.properties files in c:\vision\examples\motion etc. There is currently a version in each example directory and you need to edit each one to run the example. You need to set video-device-name and sound-device-name to the exact text in the JMF registry. If Direct Sound capture is available on your windows machine, use that, as Java Sound capture does not seem to work.

Step 5 - Copy Lego Vision Command Sound files

Copy the Effects directory from the Lego Vision Command VCSData and Piano and any other instrument directory you want from the VCSData\Instruments directories. For a default installation of the Lego Vision Command software, the full path for VCSData is: c:\Program Files\LEGO MINDSTORMS\Vision Command\data\hdData\VCSData They should be copied to subdirectories of c:\vision, e.g. c:\vision\Efects and c:\vision\Piano.

Running the Examples

Motion example
Change directory to c:\vision\examples\motion and do: javac Motion.java java Motion An alarm is sounded and a file Intrudern.jpg is produced whenever motion is detected.

Alarm example

Change directory to c:\vision\examples\alarm and do: javac Alarm.java java Alarm This is a security alarm that can be turned off by the correct color being detected in region 3. You will proibably need to edit the source to change the color to one that is detected by your camera. The current average RGB value is what is being detected in region 3 is displayed on System.out.

Music example
Change directory to c:\vision\examples\music and do: javac Music.java java Music Piano You can use other instruments that are available with the Lego Vision Command software, e.g. "Guitar2"

Light Seeking example

This example uses the RCX remote control class to respond to what the camera sees. It is a very simple light seeking example. When it sees light in the central region it moves forward. When it sees light in the left or right regions, it turns towards the light. This example needs the Rover example program running on the RCX. Change directory to c:\vision\examples\light\rcx and do: lejosc Rover.java lejos Rover and press the Run button on the RCX. You then need to compile and run the PC program. To do this, change directory to c:\vision\examples\light\pc and do: javac Light.java java Light Shine a bright light at the camera to move the robot about.

Remote control example

This example uses the RCX remote control class to control the robot and let the robot respond to what it sees. It needs the Rover example program running on the RCX. Change directory to c:\vision\examples\light\rcx and do: lejosc Rover.java lejos Rover and press the Run button on the RCX. You then need to compile and run the PC program. To do this, change directory to c:\vision\examples\remote\pc and do: javac *.java java RemoteControl You should see a display with the image viewer in the center, remote controls for the robot on the left, and camera and color controls of the right. Move the robot about with the remote controls. Flip the camera up and down, as necessary, to see what you need to see. When the robot gets near a red object, it will move towards it and attempt to push it forwards. Click on the little Media Properties control at the bottom right of the ImageViewer and choose the Plug-In Settings, to control color sensitivity and the proportion of the region that needs to be filled with the specified color. Edit the RemoteControl.java program and change the color if you need to.

Speech control example

See Speech Control example. Have fun.