Real-Time Computer Vision Processing on a Wearable System

Kevin DeMarco kevin.demarco@gtri.gatech.edu CS 6235 Real-Time Systems August 28, 2012

Contents
1 The Problem and Its Importance 2 Technical Approach 2.1 Central Processing Unit . . . . . . . . . . . . 2.1.1 Linux Distribution Development . . . 2.1.2 Managed Package Installation Process 2.2 Software Architecture . . . . . . . . . . . . . 2.2.1 Thread Task Designation . . . . . . . 2.2.2 TCP Commands . . . . . . . . . . . . 2.2.3 TCP Field Processing . . . . . . . . . 2.2.4 System Heartbeat . . . . . . . . . . . 2.2.5 Thread Monitor . . . . . . . . . . . . 2.3 Android Smartphone Application . . . . . . . 2.3.1 Android Graphical User Interface . . . 2.3.2 Java Socket Communication . . . . . . 2.3.3 Android SDK Lessons Learned . . . . 2.4 Field Programmable Gate Array (FPGA) . . 2.4.1 FPGA System Architecture . . . . . . 2.5 Network Conguration . . . . . . . . . . . . . 2.6 Motor Controller . . . . . . . . . . . . . . . . 2.7 System Integration . . . . . . . . . . . . . . . 3 Image Processing 4 System Evaluation 4.1 Network Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Image Processing Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Conclusions 5.1 Future Development . . . . . . . 5.1.1 Simultaneous Localization 5.1.2 Facial Recognition . . . . 5.1.3 Computer Simulation . . 2 2 2 3 3 4 5 6 7 7 7 8 8 9 9 9 10 10 11 11 11 13 13 13 14 14 14 14 14 14 14 14

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . and Mapping (SLAM) . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

6 Resource Used 6.1 Hardware Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Software Resources Utilized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Real-Time Computer Vision Processing on a Wearable System

Kevin DeMarco CS-6235 : Real-Time Systems Project Proposal August 28, 2012

Objective
The Real-Time Computer Vision Processing on a Wearable System project involved the development of a wearable system that is capable of processing visual imagery from a camera in real-time and providing the user with actionable information based on features detected in the environment. The system was tested with a standard USB camera, but one of the objectives of this project was the development of a framework that will allow for additional sensors to be integrated into the existing wearable system.

The Problem and Its Importance

Ubiquitous computing has become a reality in modern society. Almost all individuals carry a cell phone for communication and many carry a smartphone capable of connecting to the internet and providing important information to the user. While smartphones are great for providing the user information on demand, wearable systems can provide the user with context-sensitive information even before the user requests the data. A wearable system, which is usually composed of sensors, cameras, or a head-mounted display, is continuously monitoring the environment and assessing the current scenario. If the wearable system determines that the user should be made aware of an event or object in the environment, the wearable system will provide the user with information in the form of video, sound, or physical feedback. There are a number of practical applications for wearable systems ranging from soldiers using wearable systems to detect threats in the environment to helping the blind navigate unknown locations. The system discussed in this paper was developed to assist the blind in navigating foreign environments.

Technical Approach

The design of the wearable system consisted of both hardware and software development. Figure 1 shows an overview of the wearable system and the main components that comprise the system. The wearable system is composed of several hardware components: a central processing unit (CPU), a eld programmable gate array (FPGA), sensors, haptic feedback actuators, an Android smartphone, and power supplies. Software was written for the CPU, the FPGA, and the Android smartphone in order to process sensor data, control haptic feedback actuators, and allow for communication between the devices through Ethernet sockets.

2.1

Central Processing Unit

At the heart of the wearable system is the Gumstix Overo Air computer-on-module (COM) [1]. The Overo Air is an ARM Cortex-A8 OMAP3503 embedded system that has the capability to communicate via Bluetooth and the 802.11g wireless networking protocols. When paired with the Tobi expansion board, the Overo Air

Figure 1: Wearable System Architecture also has hardware available to communicate over hardwire Ethernet, control UARTs, control an SPI bus, convert analog voltages to digital values, control USB devices, and display video over the HDMI interface. 2.1.1 Linux Distribution Development

While the Gumstix is a robust hardware platform and capable of running desktop versions of Linux without serious performance issues, compilation on the embedded target can be slow and tedious. For this project, several complex C++ libraries were required and having to compile each of these libraries from source on the Gumstix would be dicult due to the large number of dependencies for each of the libraries. In order to overcome this, the Open Embedded cross compilation environment was utilized. The Open Embedded tool chain is a collection of Bitbake recipes that assists in package cross compilation by keeping a list of URLs where package source code can be compiled and a list of the dependencies for each package[TODO:ref]. Bitbake is the Python program that processes the build scripts[TODO:ref]. While the rst build usually takes more than 12 hours in order to generate the Linux root le system image and kernel, building packages after the initial system build are on the order of minutes for complex packages. For this project, the initial image was built with the command:
Listing 1: Bitbake Build Example bitbake omap3-console-image

The rst build was attempted on a 2010 Macbook Pro with a native version of Linux installed in an 80 GB partition. Unfortunately, the Open Embedded build consumed more than 65 GBs of hard drive memory, resulting in the build crashing. Thus, the build was performed on the authors personal Linux server with 500 GBs of hard drive space, which resulted in a successful build. 2.1.2 Managed Package Installation Process

After a successful build, the omap3-console-image was written to a properly formatted Secure Digital (SD) card along with its corresponding bootloader les: MLO, u-boot.bin and uImage. Once the SD card was inserted into the Gumstix and the Linux distribution booted up correctly, additional packages could be installed by SCP-ing the appropriate packages *.ipk le and running the opkg command. Unfortunately, a major drawback with this manual method of installation is that package dependencies cannot be installed automatically for a given *.ipk le. In order to overcome this issue, the opkg installation on the Gumstix was congured to look for installation packages on the Linux desktop that originally built the

Linux image and all necessary packages. Due to the dynamic IP address assignment by the Internet service provider (ISP), the NETGEAR router for the home network was congured with Dynamic DNS such that the address demarco.dyndns.org pointed to the NETGEAR router. The Apache web server was also installed on the Linux desktop and a symbolic link between the built *.ipk packages and the web servers root directory was created to give access to the packages to the Gumstix over the Internet. Finally, the le /etc/opkg/demarco-base.conf was created with the following contents:
Listing 2: Custom Package Repository src/gz demarco-base http://demarco.dyndns.org/ipk/armv7a src/gz demarco-all http://demarco.dyndns.org/ipk/all src/gz demarco-overo http://demarco.dyndns.org/ipk/overo

Thus, in order to install the emacs package to the Gumstix, the following commands were executed on the Gumstix over ssh.
Listing 3: Emacs Package Installation $ opkg update $ opkg install emacs

The opkg command handled the downloading and installation of dependencies for emacs and since the server utilized Dynamic DNS, the installation of packages worked both within the home network and on external networks.

2.2

Software Architecture

The Gumstix COM was the at the heart of the Wearable system, thus, it had to be able to perform a number of tasks in parallel. The following is a list of tasks the Gumstix software had to manage: Connect to TCP server as a client Act as a TCP server for other clients Parse and execute TCP commands from client Read images from USB 2.0 camera Execute computer vision algorithms on images Monitor message queue buers and thread statuses Generate system heartbeat UDP message Obviously, a single-threaded program would not suce. Instead, the program was broken into multiple threads using the Linux pthread library. In order to improve the portability of the C++ code, the pthread library function calls were wrapped in a custom class called ThreadConfig, which handled the creation, joining, and killing of threads. Thus, if the code is moved to a platform that does not support the pthread library, only the few calls to the pthread library in the ThreadControl class will have to be modied. The instances of the ThreadControl class were maintained with the C++ STL std::vector container, which allowed for the thread monitor function to easily iterate through all of the threads and check their statuses. The threads were created and executed with the following code:
Listing 4: Thread Creation std::vector<ThreadConfig *> threads; threads.push_back(new ThreadConfig(TcpServerThread,&TCPServer_Thread,30,(void*)&threadArgs)); threads.push_back(new ThreadConfig(TcpClientThread,&TCPClient_Thread,30,(void*)&threadArgs)); ... threads.push_back(new ThreadConfig(HeartThread,&Heart_Thread,30,(void*)&threadArgs));

Similar to the thread class, a message queue class was created to wrap the POSIX message queue function calls. The POSIX message queue functions (msgctl(), msgget(), msgsnd(), msgrcv(), etc.) are basic system calls within Linux, thus, an additional message queue library was not linked into the nal executable. For each message queue, the QueueConfig class was instantiated within the std::vector container.
Listing 5: Queue Creation std::vector<QueueConfig *> queues; queues.push_back(new QueueConfig(MotorControlQueue, std::string("../main.cpp"), B)); ... queues.push_back(new QueueConfig(TCPCmdQueue, std::string("../main.cpp"), C));

Both the queues and the threads can be iterated in a similar fashion. A common function that was used to search for a specic Queue was the getQueue() function.
Listing 6: Queue Iteration QueueConfig* getQueue(std::vector<QueueConfig*> *queues, QueueName name) { std::vector<QueueConfig*>::iterator it; for ( it=queues->begin() ; it < queues->end(); it++ ) { if( (*it)->getName() == name ) { return *it; } } return (QueueConfig*)NULL; }

The ThreadControl thread utilized this function to display the status of a specic queue during the debug process to check for buer overows. 2.2.1 Thread Task Designation

The wearable system was divided into six threads: 1. TCP Server and Command Processor 2. TCP Client 3. UDP Multicast Broadcaster 4. System Heartbeat Generator 5. USB Camera Reader and Obstacle Detector 6. Thread and Message Queue Monitor The ow of the data between the Threads is shown in Figure 2. The wearable system has three socket connections to external devices: the WiFi / Android smartphone connection, the UDP multicast / Network connection, and the TCP client / Xilinx FPGA connection. Most of the system bandwidth is consumed by the Computer Vision Processing thread, which is constantly reading frames from the USB camera and executing one of several feature extraction algorithms on each frame. When the Computer Vision Processing thread determines that it has detected an obstacle that the user needs to notied about, the thread sends a message to the Motor Controller thread via the Motor Control message queue. The message that is exchanged is a simple protocol that simply denes whether the user should be instructed to move left or right and the urgency of the notication. A higher urgency will result in the motor being driven at a higher rate. The Motor Controller thread then passes on the message to the TCP client, which writes the appropriate data to the TCP socket connection with the FPGA.

Figure 2: Wearable Software Architecture Command SET GET START STOP BIT Arguments <Field>=<Value> <Field> N/A N/A N/A Description Assigns a value to the designated eld Retrieves the value of the designated eld Enables processing of camera frames and control of motors Disables processing of camera frames and control of motors Performs system Built-in-Test (BIT) Table 1: TCP Commands 2.2.2 TCP Commands

The main purpose of the TCP server and Command Process thread is to modify and monitor system variables at runtime. The TCP server processes commands dened in Table 1. With the dened commands and elds, the user or other software is able to congure the wearable system at runtime and modify its behavior. Listing 7 shows an example TCP session with the TCP server where the TCP client requests the available elds and then sets the HEARTBEAT PERIOD eld to a value of 10 seconds. The user then initiates the BIT command followed by the START command to enable the processing on the system. The client commands are shown in lowercase, which is acceptable since the TCP command processor is case-insensitive.
Listing 7: TCP Command Session get 101 101 101 101 101 101 000 NAME="WEARABLE" HEARTBEAT_PERIOD=0x05 VERSION_CSCI="1.0" VISION_MODE=0 VISION_THRESH=0 RUN_MODE="STOPPED" WEARABLE Tue Dec 6 07:39:10 2011

set heartbeat_period=0x0a 101 HEARTBEAT_PERIOD=0x0a

000 WEARABLE Tue Dec

6 07:39:15 201

bit 100 BIT <RUNNING BIT> 000 WEARABLE Tue Dec 6 07:39:23 201 start 100 START 000 WEARABLE Tue Dec

6 07:39:30 201

2.2.3

TCP Field Processing

The author has used this TCP structure in the past and often became frustrated with the tediousness and error-proneness of adding additional elds to the eld list. In order for alleviate these issues, a Field class was created that contains the basic elements that each eld requires. The base-class, Field holds the name of the eld as a std::string and the write access mode of the eld as an enumeration (ReadOnly, WriteOnly, ReadWrite). Additional functionality is added by creating subclasses of the Field class and adding class methods such as setValue(), setStringValue(), getStringValue(), etc. Other classes that were created included the Field Number class, the Field String class, and the Field Time class. Each class overloaded the appropriate class methods in order to provide the appropriate output when a SET or GET command was issued. One of the main reasons that all of the elds inherited from the same base class was so that all of the dierent subclasses could be pushed onto the same C++ STL std::vector as is shown in Listing 8.
Listing 8: Field List Creation std::vector<Field *> fields; fields.push_back(new fields.push_back(new fields.push_back(new fields.push_back(new fields.push_back(new fields.push_back(new Field_String(NAME_FIELD,ReadOnly,"WEARABLE")); Field_Number(HEARTBEAT_PERIOD_FIELD,ReadWrite,0x0A,2, Hex)); Field_String(VERSION_CSCI_FIELD,ReadOnly,"1.0")); Field_Number(VISION_MODE_FIELD,ReadWrite, GoodFeatures, 2, Dec)); Field_Number(VISION_THRESH_FIELD,ReadWrite, 0, 2, Dec)); Field_Run_Mode(RUN_MODE_FIELD, ReadWrite, Stopped));

This method of pushing elds onto a std::vector makes it very easy for the software engineer to add and remove elds from the eld list. Then all the software engineer has to do is overload the appropriate functions in the specic elds class. 2.2.4 System Heartbeat

When working with wireless systems, it is often dicult to determine if a particular device within a system has crashed due to a failure or if the network is malfunctioning. In an attempt to alleviate this problem, a system heartbeat message is generated and multicasted to the network from the Gumstix. The heartbeat message is a small UDP packet with the character sequence Heartbeat and a message counter that designates the number of heartbeat messages that have been generated since system initialization. Any system that is connected to the wireless network can subscribe to the multicast IP address and monitor the message count in the heartbeat message to determine if packets are being dropped or if the system is resetting. 2.2.5 Thread Monitor

When ne-tuning the thread priorities it is often useful for the system designer to know the sizes of the various message queue buers throughout the system. Also, the software engineer may want to know the particular state of a thread, whether the thread is pending, blocked, running, or if it has crashed. The Thread Monitor thread allows the software engineer access to this information by monitoring the threads in the ThreadConfig std::vector and the message queues in the QueueConfig std::vector. The monitor thread can be queried through the TCP server on the Gumstix to return thread and queue information or it can be used to kill and restart specic threads. 7

2.3

Android Smartphone Application

While the wearable system was designed with the visually-impaired in mind, access to the Gumstix through a graphical user interface (GUI) was essential for system debugging and control during development. In order to control and monitor system variables within the Gumstix and attached FPGA, an Android application was written using the Android SDK. The Android application was developed with the following capabilities in mind: Connect to the TCP server on Gumstix via WiFi Monitor heartbeat UDP messages from Gumstix via WiFi Instruct the Gumstix to execute its Built-in-Test Issue GET and SET commands to the Gumstix Modify computer vision algorithms and parameters during runtime for system ne-tuning 2.3.1 Android Graphical User Interface

Designing and building the GUI for the Android application proved to be a fairly straightforward process thanks to the Android SDK for the Eclipse IDE. The application was basically a user input form consisting of TextView, EditText, Button, and SeekBar components that were dened in the main.xml le. The components are organized within horizontal or vertical layouts or within table structures in the XML le. The Android SDK then automatically generates the R.java source le that allows the user to link the XML le attributes to the applications source code. The results of the GUI development are shown in Figures 3(a) and 3(b).

(a) Wearable Android Application

(b) Vision Options

Figure 3: Wearable Android Application Screen Captures The user usually follows the following steps when using the Wearable Android application: 1. Enter Gumstix IP address and port number 2. Click on the Connect button to open a TCP socket to the Gumstix TCP server 8

3. Ensure that the radio button on the right side of the Disconnect button lights up signifying that the application made a successful TCP connection with the Gumstix TCP server 4. Click on the BIT button to test the complete data ow from the Android application to the motors 5. Click on the Start button to enable image processing and motor control At this point, the user can begin moving within his/her environment and the motors will indicate to the user the directions to take in order to avoid obstacles detected with the video camera. With the GET button the user can request the current status of the system variables. Also, the slider is used to adjust computer vision algorithm parameters. The possible computer vision algorithms that can be run on the Gumstix are shown in Figure 3(b). 2.3.2 Java Socket Communication

In the large text window in Figure 3(a), the response from a TCP command is shown. This response is similar in syntax to the TCP command structure discussed earlier because the Android application uses that command structure to communicate with the TCP server on the Gumstix. In Figure 3(a), the user has just nished sliding the slider at the bottom of the screen, which resulted in the VISION THRESH eld being set. Upon the successful SET-ting of the eld, the Gumstix responded with the new value of VISION Thresh. Since the Android SDK is based on Java, the TCP socket communication had to be implemented with the java.net.Socket class. A wrapper class called TCPClient was written to encapsulate the socket calls and provide a simple transmit() and receive() function interface to the main application. Thus, the transmittal and reception of TCP commands could be implemented with the following calls:
Listing 9: Java TCPClient Class TCPClient client = new TCPClient(hostname, port); client.connect(); client.transmit(txData, txData.length); client.receive(); System.out.println("Received: " + client.getRxBytes() + " bytes); System.out.println("Rx Msg: " + new String(client.getRxData()));

2.3.3

Android SDK Lessons Learned

The developers of the Android SDK were correct in making the SDK a true plugin for Eclipse. Too often, companies like Wind River, NetBurner, and other companies that require an IDE for their tool chains, merely fork o Eclipse for their own needs. However, this makes the user install multiple versions of Eclipse and it also limits updates that the forked versions of Eclipse can obtain from the mainline Eclipse open source project. Finally, the development, deployment, and debugging of the Android application was straightforward with the Android Debug (adb) program. However, the one problem that occurred was that when the TCPClient socket class was rst being tested, the socket kept returning a permission error. After doing some research on the internet, it was determined that the XML attribute:
Listing 10: Internet Permissions <uses-permission android:name="android.permission.INTERNET" />

had to be set in the AndroidManifest.xml le in order to grant Internet access to the Android application.

2.4

Field Programmable Gate Array (FPGA)

One of the main objectives of this project was to develop a robust architecture for a wearable system. Thus, while adding an FPGA to the wearable system for this project may have seemed to over complicate the design, the FPGA makes it possible to easily expand the systems capabilities in the future. Avnets Spartan-6 LX9 9

Microboard was chosen for the system because of its low price tag at $89 and because a MicroBlaze 32-bit microprocessor could be instantiated within the FPGA. The MicroBlaze processor is capable of running the lwIP lightweight IP stack, which gives the FPGA network connectivity. The user can also write custom rmware in VHDL or Verilog that attaches to the processor local bus of the MicroBlaze processor, which allows the wearable system to interface with almost any other sensor, actuator, or system as long as the mixed-signal circuitry is provided externally. However, one of the main drawbacks of FPGAs is large power consumption. 2.4.1 FPGA System Architecture

The system architecture for the FPGA is shown in Figure 4. The gure shows the main components of the FPGA system and how they components are attached to the processor local bus (PLB). Of particular interest Figure 4 are the microblaze 0, Ethernet MAC, SPI FLASH, and xps gpio 0 modules. The microblaze 0 module is the software 32-bit processor that runs C code and initializes the system through its PLB register interface. The microblaze 0 also processes TCP commands from the textttEthernet MAC to modify system variables at runtime. The SPI FLASH is used to store the boot loader, FPGA image, and main program code so that the FPGA does not have to be programmed with a JTAG probe at startup. The SPI FLASH also assists in the storing of persistent variables in ash memory. Finally, the xps gpio 0 is used to control the enable signal to the motor controller circuit. The typical ow of data through the MicroBlaze and FPGA is shown in Figure 5. Upon startup, the MicroBlaze processor is running a TCP server to which the Gumstix TCP client can connect and issue commands such as LEFT 200 and RIGHT 200, which enable the left and then the right motors, respectively, for 200 ms each. This is a fairly simple communication protocol and it will most likely be extended to the same protocol that the Gumstix TCP server uses.

Figure 4: FPGA System Architecture

2.5

Network Conguration

The organization of the wearable systems network conguration is worth mentioning for clarication. The Gumstix and the Spartan-6 FPGA communicate via an Ethernet cable on the 10.10.1.1 network. This 10

Figure 5: FPGA Data Flow is the communication channel that the Gumstix uses to enable the left and right motor controllers through the LEFT <duration> and RIGHT <duration> TCP commands. The Gumstix bridges the 10.10.1.1 network with the 10.11.1.1 network, which consists of the wireless router, Android smartphone, and Linux laptop. Both the Linux laptop and the Android smartphone can use their respective TCP clients to connect to the TCP server running on the Gumstix in order to modify and monitor Gumstix system variables.

Figure 6: Network Conguration

2.6

Motor Controller

The external circuitry that was used to drive the motors in the wearable system is shown in Figure 7. The circuit elements designated by V3 and V2 are the motors and are enabled when their respective NMOS circuit elements are turned on through the gate input. The PMOD1 and PMOD2 input pins are connected directly to the GPIO on the FPGA. At rst, the FPGA GPIO pins were connected directly to the gates of the NMOS circuits, but the FPGA output pins did not drive enough current to turn the NMOS elements on. Thus, the U1 and U2 Op Amps were inserted in between the FPGA and the NMOS elements to buer the GPIO signal. The entire circuit runs on a completely separate power supply from the Gumstix and FPGA, but the grounds are obviously tied together.

2.7

System Integration

Most of the system design went into the development of the software and electronics for the wearable system, but in the end, the individual components were placed in a camera bag with a shoulder strap. As shown in Figure 8(a), the video camera was attached to the camera bags shoulder strap, which proved to be a sturdy platform. The Gumstix, FPGA board, power supply, and associated cables were all placed inside the camera bag. The motors were attached to the shoulder strap of the bag as well as show in Figure 8(b).

Image Processing

The wearable system utilized the OpenCV C++ library to execute basic computer vision algorithms on decompressed images from the video camera. The OpenCV library is an open source program managed by Willow Garage that is unique in that it enables matrix like operations within C++. This helps bridge the gap between theoretical digital signal processing (DSP) and the implementation of the DSP algorithms on a 11

Figure 7: Motor Controller Circuit

(a) Video Camera Placement

(b) Motor Placement

Figure 8: Placement of Video Camera and Motor real system. For example, images are represented as matrices in OpenCV and the frame from a camera can be read with the following code:
Listing 11: Read Camera Frame cv::VideoCapture cap(1); // open the camera if(!cap.isOpened()) { // check if we succeeded cout << "Cant find camera" << endl; exit(-1); } cv::Mat origImage; // create matrix to hold camera image cap >> origImage; // get a new frame from camera

The Obstacle detection algorithm was implemented with the use of the cv::goodFeaturesToTrack() function and the cv::Canny() edge detector OpenCV functions. The outputs of the two algorithms are shown in Figures 9(a) and 9(b). Note how in Figure 9(a) the green circles are centered on the vacuum cleaner. This is part of the custom C++ code that was implemented that detected high priority obstacles, which then resulted in the issuing of motor commands to the FPGA to instruct the user to change direction.

12

(a) cv::goodFeaturesToTrack()

(b) cv::Canny()

Figure 9: OpenCV Image Processing

System Evaluation

The Gumstix platform was evaluated based on two metrics: network performance and image processing time. The Gumstixs performance was compared to the performance of a 32-bit Linux laptop with an Intel i5 processor and 4 GBs of RAM.

4.1

Network Performance

The network performance was tested with the use of iperf. The results are shown in Table 2. The Gumstix demonstrated surprising performance for an embedded system. Laptop 95.1 Mbits/sec 21.9 Mbits/sec Gumstix 94.5 Mbits/sec 19 Mbits/sec

Wired Wireless

Table 2: Network Performance Tests

4.2

Image Processing Performance

Unfortunately, the Gumstix did not measure up to the laptop in the image processing metric. Table 3 shows the number of s it took for each system to process a single frame from the video camera and execute the given computer vision algorithm. The results were averaged over 5000 frames. The Gumstix was comparable to the laptop on the Canny edge detector, but the Gumstix greatly suered when trying to run the Good Features detector. Good Features Canny Laptop 180686 151881 Gumstix 927002 297302

Table 3: Image Processing Time Tests

13

Conclusions

The wearable system development was a success. A system was developed that could process real-time video data and provide the user with actionable information. Also, the FPGA in the system will facilitate the addition of extra sensors during the next phase of the program. More work is denitely needed in the processing of video data, but sensor fusion will surely assist in future algorithms due to the fact that video imagery does not provide absolute distance to obstacles. Also, it is questionable whether any ARM-based system will be able to provide the image processing power that is needed for a real-time wearable system. Another option may be to put the image processing algorithms in the FPGAs hardware in order to alleviate processor workload.

5.1
5.1.1

Future Development
Simultaneous Localization and Mapping (SLAM)

SLAM has been researched heavily in the robotics community. Many of the SLAM techniques could be utilized to develop a wearable SLAM system. One of the biggest challenges will be establishing the cameras pose since it will be loosely attached to the user where in robotics platforms, the camera is rigidly attached to the platform. 5.1.2 Facial Recognition

The OpenCV computer vision package has tools within it to perform facial recognition. This could be a great way of helping the user know with who he is talking and for helping the user to remember people he meets in public. 5.1.3 Computer Simulation

Finally, a computer simulation of a visually-impaired individual navigating the environment could be developed. There are a number of simulators that exist for testing robotic SLAM algorithms. The Blender / Modular OpenRobots Simulation Engine (MORSE) simulator creates a 3D game environment that accurately models sensor noise and motion for the purpose of robotics simulation [2]. The developers of MORSE also created a model of a human being which could be used to test some of the algorithms discussed in this proposal.

6
6.1

Resource Used
Hardware Resources
Gumstix Overo Air COM with Tobi Expansion Board Avnet Spartan-6 LX9 Microboard Logitech Webcam C525 Android HTC EVO 4G Smartphone Netgear Wireless Router DC Motors

6.2

Software Resources Utilized

Open Embedded Cross-Compilation Environment OpenCV Android API 14

 Pthread Library POSIX Message Queues

References
[1] Gumstix small open source http://www.gumstix.com/ hardware, http://www.gumstix.com/. [Online]. [Online]. Available: Available:

[2] morse openrobots, http://www.openrobots.org/wiki/morse/. http://www.openrobots.org/wiki/morse/

15