Vision system simplifies robotic fruit picking - Vision Systems Design

Page 1 of 4
About Us | Advertise | Contact Us | Site Map

Subm

Search:

VSD Home
Subscribe:

Current Issue
eNewsletter

Product Center

Magazine

Webcasts

Articles

Resources

White Papers

Events

Industry Links

Video

Advanced Search

Featured Topics

CURRENT ISSUE

Current Issue
Issue Archive

Home > Issue Archive > Vision Systems Design > Article Display

Print this article

Email this article

Add RSS Feed

Vision system simplifies robotic fruit picking

By Winn Hardin
Robotic apple picker relies on a camera inside the gripper and off-the-shelf components and
software.
ADVERTISEMENT

By Winn Hardin, Contributing Editor

In the last few years, Belgian farmers have experienced problems with finding reliable seasonal
workers for harvesting choice fruit, including apples. In response to this problem, the Automation
Centre for Research and Education (ACRO) Institute developed a robotic apple harvester. ACRO is
a spin-off company and technical consultant to the Catholic University College of Limburg, Belgium,
and also provides PLC and machine-vision technical training and research to the European
community. ACRO is partially funded by the Belgian government.

FIGURE 1. Horizontal 2-D stabilization of the APFM consists of two hydraulic feet and one turnover
cylinder, configured as a three-point suspension. Controlled by two level sensors, the system ensures
stable positioning during picking. The white funnel-formed gripper unit is at the end of the robot arm.
The European standard height of fruit trees on professional plantations must be between two and
three meters. The ACRO automated fruit-picking machine (AFPM) harvester uses a unique
vacuum-gripper design to pick the fruit and ease coordination between the vision system and robot
controller. Mounted behind a common agriculture tractor, the AFPM platform supports a Panasonic
industrial robot to pick the fruit (see Fig. 1).

RECENT ARCHIVES

Global Network
Into the Light
Molten Steel Safety
Reaching Further
Technologies and tutorials
highlight NIWeek

http://www.vision-systems.com/display_article/303062/19/none/none/Feat/Vision-system-sim... 08-12-2008

Vision system simplifies robotic fruit picking - Vision Systems Design

Page 2 of 4
Interested in a subscription to Vision
Systems Design Magazine?
Subscribe to the magazine
Subscribe to the Features news
feed
Subscribe to the newsletter

FIGURE 2. ACROs use of Profibus as the network connection between the vision system, PLC, and
robot controller is due in part to ACROs position as a Profibus training center.
Click here to enlarge image

The AFPM apple harvester also consists of a tractor-driven generator for power supply, a horizontal
stabilization unit, a seventh external vertical axis to enlarge the operation range, a SICK safety
scanning device, a Siemens central control unit, and touch panel PC with human-machine interface
(HMI). MVTec Softwares Halcon image processing software provides the robot guidance
coordinates, while a canopy and curtain, which can be folded up during transportation, reduce the
affects of ambient light (see Fig. 2). The AFPM needs one driver on the tractor while it effectively
handles the workload of six workers.

CAMERA IN GRIPPER
One of the most challenging problems was the design of the fruit gripper. An entirely new end-ofarm tool (EOT) had to be developed to pick the apples with extreme precision and care. Not only
did this device have to assume the unique form of each apple, but the force used to pick the fruit
could not damage the fruit. ACROs engineers decided to create a vacuum-based gripper that could
also ease the vision systems task of guiding the Panasonic VR006L six-axis robot by concentrically
locating a digital camera inside the cone of the vacuum suction cup. With these design goals in
mind, the final gripper design is a combination of a white flexible silicone cone surrounding an IDS
UI-2230RE-C uEye USB 2.0 color camera. The cone is fed by a reversible vacuum/blower (see Fig.
3).

FIGURE 3. Vacuum gripper unit of the APFM has a USB 2.0 camera inside.
By placing the camera inside the gripper, the position of the camera is fully controllable because it
is fixed to the arm of six-axis robot. The camera can point its optical axis at the apple, reducing
image distortion and eliminating repetitive calibration steps during apple picking. A final advantage
is that the camera is protected against collisions or bad weather conditions, as well as against direct
sunlight.

FRUIT PICKING
Once the apple harvester is moved in front of the tree and its canopy opened around the AFPM
platform (see Fig. 4), the harvester has to be actively stabilized. For this purpose, the machine is
equipped with different hydraulic foot and elevator systems for front-behind and left-right
movements. These stabilizers are controlled by electronics bundled with the hydraulic system and
automatically controlled through the use of water level sensors.

FIGURE 4. The APFM picks apples in the orchard. The canopy covers an apple tree against intense
sunlight.
After the rig is adjusted, the camera scans the tree from 40 preprogrammed positions. In this
manner, the device acquires a grid of images of the front of the tree with the grid broken into
horizontal and vertical blocks created by panning the camera across in increments of 10, and
then panning it down by 10 and beginning the next grid row. Thus, each tree is divided into 40
sectors or images.
The programmed location and orientation of the robotic arm is stored with each image. Location
data is stored in the PCs RAM. For each sector, all ripe apples are identified by the image
processing software, listed, and picked one by one in a looped task.

http://www.vision-systems.com/display_article/303062/19/none/none/Feat/Vision-system-sim... 08-12-2008

Vision system simplifies robotic fruit picking - Vision Systems Design

Page 3 of 4

TARGETING FRESH FRUIT

To improve the efficacy of the image processing algorithms in finding and locating the apples on the
tree, the platform was designed to control the lighting conditions to the greatest extent possible,
using a canopy to cover the entire tree and the AFPM platform to reduce the effects of changing
ambient lighting conditions and provide a uniform background (blue) to ease locating the (red and
green) apples.
Image processing is conducted on an industrial PC with 2 GHz Pentium IV microprocessor and 1
Gbyte RAM running Windows XP. For image-processing software, the AFPM designers elected the
standard machine vision library Halcon 7.1 from MVTec because of its accuracy and reliability,
according to Eric Claesen, lead designer for the project at ACRO. The system can pick 85% of all
apples of a tree, similar to the achievements of manual picking operations.

FIGURE 5. After filtering out leaf, background, noise reduction, and filling holes in the images, the
apples positions clearly emerge. Iterative triangulation guides the robot to the apple for picking. The
gripper is designed to handle apples that vary from the target size by 10% without dropping the apple.
During system calibration, the first step is to train the system on the color of the apple trees leaves
and the blue background canopy. A series of color thresholding steps are used to filter out these
unwanted features. First the blue is filtered out of each of the 40 images that represent one side of
the tree, and then the leaves are filtered out. The green and the red parts of the apples surfaces
are located via color thresholding. After noise reduction, using Halcons opening_circle and
select_shape convolutions and filling holes in the images, the apples positions are clearly emerged
(see Fig. 5). After separating the individual apples from clusters (when present) by watershedfiltering, each apple is selected and transformed into a circle.

ITERATIVE GUIDANCE
To pick one apple, the robot has to determine the distance between the camera and the apple and
the path to get there. The camera measures this distance by triangulation. The measuring is done
in several steps. The camera first acquires an image, and then the camera is turned so that the
apple is situated in the center (see Fig. 6). The camera then acquires a second image, and finally
the diameter is calculated by processing these two images.

FIGURE 6. Distance between camera and apple is measured by triangulation, and robot guidance is
performed on iterative Stop and Look method based on extracting position data from periodic, static
images rather than continuous monitoring of real-time video of each apple common to visual servoing.
Click here to enlarge image

At this point, the vision system determines if the apple falls within the acceptable size range. If so, a
signal is sent across a Profibus connection to the robot controller, and the robot arm is allowed to
continue toward the apple.
While approaching the apple, several images are processed to calculate by triangulation the
remaining distance to the apple while air is blowing through the gripper at about 300 m/min to free
the fruit from leaves that might conceal it. With each image acquired, the system calculates the
remaining distance using a proprietary formula.
Since the apple remains centered in the image, the correlation of a given apple in subsequent
images is trivial. When the selected apple is approached, air blows through the picking device to
clear the apple from leaves. Once the apple is within a well-defined range of the gripper, the
vacuum device is activated. If a vacuum is detected, the apple is rotated and tilted softly, picked
and then put beside.

http://www.vision-systems.com/display_article/303062/19/none/none/Feat/Vision-system-sim... 08-12-2008

Vision system simplifies robotic fruit picking - Vision Systems Design

Page 4 of 4

Features, advantages, benefits

Says ACROs Eric Claesen, The goal, at the beginning-three years ago-of the project, was to
research if it was possible to pick an apple with the technology of today. Now at the end of the
project we can say that the project succeeded. We can detect and pick 80% of the apples in a tree
correctly. At the moment the picking machine picks an apple every 8 s but this will be reduced to 6
or 5 s after the pick season in 2007.
The designers believe that this time span can be lowered to about 5 s, chiefly by reducing the
communication bottleneck between the vision system and the central controller unit. The designers
are also planning to better the automation of the navigation through the orchard and to make the
device suitable for picking other fruit, such as pears.
Wed Aug 01 00:00:00 CDT 2007

Add RSS Feed

More Agosto, 2007 Articles >

Search Archives >

Search Products Buyer's Guide >

Search Industry Jobs >
Magazine & E-Newsletter Subscriptions >

Home | About Us | Contact Us | Site Map | Advertise

Copyright 2007: PennWell Corporation, Tulsa, OK; All Rights Reserved. | Privacy Policy | Terms & Conditions
SITE TOPICS: Infrared Imaging | Lighting & Optics | Smart Cameras | Gigabit Ethernet | Machine Vision Software | Articles | Products | Jobs
RELATED PENNWELL WEBSITES: Laser Focus World | Industrial Laser Solutions | Military & Aerospace Electronics | View All PennWell Sites | View All PennWell Events | PennWell Corporate
Site BioOptics World

http://www.vision-systems.com/display_article/303062/19/none/none/Feat/Vision-system-sim... 08-12-2008