The development of a mobile manipulator imaging system for

bridge crack inspection

Pi-Cheng Tung
*
, Yean-Ren Hwang, Ming-Chang Wu
Department of Mechanical Engineering, National Central University, 32054 Chung-Li, Taiwan
Accepted 15 February 2002
Abstract
A mobile manipulator imaging system is developed for the automation of bridge crack inspection. During bridge safety
inspections, an eyesight inspection is made for preliminary evaluation and screening before a more precise inspection. The
inspection for cracks is an important part of the preliminary evaluation. Currently, the inspectors must stand on the platform of a
bridge inspection vehicle or a temporarily erected scaffolding to examine the underside of a bridge. However, such a procedure
is risky. To help automate the bridge crack inspection process, we installed two CCD cameras and a four-axis manipulator
system on a mobile vehicle. The parallel cameras are used to detect cracks. The manipulator system is equipped with binocular
Charge Coupled Devices (CCD) for examining structures that may not be accessible to the eye. The system also reduces the
danger of accidents to the human inspectors. The manipulator system consists of four arms. Balance weights are placed at the
ends of Arms 2 and 4, respectively, to maintain the center of gravity during operation. Mechanically, Arms 2 and 4 can revolve
smoothly. Experiments indicated that the system could be useful for bridge crack inspections. D 2002 Elsevier Science B.V. All
rights reserved.
Keywords: Bridge crack inspection; Binocular image; Manipulator system
1. Introduction
A bridge is one of the most critical transportation
structures. Serious damage to a bridge due to aging, or
destruction arising from external forces, may adversely
affect a bridges structural safety. Therefore, overall
inspections and evaluations are essential to give a
thorough picture of the current condition of a bridge
to evaluate those which are necessary to carry out
maintenance or repairs to any damaged structural
components, ensuring the safety of the bridge.
Generally, bridge inspection consists of two steps:
a preliminary inspection and a detailed inspection.
The preliminary inspection is mainly performed by
people, and the results are used for a preliminary
evaluation of the bridges safety [1,2]. Inspection for
cracks is an important part of the preliminary inspec-
tion. A more detailed inspection, such as, for non-
fracture or fracture inspections, loading tests and
earthquake resistance evaluations, means of further
inspections with different kinds of instruments [3].
Therefore, in terms of the overall efficiency of the
bridge, maintenance on the eyesight inspection may
0926-5805/02/$ - see front matter D 2002 Elsevier Science B.V. All rights reserved.
PII: S0926- 5805( 02) 00012- 2
*
Corresponding author. Tel.: +886-3-426-7304; fax: +886-3-
425-4501.
E-mail address: t331166@ncu.edu.tw (P.-C. Tung).
www.elsevier.com/locate/autcon
Automation in Construction 11 (2002) 717729
discover damage to a bridges structure earlier, ena-
bling the problem and the extent of the damage to be
roughly estimated in advance. The information
obtained from an eyesight inspection can then be used
as a preliminary evaluation basis for screening before
further inspection with instruments is made.
There are some major advantages to the eyesight
inspection of a bridge, i.e. it is easy to do, it saves time
and costs, and it is efficient. Currently, the inspectors
must stand on the platform of a bridge inspection
vehicle or on a temporarily erected scaffolding to exa-
mine the structure underside of the bridge and the
portions above the water surface that cannot be seen
directly by the eye. Fig. 1 shows the inspectors stan-
ding on the platform of a bridge inspection vehicle.
Fig. 2 shows the inspectors standing on a temporary
scaffolding [4,5]. As there are so many bridges, how
to heighten inspection efficiency, while at the same
time protecting the safety of the inspectors becomes
an important issue. A robot system for the underwater
inspection of bridge piers has already been investi-
gated [6].
Inspection by means of the above-mentioned ins-
pection vehicle or temporary scaffolding may lead to
accidents involving the inspectors. To eliminate such a
danger, we developed a manipulator system, equip-
ped with binocular Charge Coupled Devices (CCD)
cameras. Two CCD cameras are installed on a two-
Fig. 1. Inspectors standing on the platform of a bridge inspection vehicle.
P.-C. Tung et al. / Automation in Construction 11 (2002) 717729 718
axis rotational frame laid on the front end of Arm 4
of the manipulator system. Binocular stereo images
are simultaneously captured by CCD cameras and
transmitted to the computer through a transmission
cable.
The CCD images, which contain physical noise,
need to be processed before crack positions can be
determined. Traditional pattern matching algorithms
[710] require a large memory and a long computa-
tion time. Furthermore, these methods are also sensi-
tive to image noise. To solve these problems, we
propose a new algorithm that can integrate the gray-
ness variation along the horizontal axis and thus
reduce the processing time.
Fig. 2. Inspectors standing on a temporary scaffolding.
P.-C. Tung et al. / Automation in Construction 11 (2002) 717729 719
Fig. 3. The coordination system of the parallel binocular CCD cameras.
Fig. 4. Images from the (a) left and (b) right cameras. Fig. 5. The (a) left and (b) right images after the Sobel operation.
P.-C. Tung et al. / Automation in Construction 11 (2002) 717729 720
The remainder of the paper is organized as follows:
in Section 2, we discuss the new binocular CCD
images comparison algorithm, and then obtain the
cracks position. The experimental results are dis-
cussed in Section 3 and a conclusion is given in
Section 4.
2. Crack inspection via binocular CCD camera
images
We used two parallel CCD cameras, to determine
the distance between the object and the cameras. Fig.
3 shows the geometric relationship of an object
Fig. 6. Total gray value summation along the x-direction for the (a) left and (b) right images.
P.-C. Tung et al. / Automation in Construction 11 (2002) 717729 721
appearing before the two cameras. A coordinate
system is defined at the center of the first CCD camera
with its Z-axis along the normal direction of the CCD
chips and the X- and Y-axes along the images x and y-
axes. The following formula can be derived [10]:
Z k
kB
x
2
x
1
; 1
where k is the lens focus length, Z represents the
distance between the object and the planes of the
camera, B represents the distance between the two
CCD camera centers, x
1
,y
1
are the image coordinates
of the first camera and x
2
,y
2
are the image coordinates
of the second image. Using Eq. (1), one can find Z, as
long as the difference between x
1
and x
2
is available.
Once Z is found, X and Y can be obtained by the
following equations [10].
X
x
1
k
k Z 2
Y
y
1
k
k Z: 3
Since the two camera images have a horizontal
shift, the value of (x
1
x
2
) can be found by comparing
any disparities between the two CCD images. Pre-
vious comparison algorithms for finding the corre-
spondence between two images have focused on
matching region segments [7] and/or points, and lines
[710]. Due to differences between any two cameras,
there may exist variations between images, such as
brightness or image noises. A direct comparison of
two images using the region matching methods [7]
does not usually provide good results for determining
these disparities. Although the comparison of signifi-
cant image features (such as lines, circles, etc. [710])
may provide good results, this also requires a long
computation time. Since our CCD cameras are ins-
talled in parallel on a two-axis rotational frame laid on
the front end of Arm 4 of the manipulator system, the
images captured by the cameras will have a horizontal
dislocation in the images X-direction, as shown in
Fig. 4. Therefore, we developed a new algorithm to
compare the total projection gray values along the
images horizontal lines.
2.1. Projection algorithm
The algorithm has five steps.
Step 1: Grab the left and right images, i.e. and
I
l
(x,y) and I
r
(x,y). Illustrated in Fig. 4.
Fig. 7. Total summation difference between the two images along the x-direction.
P.-C. Tung et al. / Automation in Construction 11 (2002) 717729 722
Step 2: For the left and right images (denoted by
I
l
(x,y) and I
r
(x,y), respectively), we find their corre-
sponding images (denoted by I
w
l
(x,y) and I
w
r
(x,y),
respectively) after the Sobel operation. The images
after the Sobel operation are shown in Fig. 5.
Step 3: Project the gray values of I
w
l
(x,y) and I
w
r
(x,y) onto a line parallel to the images x-axis. These
values are plotted in Fig. 6.
P
l
j
X
m
i1
I
w
l
j; i; j 1; 2; 3 . . . n
P
r
j
X
m
i1
I
w
r
j; i; j 1; 2; 3 . . . n;
where m and n represent the height and the width of
the image, respectively.
Step 4: Define a function J(k) as
Jk
X
n
j1
AP
l
j k P
r
jA; k 1; 2; 3 . . . n:
The result is shown in Fig. 7. The value of k, which
minimizes J, represents the disparity, or the value
(x
1
x
2
), for the two images.
Step 5: Utilize Eqs. (1) (3) to calculate the coor-
dinates Z, X and Y.
2.2. Parameter adjustment
We designed a series of experiments using different
lens focus lengths and variable distances to verify the
projection algorithm results. Fig. 8 shows the exper-
imental results when the lens focus length was set to
500 mm. The upper (and the lower) curve represents
the actual Z value (and the estimated Z value) versus
the disparity of the two images. Due to errors in the
estimation of Z, the errors in the estimates of X and Y
became too large to be used for the manipulator
system. Possible reasons include: (i) B, k measure-
ment errors, (ii) the non-parallel effect of CCD chips.
It is difficult to adjust CCD chips, because they are
installed inside the cameras. Even if we could ensure
that the cameras are exactly parallel to each other, the
normal vector of the CCD chips may not be parallel.
Hence, we must add two adjusting parameters to the
estimation formula (Eq. (1)).
Z k
kB
x
2
x
1
m
2
m
1
; 4
where m
1
and m
2
are the compensation parameters.
Parameter m
1
can be considered as the focus length k
adjustment, while m
2
can be considered as the B ad-
Fig. 8. Distance estimation for the binocular CCD cameras.
Table 1
Maximum and mean errors after calibration
Experiment Lens focus
distance (mm)
Object movement
range (cm)
B (mm) k (mm) m
1
m
2
Maximum
error (cm)
Average
error (cm)
Ex. 1 Various 70210 133 25 192.8 1.49 3.25 1.76
Ex. 2 700 mm 70210 133 25 26.90 1.1600 0.684 0.278
Ex. 3 Infinite 70210 133 25 38.90 1.14 1.82 0.65
Ex. 4 1050 mm 70210 133 25 22.62 1.1600 1.155 0.502
Ex. 5 500 mm 6040 44 26.31 0.079 1.049 0.35 0.168
Ex. 6 2000 mm 6040 44 25.31 7.20 0.983 0.438 0.178
P.-C. Tung et al. / Automation in Construction 11 (2002) 717729 723
justment. By minimizing the least square errors of all
differences between the actual and the estimated Z
values, one can obtain optimal m
1
and m
2
values.
Table 1 lists the results for different focus lengths and
the maximum and average errors after calibration. Fig.
9 shows that, after calibration, the errors between the
actual and estimated Z values have been reduced
dramatically. As listed in Table 1, the maximum error
is 3.5 mm and the mean error is 1.68 mm when the
focus is 500 mm, and the working range is 4060 cm.
The corresponding errors for the estimated X and Yare
less than 1 mm.
3. Experimental setup and results
The manipulator system discussed in this article
has four arms. Arm 1 is fixed on a revolving platform
mounted on the vehicle. Arms 1, 2, 3 and 4, as well as
the revolving platform are placed on the vehicle as
shown in Fig. 10. The four arms are arranged as
follows. Arm 1 is placed vertically on the platform.
Arm 2 is laid vertically to Arm 1. On the vertical end
of Arm 2, Arm 3 is fixed to a 1.8-m long C-shaped
steel beam and two other 1.8-m C-shaped steel beams
equipped with slides. Through the action of a sliding
Fig. 9. The resultant curves after calibration.
Fig. 10. The manipulator system.
P.-C. Tung et al. / Automation in Construction 11 (2002) 717729 724
block, Arm 3 can move vertically in the direction of
the Z axis. The dynamic source for the sliding comes
from the lifting device mounted on Arm 2. Arm 4,
which is connected perpendicular to the bottom of the
Arm 3 extension, can revolve around the Arm 3 axis.
As Arm 4 can be extended up to 4 m, it is divided into
two sections in order to facilitate storage; each section
can revolve. The CCD cameras are fastened to the
front end of Arm 4, and the images are transmitted via
BNC cable to the screen of the control computer.
The manipulator system may either revolve or
move linearly. Arm 4, driven by a servomotor and a
velocity reducer, enables a planar revolution facilitates
the observation of bridge cracks. An oil-pressure
motor and gears drives the revolving platform. Arm
3 can move up and down linearly.
Table 2
Size and function of the manipulator system
Number of arm Size Weight (kg) Function
Arm 1 H-shape steel beam:
250 250 1500 mm
151.8 This is the main support beam of the system;
it supports the entire load of the whole structure
and can move up and down, which permits Arm
3 to move over the bridge railing and then down
to facilitate detection.
Arm 2 H-shaped steel beam:
250 250 3400 mm
169.7 Pushes Arms 3 and 4 over the bridge railing by
a revolving movement and supports the load.
Balance weight is laid
on the arms rear end
300
Arm 3 C-shape steel beam:
200 75 1800 mm
22.9
76.15
Pushes Arm 4 below the underside of the
bridge surface by a lifting up and down
First section: sliding
block and sliding rail
200 10 1800 mm
expansion movement.
Second section: sliding
block and sliding rail
200 10 1800 mm
76.15
Total weight of Arm3 175.2
Arm 4 Section 1, aluminum extrusion:
60 60 2500 mm
7 Pushes the CCD camera to the underside of the
bridge surface by a revolving action.
Section 1, sleeve:
90 80 250 mm
3
Section 1, balance weight: 10
Section 2, aluminum extrusion:
80 80 2500 mm
13.5
Section 2, front sleeve:
100 100 250 mm
1.5
Section 2, balance weight: 84
Section 2, rear sleeve:
110 100 250 mm
2.9
Front ad rear shafts: 2.9
Total weight: 124.8
Total weight of Arms 1, 2, 3, 4
and balance weights
922
Fig. 11. Image transmission system.
P.-C. Tung et al. / Automation in Construction 11 (2002) 717729 725
The dimensions of the manipulator system are as
follows: Arm 11.7 m high, Arm 23.4 m long,
Arm 35 m long, and Arm 44 m long. Balance
weights are placed at the ends of Arms 2 and 4,
respectively, to maintain the center of gravity during
operation. Thus, Arms 2 and 4 can rotate smoothly. To
Fig. 12. The bridge to be inspected and the manipulator.
Fig. 13. Arm 2 is approximately perpendicular to the bridge.
P.-C. Tung et al. / Automation in Construction 11 (2002) 717729 726
allow these arms to revolve smoothly, thrust bearings
are used. Arm 4 is made of A6N01S-T5, an integrally
formed aluminum intrusion. This type integral forma-
tion is used as much as possible during processing in
order to reduce the stress concentration. The total
weight of the system, including the balance weights,
is around 922 kg; for further details about the size and
function, please refer to Table 2.
The image transmission system is comprised of
three parts shown in Fig. 11, including a camera
system, an image capturing system and a computer.
A SONY XC-75 camera is used, which has a gene-
Fig. 14. Arm 4 makes both horizontal and circular movements.
Fig. 15. Crack a measured by the CCD cameras right eye.
P.-C. Tung et al. / Automation in Construction 11 (2002) 717729 727
ral resolution 640 480, or at best 769 494. The
image capturing system uses a Matrix Meteor-II
Standard image capture card, which can catch a video
signal at up to 60 frames/s with a resolution of
640 480. The system is operated by a multi-media
computer.
The video signal of the image captured by the CCD
camera is transmitted through the system via a BNC
cable, which sends it to a personal computer, where it
is then displayed on a screen after being processed by
the computers processing unit.
For practical applications, the manipulator system
is transported to the desired inspected bridge. Fig. 12
shows a bridge to be inspected and the manipulator
system. Arm 1 is fixed on a platform that revolves on
the base plate, powered by a 7000-W electric gener-
ator. After rotating the revolving platform toward the
inspection area, Arm 2 will be approximately perpen-
dicular to the bridge railing, as shown in Fig. 13.
Through the action of the sliding block, Arm 3 can
move vertically in the direction of the Z-axis. Arm 4,
which is connected perpendicular to the bottom of the
Arm 3 extension, can now revolve around the Arm 3
axis. Arm 4 is driven by a servomotor and a velocity
reducer to produce a planar revolution, which facili-
tates the bridge cracks observation. Fig. 14 shows that
Arm 4 can make both horizontal and circular move-
ments that enable it to be extended to the underside of
the bridge to observe cracks with the binocular CCD
cameras. The images for the same crack a captured
from the right and left cameras are shown in Figs. 15
and 16, respectively. One can find the horizontal
image difference, that is x
2
x
1
, for the crack a is
119 pixels. By applying Eq. (4) derived in Section 2,
one finds that the estimated distance from the crack
a to the camera is 1.93 m. Also, the crack length is
estimated as 8 cm by applying Eqs. (2) and (3). This
shows that the high degree of accuracy of the system
during on-site observations.
4. Conclusion
We developed a manipulator system using binoc-
ular CCD cameras, which can offer another option to
the current manual bridge crack inspection process.
This system uses two cameras operated in parallel to
detect cracks. A new algorithm is also proposed that
will process the binocular images and calculate the
crack position. Compared with the current method of
inspection, by an inspector standing on the platform of
an inspection vehicle or on a temporary scaffolding,
the manipulator system decreases the danger of acci-
dents. Currently, the use of CCDs with the manipu-
lator system is not intended as a human substitute for
all inspection works, but may only involve a portion
of work, since the human who is put in the same spot
as the CCD cameras will take more intensive advant-
age of human stereovision capabilities, recognition of
color-shades, and ability to perform interactive tests
Fig. 16. Crack a measured by the CCD cameras left eye.
P.-C. Tung et al. / Automation in Construction 11 (2002) 717729 728
such as scratching of the surface and other tactile
investigations.
References
[1] Federal Highway Administration (FHWA), Bridge Inspec-
tions Training Manual, July 1991.
[2] Bridge Maintenance Training Manual, US Federal Highway
Administration, FHWA-HI-94-034, Prepared by Wilbur Smith
Associates, 1992.
[3] B. Bakht, L.G. Jaeger, Bridge testinga surprise every time,
Journal of Structural Engineering, ASCE 116 (5) (May 1990)
13701383.
[4] Product Catalog, Paxton-Mitchell SnooperR Underbridge In-
spection Machines, 26 Broadway26th Floor New York, NY
10004 USA.
[5] Shibata Tsutomu, Shibata Atsushi, Summary Report of Re-
search and Study on Robot Systems for Maintenance of High-
ways and Bridges, Robot, no. 118, Sep. 1997, JARA Tokyo,
Japan, pp. 4151.
[6] J.E. De Vault, Robot system underwater inspection of bridge
piers, IEEE Instrumentation and Measurement Magazine 3 (3)
(Sept. 2000) 3237.
[7] G. Medioni, R. Nevatia, Segment-based stereo matching,
Computer Vision, Graphics and Image Processing, vol. 31,
(1985) 218.
[8] K. Kawasue, T. Ishimatsu, 3-D measurement of moving papers
by circular image shifting, IEEE Transactions on Industrial
Electronics (1997) 703706.
[9] N. Ayache, B. Faverjon, Efficient registration of stereo images
by matching graph descriptions of edge segments, Internation-
al Journal of Computer Vision (1987) 107131.
[10] K.S. Fu, R.C. Gonzalez, S.G. Lee, Robotics Control, Sensing,
Vision, and Intelligence, McGraw-Hill, New York, 1987.
P.-C. Tung et al. / Automation in Construction 11 (2002) 717729 729