Transactions on Industrial Electronics

Design of A Novel Low-cost UAV and Vision-based Concrete

Crack Inspection Method

Journal: Transactions on Industrial Electronics

Manuscript ID 18-TIE-2203
Fo
Manuscript Type: Regular paper

Manuscript Subject: Diagnosis and Monitoring

rP

Keywords: Image processing, Intelligent systems, Monitoring

Are any of authors IEEE

Yes
ee

Member?:

Are any of authors IES

No
Member?:
rR
ev
iew
Page 1 of 8 Transactions on Industrial Electronics
IEEE TRANSACTION ON INDUSTRIAL ELECTRONICS

1
2
3 Design of A Novel Low-cost UAV and
4
5
6
Vision-based Concrete Crack Inspection Method
7
8
9
10 Abstract—With the explosive development of the com- thermography [16], light detection and ranging (LIDAR) sen-
puter vision technology, more and more vision-based in-
11 sors [17], microwave systems [16], etc. Moreover, UAVs can
spection methods enabled by Unmanned Aerial Vehicle
12 (UAV) technologies have been researched on the crack
have small size [16], low cost [16], excellent maneuverability
13 inspection of the sundry concrete structures. However, and versatility [18], and adaptability in complex environ-
14 because of the limitation of the low-cost UAV hardware, ments [19]. Therefore, UAVs can perform the inspections of
15 most of vision-based methods are difficult to be imple- structures which are inconveniently accessible by traditional
mented on the low-cost UAV for real-time crack inspection. inspectors [16]. Especially, UAVs integrated with optic devices
16
To address this challenge, in this paper, a new computa- and image processing has been increasingly applied in the
17 tionally efficient vision-based crack inspection method is
18 designed and successfully implemented on a low-cost UAV. photogrammetric three-dimensional (3D) structural model re-
19 Furthermore, to reduce the acquired data samples, a new construction and analysis [9],[16], remote sensing [9],[20], and
Fo
20 algorithm entitled, Crack Central Point Method (CCPM), is various structural health detections [11],[16],[17-18],[21],[22].
designed to extract the effective information from the pre- Generally, the application of UAVs can overcome many flaws
21 processed images. The proposed vision-based crack detec-
22 of traditional SHM methods [13],[23], improve the quality of
tion method includes three major components, 1) the image
23 inspections [12], reduce inspection time [8], cost and work
rP
pre-processing algorithm, 2) CCPM, and 3) the Support Vec-
24 tor Machine (SVM) model based classifier. To demonstrate incidents[21], decrease logistics [21] and traffic interference
25 the effectiveness of the new inspection method, a concrete [8], and increase structural accessibility [8],[16].
structure inspection experiment is implemented. The ex-
26 Computer vision technology applies multi-disciplinary tech-
ee

perimental results indicate that this new method is able to

27 accurately and rapidly inspect the cracks of concrete struc- nologies, which include optical imaging, image processing,
28 ture in real time. This new vision-based crack inspection artificial intelligent and machine learning, to derive meaningful
29 method shows great promise for practical application. information and gain high-level understanding from images or
rR

30 Index Terms—Vision-based Inspection; Crack Inspec- videos [24]. In the last two decades, computer vision has been
31 tion; Computer Vision; Support Vector Machine (SVM); increasingly utilized in SHM [6-7],[14],[21], which expedites
32 Structural Health Monitoring (SHM); Unmanned Aerial Ve- inspection process [16], decreases costs and safety risk [16],
ev

33 hicle (UAV). increases accuracy and facilitates access of structure [16].

34 Unquestionably, computer vision-based SHM (CVSHM) has
35 been a state-of-the-art research area [16]. Recently, computer
I. I NTRODUCTION
36
iew

vision-based and UAV-enabled methods have been implement-

37 GING and degradation of civil and industrial infrastruc- ed for SHE of sundry concrete structures, which is shown in
38
39
A tures has posed notable public safety threat, and attracted
many attentions from researchers [1-2]. Consequently, an
Table 1. However, almost all applications in Table 1 just use
UAVs as a data collection tool for SHM. Even though Pereira
40 increasingly number of Structural Health Monitoring (SHM) et al. [19] and Sankarasrinivasan et al. [22] have proposed
41 applications with noncontact means have been researched and the integrated UAV and Image Processing System to enable
42 applied for the monitoring and inspection of a variety of real-time SHM, the integration of UAVs and computer vision
43 critical structures [3-4], such as bridge [1],[5-8], dam [2],[9] technology have not been fully explored and implemented
44 and wind turbine [10-11]. However, the majority of noncontact to automate and optimize the SHE process. Till now, how
45 SHM methods can be complex [12], time-consuming [13- to integrate computer vision with the cutting-edge machine
46 14], labor-intensive [5],[8],[14], expensive [9],[12-13], or error learning algorithm, into low-cost UAVs for accurate, efficient
47 prone due to the limitation of the skill and experience of and real-time SHM remains an unsolved issue.
48 inspectors[5]. Additionally, it engenders unexpected safety To address this issue, this paper proposes a vision-based
49 risks when the inspectors access the unsafe structural areas crack inspection method with Support Vector Machine (SVM),
50 [9],[12-13]. Therefore, developing the cost-effective, accurate which can bex integrated into the low-cost UAV. To the best
51 and automated noncontact solutions for efficient and reliable knowledge of the authors, this is the groundbreaking research
52 SHM has been an emerging research topic [8]. to develop a low-cost UAV enabled, vision and machine
53 learning based, and real-time crack inspection method with
54 Nowadays, the rapid development of Unmanned Aerial
Vehicle (UAV) propels SHM of the large-scale concrete struc- two significant contributions.
55
56 tures to advance with the direction of process automation 1) To acquire the smaller set of valid data samples for
57 and remote monitoring [15]. Currently, UAVs can carry a the further crack recognition, a novel Crack Central Point
58 wide range of imagery and sensing instruments, including Method (CCPM) [28] has been developed to extract the
59 cameras [13], radar [16], optical imagery[16-17], infrared
60
 Transactions on Industrial Electronics Page 2 of 8
IEEE TRANSACTION ON INDUSTRIAL ELECTRONICS

1
TABLE I: T HE E XAMPLES OF UAV VISION - BASED SHM A PPLICATION
2
3 Authors Applications Data collection Image processing Methods
4
Khaloo et al. [9] Applied multiple UAVs and multi-scale Collected the images by the UAVs from Multi-scale photogrammetry and com-
5 photogrammetry for dam inspection. a hydroelectric dam. puter vision to detect dam defects.
6
Hallermann et al. [12] Applied the low-cost UAVs to monitor Collected the images by UAVs. Computer Vision to detect displacement
7 large scale structures (dams, walls, etc.) and deformation in post flight analysis.
8 Kim et al. [13] Applied UAVs for the concrete cracks Collected the images by the UAVs. Image Binarization and Hybrid Image
9 assessment. Processing in workstations.
10 Brooks et al. [17] Applied UAVs for traffic monitoring Collected the images and other infor- Vision-based image processing to re-
11 and infrastructural health inspection. mation by the UAVs. construct 3D sites and identify damage.
12 Pereira et al. [19] Applied UAVs to automatically detect Captured the images and other data by Edge detection and particle filter in
13 cracks in civil constructions. the UAVs. MatLab / embedded sensors on UAVs.
14 Ellenberg et.al [20] Applied UAVs for remote sensing and Collected the high-quality images and Vision-based image processing and
15 inspection of the large-scale infrastruc- other data by UAVs. quantitative assessment to evaluate the
ture condition. infrastructural health.
16
Sankarasrinivasan et al. [22] Applied UAVs to detect crack and sur- Acquired the images and other data by Hat transforms, HSV and grey scale
17 face degradation of large structures. the UAVs. thresholding in MatLab.
18
Wang et.al [25] Applied UAVs to inspect the cracks of Collected the high-quality images by Haar-like features, LogitBoost, Deci-
19 wind turbine blade surface. the UAVs. sion Tree, and SVM to detect cracks.
Fo
20 Eschmann et al. [26] Applied UAVs for remote damage as- Captured the high resolution images by Image post-processing and pattern
21 sessment and monitoring purposes. the UAVs. recognition to assess damages.
22 Kang et al. [27] Applied UAVs, ultrasonic beacons for Collected the images by the UAVs. Deep convolutional neural network (C-
23
rP
damage localization and detection. NN) to detect damage.
24
25
26
ee

cracks characters from the pre-processed images. CCPM has A. Principle Statement
27 improved adaptability and robustness compared to both the
28 traditional edge detection methods such as Canny, Prewitt and Traditionally, a computer vision task consists of the methods
29 Sobel algorithms, and K-mean clustering based method [28]. of obtaining, processing, analyzing and interpreting digital
rR

30 With CCPM, the size of training data set for the cracks recog- images or videos, and extracting multi-dimensional data to
31 nition can be significantly reduced [28], which provides the generate numerical or symbolic information [24]. To rapidly
32 possibility of fast computation and real-time crack detection. and effectively detect the cracks from the captured images, an
ev

33 innovative vision-based and SVM supported crack inspection

34 2) To accurately and rapidly recognize different types of method is proposed. Fig. 1 shows the working process of the
35 concrete cracks with fewer training samples, a SVM based proposed inspection method, which includes three steps: 1)
36 crack classification algorithm has been researched and devel-
iew

original image pre-processing; 2) crack feature extraction with

37 oped. SVM requires fewer training images and takes less train- CCPM method; and 3) crack classification and recognition.
38 ing time than other machine learning especially deep learning The general illustration of each step is presented as follows.
39 methods. Due to its low computational complexity and high
detection accuracy, this algorithm can be implemented in the First, the images of the concrete surface, which are acquired
40
low-cost UAVs for real-time and automated crack inspection. by the UAVs, are converted to grayscale binary images via pre-
41
processing. Second, CCPM is applied to extract the center of
42 The rest of this paper is organized as follows. Section II each crack from the grayscale binary images. CCPM have
43 presents the design process and mathematical models of the improved performance than the traditional edge detection
44 proposed vision-based crack inspection method. Section III
45 describes the experiment to illustrate the effectiveness and
46 applicability of the novel vision-based method. Concluding
47 remarks and future works are summarized in Section IV.
48
49 II. A N EW V ISION - BASED C RACK I NSPECTION M ETHOD
50
51 The goal of our research is to develop an effective recog-
52 nition method to quickly inspect and detect various types of
53 cracks from images captured by the UAVs. Meanwhile, this
54 proposed method should have low computational complexity
55 to enable the real-time crack inspections using the low-cost
56 UAV devices. To design this method, the whole process will
57 begin from the principle analysis of the research problem.
58 Then mathematical models of each step will be assembled to
ensure the validity of key design components of this method. Fig. 1: Method Working Process.
59
60
Page 3 of 8 Transactions on Industrial Electronics
IEEE TRANSACTION ON INDUSTRIAL ELECTRONICS

1
2 methods, especially when the images are captured with uneven
3 nature light, dirty spots and rusts on the structural surface, and
4 environment noise [28]. Also, CCPM has advantages over K-
5
means clustering algorithms, due to the more computational
6 time, sensitivity to noise points and variability of K value
7 in K-means clustering [28]. In this crack extraction process,
8 the crack is characterized by the existing minimum grayscale
9 value, which exhibits the property of parabolic function in the
10 row of crack area [28]. The eigenvalues selected by CCPM can
11 differentiate four types of cracks: Transverse, Longitudinal,
12 Oblique and Mesh crack [28]. Finally, the characteristics of the
13 extracted cracks should be mutually independent. To meet the
14 real-time computational capacity of the low-cost UAV, SVM-
15 based method is selected as the crack classifier. The crack
16 projection characteristics in X and Y coordinates, crack area
17 and distributed density are the input features.
18
19
B. Mathematical Model for Proposed Method
Fo
20
21 According to the working process in Fig. 1, the correspond-
22 ing mathematical model of each step is shown as follows.
23
rP
where C(α) = pi · α, and δ(C(α), pi ) = kC(α) − pi k2 . Then
the minimization problem is to minimize the objective function
N X
X
L(pi ) = arg min kC(α) − pi k2
P ={pi }N
i=1 i=1 pi ∈P
(2)
= kDak2 .
which is subject to the constraint that kαk2 = 1. The design
matrix D is a n × 6 matrix with rows xi . Then the problem
becomes an eigenvector problem DT Dα = λα, where αmin
is the eigenvector corresponding to the smallest eigenvalue.
2) Crack Feature Extraction: The crack is characterized
based on the existed minimum grayscale value which displays
a parabolic property and is calculated from Eq.(2). Through
analyzing the characteristics of the crack areas in the pre-
processing image, the image function is f (r, c). The gradient
of the points in digital image is a vector. Then, the gradient is
determined by the vector of the first-order partial derivative.
∂f (r, c) ∂f (r, c)
4 f (r, c) = ( , ) = (fr , fc ), (3)
∂r ∂c
where fr and fc are the gradients along the row and column

1) Original Image Pre-Processing: The original digital directions respectively. The respective row and column deriva-
24
color image Io is represented by a Ho × Wo × 3 matrix, where tives, and direction angles are:
25
Ho and Wo are the height and width of the image. The pixel
26 ∂f (r, c) ∂f (r, c)
ee

xi,j in image Io is denoted as [b, g, r] representing the color (4f (r), 4f (c) = ( , ),
27 ∂r ∂c
of the pixel, where i = {0, 1, . . . Ho }, j = {0, 1, . . . Wo }. Io
28 fr
is then converted into image Ig in grayscale. The grayscale φ = −arctan( ), (4)
29 fc
rR

value in the pixel xi,j in Ig can be represented as

30 0 1
31 fi = (fi+1 − fi−1 ).
xi,j = 0.299r + 0.587g + 0.114b. 2
32 0
where fi = 0, and the minimum grayscale value can be
ev

33 The threshold method is applied to create the binary image

Ib . To reduce the impact of the different lighting conditions obtained at the corresponding point. Since the distribution of
34
on the image, adaptive threshold method is used. The value in the grayscale value f (r, c) in crack area is a parabolic function,
35
pixel xi,j in Ib is determined by two adjacent points r and c can be taken to fit the parabola
36
iew

with f (r − 1, c − 1), f (r + 1, c + 1). Then the symmetrical

37 (
M axV alue, if xi,j in Ig > Wi,j , line position of the parabola, the center of the crack (ri , cj ),
38 xi,j =
0, otherwise. where i < r and j < c in a row or column, can be calculated.
39
40 where Wi,j is the is a weighted sum (cross-correlation with a However, in the real complex environment, some factors
41 Gaussian window) of the neighborhood of xi,j . including the presence of noise, rust, and others, result in the
42 grayscale value to be no more than the minimum grayscale
43 The contours C in the binary grayscale image are detected value in the crack area [28]. Consequently, the results of the
44 based on the topological analysis of binary images introduced crack detection, feature extraction and crack classification are
45 in [29], in which the contours are determined to be the affected as well. To address this essential issue, the threshold
46 boundaries between the black and white points. Each detected can be adjusted as a basic characteristic of detecting the crack
47 contour of C is a set of 2-D points and i = {1, 2, , N }, according to the features of the crack in the image. The
48 where N is the number of contours. The ellipse fitting of the threshold is T , and detected length is H. Characteristics are:
49 detected contours C is based on the Least Squares algorithm (
[30]. In our method, given a set of 2-D points in the image H ≥ T, (crack characteristics),
50
51 P = {pi }N i=1 where, pi = (xi , yi ) and xi , yi are two- 0 ≤ H < T, (other characteristics).
52 dimensional coordinates of the point, pi . An ellipse curve
C(α) is parameterized by the vector α and the distance metric The crack width is defined to be larger than 2mm [31]. If the
53 length of the recognized feature is no less than the threshold
54 δ(C(α), pi ) which measures the distance from a point pi to the
curve C(α), find the global minimum grayscale value through (100 or 400), it’s considered as a crack. Otherwise, it’s not.
55
the objective function, which is given by To find the edge coordinates of the crack, we take the second
56
derivative of the Eq.(3).
57 N X
X
58 L(pi ) = arg min δ(C(α), pi ). (1) ∂ 2 f (r, c) ∂ 2 f (r, c)
P ={pi }N 42 f (r, c) = ( + ) = (frr + fcc ), (5)
59 i=1 i=1 pi ∈P
∂r2 ∂c2
60
 Transactions on Industrial Electronics Page 4 of 8
IEEE TRANSACTION ON INDUSTRIAL ELECTRONICS

1
2 In practice, the formula of second derivatives is: dual objective function Ψ is still quadratic in α. Thus, the
3 object function of SVM model is given as,
00 1
4 f (i) = (fi+1 − 2fi + fi−1 ), N X
N N
2 1
min Ψ(→
− yi yj K(→
−
xi , →
−
X X
5 α ) = min xj )αi αj − αi ,
α α 2
6 00
When fi = 0, we can obtain its maximum and minimum. The i=1 j=1 i=1
7 crack edge coordinates are the coordinates of maximum and 0 ≤ αi ≤ C, ∀i,
8 minimum values, (ri+1 , cj+1 ) and (ri−1 , cj−1 ) respectively. N
9
X
Finally, the crack width, W , can be calculated below: yi αi = 0.
10 i=1
11 RowR − Row (9)
12 W =2×{
cos(φ)
}. (6)
where C is the correction parameter.
13
14 where RowR and Row are the crack right and center position. Furthermore, Karush-Kuhn-Tucker (KKT) conditions are
15 necessary and sufficient for an optimal point of a positive
3) Cracks Classification and Recognition: To develop an definite SVM QP problem [35], and K must obey Mercers
16
effective and accurate cracks classifier for the low-cost UAV conditions [36]. KKT conditions for the QP problem in Eq.
17
equipment, based on our preliminary research, SVM model (9) are particularly simple. The QP problem is solved below:
18
[32] is a good selection. The key ideas and advantages of SVM
19 αi = 0 ⇔ yi fi ≥ 1,
include: 1) the efficient separability of non-linear regions,
Fo
20 0 < αi < C ⇔ yi fi = 1, (10)
which use kernel functions to generalize the similarity to the
21
new similarity measures by dot products [33]; 2) the use αi = C ⇔ yi fi ≤ 1.
22
of quadratic optimization to avert local minimum issues in
23
rP
where fi is the output of the SVM for the ith training sample.
Artificial Neural Network (ANN) [33]; 3) the result learning
24 To solve the SVM QP problem, SMO is used to optimize
algorithm will be an optimization algorithm instead of a greedy
25 the objective function, Eq. (9) based on KKT conditions as
search [33]. Thus, compared to the other traditional machine
26
ee

learning or pattern recognition methods, SVM requires fewer Eq.(10). SMO can quickly solve the SVM QP problem without
27 any extra matrix storage or using numerical QP optimization
data samples [32]. In our proposed method, the Sequential
28 steps at all. SMO decomposes the overall QP problem into QP
Minimal Optimization (SMO) algorithm is applied to deal
29 sub-problems, using Osunas theorem to ensure convergence.
rR

with the image classification and recognition. SMO algorithm

30 The SMO algorithm includes three steps: 1) an analytic
was developed to solve the SVM quadratic programming (QP)
31 method for solving for the two Lagrange multipliers, 2) a
problem [34]. SMO algorithm first decomposes the SVM
32 heuristics for selecting multipliers to optimize, and 3) com-
QP problem into several QP sub-problems [34], then solves
ev

33 puting the threshold. The three steps are shown as below:

each smallest possible optimization problem individually [34],
34
which involves two Lagrange multipliers at each step. Step 1: Solving for two Lagrange multipliers
35
36
iew

SVM model includes two different forms: linear and non- In SMO, all quantities referring to the first multiplier have
37 linear form. Considering the complexity of the available data a subscript 1, and a subscript 2 for the second multiplier.
38 sets and application, non-linear form SVM model is selected Without loss of generality, the algorithm first computes the
39 for the classification. The output formula of SVM is below, second Lagrange multiplier α2 and computes the ends of the
40 N
diagonal line segment in terms of α2 . If the target y1 does not
41 fi =
X
yj αj K(→
−
xj , →
−
x ) − b, i, j = {1, 2, · · · , N }, (7) equal the target y2 , then the following bounds apply to α2 :
42 j=1 L = max(0, α2 − α1 ), H = min(C, C + α2 − α1 ). (11)
43
44 where yj is label set, αj is Lagrange multipliers, b is bias, If the target y1 equals y2 , then the below bounds apply:
45 and K is the kernel function that measures the similarity or
L = max(0, α2 + α1 − C), H = min(C, α2 + α1 ). (12)
46 distance between the input vector →−x and the stored training
47 →
−
vector xj . In the presented method, Gaussian kernel is finally where L is bound negative support vectors and H is bound
48 selected due to its higher efficiency, lower simplicity, and positive support vectors.
49 better performance compared to other SVM kernels. The
The second derivative of the objective function along the
50 Gaussian kernel function is shown as follows.
diagonal line can be expressed as:
51 2
52 K(→
−
xj , →
−
x ) = Exp(−
kx − xj k
). (8) η = K(− →, −
x →) + K(−
x
1 1
→, −
x →) − 2K(−
x 2 2
→, −
x x→).
1 2
53 2σ 2
In normal circumstances, the objective function is positive
54 where σ is a free parameter. σ is utilized to control the definite. Also, there is a minimum along the direction of the
55 underfitting (large σ 2 for high bias and low variance) and linear equality constraint, and η is greater than zero. SMO
56 overfitting (small σ 2 for low bias and high variance). computes the minimum along the direction of the constraint:
57
58 The Lagrange multiplier αi is computed via the quadratic y2 (E1 − E2 )
program. The non-linearity alters the quadratic form, but the α2new = α2 + , (13)
59 η
60
Page 5 of 8 Transactions on Industrial Electronics
IEEE TRANSACTION ON INDUSTRIAL ELECTRONICS

1
2 where E = ui − yi is the error on the ith training sample. As Once the first Lagrange multiplier is chosen, the second
3 the next step, the constrained minimum is found by clipping Lagrange multiplier is chosen by SMO to maximize the step
4 the unconstrained minimum to the ends of the line segment: size taken during joint optimization. However, evaluating the
5  kernel function K is time consuming, so SMO approximates
 H if α2new ≥ H,
6 the step size by the absolute value of the numerator in

new,clipped new
7 α2 = α2 if L < α2new < H, Eq.(13): |E1 − E2 |. SMO keeps a cached error value E for

8 L if α2new ≤ L. each non-bound training example and then chooses an error to

9 Let s = y1 y2 . α1 is computed from the newly clipped α2 :
approximately maximize the step size. If E1 is positive, SMO
10 chooses an example with minimum error E2 . Conversely, if
11 α1new = α1 + s(α2 − α2new,clipped ). (14) E1 is negative, SMO chooses one with maximum error E2 .
12 Step 3: Computing the threshold
In unusual circumstances, η is not positive. A negative η
13
occurs if K does not obey Mercers condition, which can cause The threshold b is recomputed after each step, so that the
14
the objective function to become indefinite. A zero η can occur KKT conditions are fulfilled for both optimized examples. The
15
even with a correct kernel, if more than one training sample following threshold b1 is valid when the new α1 is not at the
16
has the same input vector x. In any event, SMO will work even bounds, since it forces the output of the SVM model to be y1
17
when η is not positive. In those cases, the objective function when the input is x1 :
18
Ψ should be evaluated at each end of the line segment:
19 b1 = E1 + y1 (α1new − α1 )K(− →, −
x →
1 x1 )
f1 = y1 (E1 + b) + α1 K(− →, −→ −
→ − →
Fo
20 (16)
x 1 x1 ) − sα2 K(x1 , x2 ), + y2 (α2new, clipped − α2 )K(−→, −
x →
1 x2 ) + b.
21 f = y (E + b) + sα K(−
2 2 2 1
→, −
x 1
→) − α K(−
x 2 2
→, −
x 2
→),
x 2
22 The following threshold b2 is valid when the new α2 is not
L1 = α1 + s(α2 − L),
23
rP

24 H1 = α1 + s(α2 − H),
25 1 1 2 −
ΨL = L1 f1 + Lf2 + L21 K(−
→, −
x → → −→
1 x1 ) + L K(x2 , x2 )
26 2 2
ee

27 + sLL1 K(−→, −
x →
1 x2 ),
28 1 1 2 −
ΨH = H1 f1 + Hf2 + H12 K(− →, −
x → → − →
1 x1 ) + H K(x2 , x2 )
29 2 2
rR

30 + sHH1 K(− →, −
x →
1 x2 ).
31 (15)
32
SMO will move the Lagrange multipliers to the end point that
ev

33
has the lowest value of the objective function.
34
35 Step 2: Heuristics for selecting multipliers to optimize
36
iew

To choose the multipliers for optimization, two separate

37 choice heuristics are required, which include one for the first
38 Lagrange multiplier. and one for the second. The choice of the
39 first heuristic provides the outer loop of SMO. The outer loop Fig. 2: Hardware of Low-cost UAV.
40 first iterates over the entire training set, determining if each
41 example violates the KKT conditions in Eq.(10). If an example TABLE II: T HE S PECIFICATIONS OF L OW- COST UAV
42 violates KKT, then it is eligible for optimization. The first
43 choice heuristic concentrates the CPU time on the examples Components Name Components Type Quantity
44 that are most likely to violate the KKT conditions: the non- Micro Controller STM32F407VGT6 1
45 bound subset. After one passes through the entire training Barometer MEAS MS5611 1
46 set, the outer loop iterates over all examples whose Lagrange Accelerometer MPU6050 1
47 multipliers are neither 0 nor C (the non-bound examples). Magnetometer HMC5883L 1
48 Wi-Fi Module ESP8266 1
49 Again, each example is checked against the KKT conditions Flash M24128-BWDW6TP 1
50 and violating examples are eligible for optimization. The outer Ultrasonic Module HC-SR04 1
51 loop repeatedly passes over the non-bound examples until all Storage Module Micro SD Card 1
52 of the non-bound examples obey the KKT conditions within ε Screen 0.96 inch OLED screen 1
53 which is the control parameter. Notice that the KKT conditions LED RGB full color LED 1
54 are checked to be within ε of fulfillment. Typically, ε is set to ESC Brushless Motor HTI-D20-491101 4
55 be 10−3 . The outer loop then goes back and iterates over the Brushless Motor A2212-13 980KV 4
56 entire training set. The outer loop keeps alternating between GPS Module Radiolink M8N GPS SE100 1
57 single pass over the entire training set and multiple passes over Figure transmission module VT58 5.8G 40CH FPV 1
58 the non-bound subset until the entire training set obeys the Power supply Module 3S lithium battery (5200mAH 11.1 V ) 1
59 KKT conditions within ε where upon the algorithm terminates.
60
 Transactions on Industrial Electronics Page 6 of 8
IEEE TRANSACTION ON INDUSTRIAL ELECTRONICS

1
2 at bounds, because it forces the output of the SVM model to C) module, Ultrasonic module, Inertial navigation module,
3 be y2 when the input is x2 : Transmission and reception module, Optical flow module,
4 b2 = E2 + y1 (α1new − α1 )K(− →, −
x →
1 x1 )
Communication module, Global Positioning System (GPS)
5 (17) module, Power loss memory module, and the power supply
new, clipped
+ y2 (α2 − α2 )K(−→, −
x →
1 x2 ) + b.
6 module.
7 When both b1 and b2 are valid, they are equal. When both Control module is composed of ST M 32F 103ZET micro-
8 new Lagrange multipliers are at bound and if L is not equal computer. It is used to control the attitude of the quadrotor
9 to H, then the interval between b1 and b2 are all thresholds UAV, exchange data between the quadrotor UAV and terminal
10 that are consistent with the KKT conditions. SMO chooses the devices, and handle the data acquisition and processing. ESC
11 threshold to be halfway in between b1 and b2 . module is used to handle the driving operation of the brushless
12 motor in the quadrotor UAV. Inertial navigation module is
13 III. E XPERIMENTS AND R ESULT A NALYSES composed of an accelerometer, gyroscope, electronic compass,
14 and barometer. It is used to provide the information feedback
15 To verify the performance of the new vision-based cracks for the smooth flight of the quadrotor UAV. Ultrasonic module
16 inspection method, a bridge structure inspection experiment is used to measure the flight height of the quadrotor UAV.
17 using the UAV combined with computer vision and SVM Transmission and reception module is used for the wireless
18 algorithm for crack detection is implemented as follows. real-time video transmission. Optical flow module is used for
19 the fixed-point control of the quadrotor UAV. Communication
Fo
20 A. Hardware configuration for crack data acquisition module uses N RF 24L01 module for the quadrotor UAV
21 and PC side of the data transmission. GPS module is used
22 The hardware platform of a low-cost quadrotor UAV has for positioning and navigation of the quadrotor UAV in the
23
rP
ten modules: Control module, Electronic Spark Control (ES- high altitude. Power loss memory module is used to handle
24
25 Algorithm I: Vision-based Cracks Inspection Algorithm
26
ee

Input: Original Color Image Matrix,I = (Ho × Wo × 3)

27
Output: Accuracy A
28
29 Step 1: Convert Color Image to Binary Image.
rR

30 GrayPixel xi,j = ConvertImage.OrigToGray(I(Ho × Wo × 3));

31 BinaryPixel xi,j = CovertImage.GrayToBin(GrayPixel xi,j );
32
Step 2: Detect the contour in grayscale binary image.
ev

33
Compute smallest eigenvalue αmin with Eq.(2);
34
35 Compute minimum grayscale value L(pi ) with Eq.(2);
36
iew

Step 3: Crack Feature Extraction.

37 Compute φ with Eq.(3) and Eq.(4);
38 Compute H with Eq.(6);
39 Step 4: Cracks Classified by SVM algorithm (SMO).
40 Training set −
→;
x j
41 Label set −
→ −
→
yj = H ;
Fig. 3: Results of crack inspection experiment.
42
Initialize: αi = 0,fi = −b;
43
Compute: L, H using Eq.(11) and Eq.(12);
44
45 Update: α1 , α2 ; using Eq.(13) and Eq.(14)
46 Repeat:
47 Update f1 , f2 using Eq.(15);
48 Compute L1 , H1 using Eq.(15);
49 Update α1 , α2 using Eq.(13) and Eq.(14);
50 Until: f1 converge to f2 ;
51
Update: b1 , b2 using Eq.(16) and Eq.(17).
52
Store new α1 and α2 .
53
54 Step 5: Recognize the cracks.
55 Input the new samples S;
56 Call SVM algorithm to identify the input sample;
57 Calculate the inspection accuracy;
58 Output the Cracks inspection accuracy A; Fig. 4: Crack inspection accuracy.
59
60
Page 7 of 8 Transactions on Industrial Electronics
IEEE TRANSACTION ON INDUSTRIAL ELECTRONICS

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Fo
20
21
22
23
rP

24
25
26
ee

27
28
29
rR

30
31 Fig. 5: The whole experimental process.
32
ev

33
34 the debugging of the flight control algorithm. Power supply reason lies in the different optical (lens) distortion, which is
35 module is used to supply the entire system with power. Fig. 2 generated by the optical design of lenses in the different height
36
iew

shows the hardware and real environment of the low-cost UAV. of UAVs. Fig. 5 shows the example images and result of the
37 The detail specifications of components are listed in Table II. real-time crack inspection by the proposed method.
38
Overall, the result illustrates the accuracy, applicability and
39
B. New Algorithm Design for the Proposed Method robustness of the proposed methods in the different flight
40
height of the UAV. The only minor issue is that SVM has
41 Based on Fig. 1, Algorithm I with C language is used to slightly lower detection accuracy for mesh cracks than for
42 implement this new method (shown in Fig. 2). SVM uses the other crack types. The reason may be that mesh cracks are
43 crack projection characteristics in X and Y coordinates, crack more sensitive to the environment noise and have less crack
44 area and distributed density as the input features and crack length. Consequently, some features of mesh cracks may be
45 type as the output label. The correct and classified label of easier to be filtered out in the image pre-processing. So mesh
46 each sample are compared to compute accuracy. cracks have more probability to be undetected or misclassified.
47
48
49 C. Result Discussion IV. C ONCLUSION AND F UTURE W ORK
50
51 In our experiment, we inspected four types of cracks: Trans- Till now, few noncontact and vision-based SHM methods
52 verse, Longitudinal, Oblique and Mesh crack in different flight have been effectively utilized and integrated with low-cost
53 heights of UAVs. These inspection images are captured by the UAVs to perform the real-time crack inspection. To solve
54 UAV in real time, and translated to the terminal devices, such this challenge, a novel low-cost UAV enabled, vision and
55 as, laptops, pads, smart phones, and others via Wi-Fi network. SVM based cracks inspection method has been designed and
56 The experimental result in Fig. 3 shows the measurement size implemented to inspect the cracks of concrete structure in
57 of each crack is pretty close to the actual size. Meanwhile, Fig. real time. The working procedures include the three steps:
58 4 demonstrates that the accuracy of the inspection of different pre-processing of the image acquired by the UAV, crack
59 cracks generally improves with the flight height increase. The characters extraction from the pre-processed images using
60
 Transactions on Industrial Electronics Page 8 of 8
IEEE TRANSACTION ON INDUSTRIAL ELECTRONICS

1
2 CCPM, and the crack classification via SVM. The results [14] V. Hoskere, Y. Narazaki, T. Hoang, B. Spencer Jr, 2018. ”Vision-
based Structural Inspection using Multiscale Deep Convolutional Neural
3 from the bridge concrete inspection experiment further validate
Networks.” arXiv preprint, arXiv:1805.01055.
4 the effectiveness and robustness of this proposed method.
[15] D. Lattanzi, G. Miller, 2017. ”Review of robotic infrastructure inspection
5 Therefore, this paper provides an exemplary research about the systems.” Journal of Infrastructure Systems, 23(3), p.04017004.
6 state-of-the-art computer vision, machine learning technology [16] A. Khaloo, D. Lattanzi, K. Cunningham, R. DellAndrea, M. Riley,
7 and low-cost UAVs application in noncontact SHM. The 2017. ”Unmanned aerial vehicle inspection of the Placer River Trail
Bridge through image-based 3D modelling.” Structure and Infrastructure
8 proposed method realizes the real-time crack inspection and Engineering, pp.1-13.
9 process automation, and displays the significant advantages [17] C. Brooks, R.J. Dobson, D.M. Banach, D. Dean, T. Oommen, R.E. Wolf,
10 over traditional noncontact SHM methods. T.C. Havens, T.M. Ahlborn, B. Hart, 2015. ”Evaluating the use of
unmanned aerial vehicles for transportation purposes.” (No. RC-1616),
11 Future research includes the three following aspects: 1). The Michigan Tech Research Institute.
12 detailed analysis of computational time and space complexity [18] I. Hernandez, T. Fields, J. Kevern, 2016. ”Overcoming the Challenges
13 of the proposed vision-based method, to implement seamless
of Using Unmanned Aircraft for Bridge Inspections.” AIAA Atmospheric
Flight Mechanics Conference, pp.3396.
14 integration with low-cost UAVs and reduce system failures. [19] F.C. Pereira, C.E. Pereira, 2015. ”Embedded image processing systems
15 2). The solutions to improve CCPM, so that its robustness can for automatic recognition of cracks using uavs.” IFAC-PapersOnLine,
16 be further enhanced especially for the low-quality images cap-
48(10), pp.16-21.
[20] A. Ellenberg, L. Branco, A. Krick, I. Bartoli, A. Kontsos, 2014. ”Use
17 tured with severe noise, dirty spots, rusts, shades, UVA jitter or of unmanned aerial vehicle for quantitative infrastructure evaluation.”
18 in poor environmental conditions. 3). The crack classification Journal of Infrastructure Systems, 21(3), p.04014054.
19 using genetic algorithm, or other machine learning algorithms [21] N. Metni, T. Hamel, 2007. ”A UAV for bridge inspection: Visual
servoing control law with orientation limits.” Automation in construction,
Fo
20 (e.x., boosting, random forest or neural network), to improve 17(1), pp.3-10.
21 the accuracy of the mesh crack detection. [22] S. Sankarasrinivasan, E. Balasubramanian, K. Karthik, U. Chandrasekar,
22 R. Gupta, 2015. ”Health Monitoring of Civil Structures with Integrated
UAV and Image Processing System.” Procedia Computer Science, (54),
23
rP
R EFERENCES pp.508-515.
24 [23] S.S. Choi, E.K. Kim, 2015.”Building crack inspection using small
[1] T. Harms, S. Sedigh, F. Bastianini, 2010. ”Structural health monitoring
25 of bridges using wireless sensor networks.” IEEE Instrumentation and
UAV.” In 17th International Conference on Advanced Communication
Technology (ICACT), (pp. 235-238). IEEE, July, 2015.
26 Measurement Magazine, 6(13), pp.14-18.
ee

[24] J. Ponce, D. Forsyth, S. Antipolis-Mditerrane, R. dactivit RAweb, L. In-

27 [2] G. Tessari, P. Riccardi, P. Pasquali, 2017, December. ”Monitoring the on- ria, I. Alumni, 2011.”Computer vision: a modern approach.” Computer,
going deformation and seasonal behaviour affecting Mosul Dam through
28 space-borne SAR data.” In AGU Fall Meeting Abstracts.
16(11).
[25] L. Wang, Z. Zhang, 2017. ”Automatic detection of wind turbine blade
29 [3] F.G. Baptista, J. Vieira Filho, 2009. ”A new impedance measurement surface cracks based on uav-taken images.” IEEE Transactions on Indus-
rR

30 system for PZT-based structural health monitoring.” IEEE Transactions trial Electronics, 64(9), pp.7293-7303.
on Instrumentation and Measurement, 58(10), pp.3602-3608.
31 [4] Y. Yu, J. Ou, J. Zhang, C. Zhang, L. Li, 2009. ”Development of wireless
[26] C. Eschmann, C.M. Kuo, C.H. Kuo, C. Boller, 2013. ”High-resolution
multisensor infrastructure inspection with unmanned aircraft systems.”
32 MEMS inclination sensor system for swing monitoring of large-scale ISPRS-International Archives of the Photogrammetry, Remote Sensing
ev

33 hook structures.” IEEE Transactions on industrial Electronics, 56(4), and Spatial Information Sciences, (2), pp.125-129, 2013.
pp.1072-1078.
34 [5] M. Magno, D. Boyle, D. Brunelli, B. O’Flynn, E. Popovici, L. Benini,
[27] D. Kang, Y.J. Cha, 2018. ”Autonomous UAVs for Structural Health
Monitoring Using Deep Learning and an Ultrasonic Beacon System with
35 2014. ”Extended wireless monitoring through intelligent hybrid energy GeoTagging.” ComputerAided Civil and Infrastructure Engineering.
36 supply.” IEEE Transactions on Industrial Electronics, 61(4), pp.1871-
iew

[28] B. Lei, N. Wang, P. Xu, G. Song, 2018. ”New Crack Detection Method
1881.
37 [6] R. Davoudi, G.R. Miller, J.N. Kutz, 2017. ”Computer Vision Based
for Bridge Inspection Using UAV Incorporating Image Processing.”
Journal of Aerospace Engineering, 31(5), p.04018058.
38 Inspection Approach to Predict Damage State and Load Level for RC [29] S. Suzuki, 1985. ”Topological structural analysis of digitized binary
Members.” Structural Health Monitoring 2017.
39 [7] J.Z. Huang, D.S. Li, H.N. Li, G.B. Song, Y. Liang, 2018. ”Damage
images by border following.” Computer vision, graphics, and image
40 identification of a large cablestayed bridge with novel cointegrated
processing, 30(1), pp.32-46.
[30] J.N. Yang, S. Lin, 2005. ”Identification of parametric variations of struc-
41 Kalman filter method under changing environments.” Structural Control
tures based on least squares estimation and adaptive tracking technique.”
and Health Monitoring, 25(5), e2152.
42 [8] D. Reagan, 2017. ”Unmanned aerial vehicle measurement using three
Journal of engineering mechanics, 131(3), pp.290-298.
43 dimensional digital image correlation to perform bridge structural health
[31] GB 50144-2008, 2008. ”Standard for Appraisal of Reliability of Indus-
trial Buildings and Structures”, Prepared by the Ministry of Housing and
44 monitoring.” (Doctoral dissertation, University of Massachusetts Lowell).
Urban-Rural Development of China, Beijing, p. 38.
[9] A. Khaloo, D. Lattanzi, A. Jachimowicz, C. Devaney, 2018. ”Utilizing
45 UAV and 3D Computer Vision for Visual Inspection of a Large Gravity
[32] M.M. Adankon, M. Cheriet, 2009. ”Support vector machine.” In Ency-
46 Dam.” Frontiers in Built Environment, 4, p.31.
clopedia of biometrics (pp. 1303-1308). Springer US.
[33] R. Berwick, 2003. ”An Idiots guide to Support vector machines (SVM-
47 [10] F.N. Catbas, A.E. Aktan, 2002. ”Condition and damage assessment:
s).” Retrieved on October, 21, p.2011.
issues and some promising indices.” Journal of Structural Engineering,
48 128(8), pp.1026-1036.
[34] Z.Q. Zeng, H.B. Yu, H.R. Xu, Y.Q. Xie, J. Gao, 2008. ”Fast training
49 [11] G. Lapoutre, 2017. ”Acoustic Emission Monitoring: of Offshore Wind
support vector machines using parallel sequential minimal optimization.”
In Intelligent System and Knowledge Engineering, 2008. ISKE 2008. 3rd
50 Turbine Support Structures for Detection and Localization of Fatigue
International Conference on (Vol. 1, pp. 997-1001). IEEE.
Crack Growth.”
51 [12] N. Hallermann, G. Morgenthal, V. Rodehorst, 2014. ”Vision-based
[35] A.F. Izmailov, M.V. Solodov, 2003. ”Karush-Kuhn-Tucker systems:
52 deformation monitoring of large scale structures using Unmanned Aerial
regularity conditions, error bounds and a class of Newton-type methods.”
Mathematical Programming, 95(3), pp.631-650.
53 Systems.” In IABSE Symposium Report (Vol. 102, No. 8, pp. 2852-2859).
[36] T.B. Trafalis, H. Ince, 2000. ”Support vector machine for regression
International Association for Bridge and Structural Engineering.
54 [13] H. Kim, J. Lee, E. Ahn, S. Cho, M. Shin, S.H. Sim, 2017. ”Concrete
and applications to financial forecasting.” In Neural Networks, 2000.
55 Crack Identification Using a UAV Incorporating Hybrid Image Process-
IJCNN 2000, Proceedings of the IEEE-INNS-ENNS International Joint
Conference on (Vol. 6, pp. 348-353). IEEE.
56 ing”. Sensors, 17(9), p.2052.
57
58
59
60