Computers and Electronics in Agriculture 75 (2011) 147158

Contents lists available at ScienceDirect

Computers and Electronics in Agriculture

journal homepage: www.elsevier.com/locate/compag

Original paper

Support vector machine approach to real-time inspection of

biscuits on moving conveyor belt
S. Nashat a , A. Abdullah b , S. Aramvith c , M.Z. Abdullah a,
a
School of Electrical and Electronic Engineering, Engineering Campus, Universiti Sains Malaysia, 14300 Penang, Malaysia
b
Faculty of Information Sciences and Technology, Universiti Kebangsaan Malaysia, 43600 Bangi Selangor, Malaysia
c
Department of Electrical Engineering, Faculty of Engineering, Chulalongkorn University, 10330 Bangkok, Thailand

a r t i c l e i n f o a b s t r a c t

Article history: An intelligent system for colour inspection of biscuit products is proposed. In this system, the state-of-
Received 10 April 2010 the-art classication techniques based on Support Vector Machines (SVM) and Wilks analysis were
Received in revised form 8 September 2010 used to classify biscuits into one of four distinct groups: under-baked, moderately baked, over-baked,
Accepted 17 October 2010
and substantially over-baked. The accuracy of the system was compared with standard discriminant
analysis using both direct and multi-step classications. It was discovered that the radial basis SVM
Keywords:
after Wilks was more precise in classication compared to other classiers. Real-time implementation
Biscuit
was achieved by means of multi-core processor with advanced multiple-buffering and multithreading
Discriminant analysis
Machine vision
algorithms. The system resulted in correct classication rate of more than 96% for stationary and mov-
Multi-core processor ing biscuits at 9 m/min. It was discovered that touching and non-touching biscuits did not signicantly
Image segmentation interfere with accurate assessment of baking. However, image processing of touching biscuits was con-
Support vector machine siderably slower compared to non-touching biscuits, averaging at 36.3 ms and 9.0 ms, respectively. The
decrease in speed was due to the complexity of the watershed-based algorithm used to segment touching
biscuits. This image computing platform can potentially support the requirements of the high-volume
biscuit production.
2010 Elsevier B.V. All rights reserved.

1. Introduction tive quality judgement as to the product status. This decision is

The bakery sector can be considered to be one of the most impor-
tant sectors in food industry. Real-time inspection is very desirable
in this industrial sector since food products like biscuits are being
produced by millions each day. Like other manufacturing processes,
quality evaluation and sorting are two essential operations per-
formed routinely in biscuit production. Among the many tests that
need to be carried out on biscuits is the measurement of colour
as colour indicates quality and defect. Colour is also an important
guide as biscuits will appear more appetising when its appearance
is optimised. Consumers expect to nd a constant colour for a same
brand of biscuit. Obtaining identical biscuits are difcult even
during a short baking period. This is due to the complexity of the
baking process where biochemical reactions and physical trans-
formations give rise to biscuits with different shades of colours
(Moore, 1991). Therefore biscuit colour inspection occupies a major
role in the biscuit manufacturing, from raw dough to nished prod-
ucts. Commonly, in this industry, quality evaluation is being carried
out by human vision of some trained inspectors who make subjec-
Corresponding author. Tel.: +60 4 5996001; fax: +60 4 5941023.
E-mail address: mza@usm.my (M.Z. Abdullah).
highly variable, and the process is tedious, labour-intensive, and
subjective.
In recent years, automated machine inspection systems have
been implemented in bakery for colour inspection. The use of robots
in food processing has many advantages over their human coun-
terpart in terms of economy and performance (Abdullah, 2008).
Automated biscuit inspection would help to standardise the qual-
ity assessment of biscuit during the baking process. Although the
development of automatic visual inspection for bakery and agri-
cultural products is steadily progressing in the last decade, most
machine vision systems are facing the restriction of real-time con-
straints and high computing volume for high-speed production.
Image processing of most high-speed commercial inspectors is gen-
erally based on thresholding and pixel counting, so that they are
effective at distinguishing products with large contrast or colour
differences. In the past there have been many developments on
the use of a digital imaging technique to inspect a single, isolated
and stationary agricultural products, such as the inspection of pota-
toes and apples (Tao et al., 1995), oil palm fruits (Abdullah et al.,
2002), rice and grains (Carter et al., 2006), citrus fruits (Blasco et al.,
2007), corns (Chen et al., 2009), and beef fat (Chen et al., 2010).
However, few of these developments are able to be implemented
for high-speed sorting of moving targets at an economical feasi-

0168-1699/$ see front matter 2010 Elsevier B.V. All rights reserved.
doi:10.1016/j.compag.2010.10.010
148 S. Nashat et al. / Computers and Electronics in Agriculture 75 (2011) 147158
Fig. 1. Example of (a) non-touching and (b) touching biscuits. In both cases, top left is under-baked, top right is moderately baked, bottom left is over-baked and bottom
right is substantially over-baked.
ble cost. Image processing of a dynamically moving target requires
machine vision with high-data throughput and high-performance
microprocessors. This problem is further compounded due to the
fact that most practical applications require time consuming vision
tasks like image transformation and extraction. Fortunately, most
of these tasks have regular, repetitive computations with pre-
dictable data dependencies, making them well suited for parallel
paradigm. In order to exploit data-level parallelism in object recog-
nition, multi-core processors are more and more used in recent
years to achieve real-time object recognition (Kim et al., 2010). For
instance Zhang et al. (2008) revealed that real-time operation over
30 frame/s for the scale invariant feature transform can be achieved
by parallelising on a multi-core system; Ach et al. (2008) developed
a multi-core processors for real-time detection of trafc signs in
different images by means of the Viola-Jones algorithm; Kyo et al.
(2008) present massively parallel single instruction multiple data
(SIMD) in the image pre-processing stage of object recognition; and
Zhao et al. (2009) exploited the thread-level and data-level paral-
lelism of image segmentation technique based on Otsus algorithm.
Clearly from this review, multi-core technology is well suited to
solve some pattern recognition problems and to processing time-
varying images. In this paper, we will present the in-depth analysis
of the multi-core implementation of the SVM-based articial classi-
er for real-time inspection of stationary as well as moving objects.
Both isolated and partially occluded biscuits are investigated. The
proposed methods and procedures are applied to inspect biscuit
samples on a conveyor belt with adjustable speed.
2. Materials and methods
2.1. Biscuit colour grading
Unlike industrial and manufacturing sectors, to the best of
authors knowledge, no single standard for colour grading of bis-
cuits exists. In this industry, the colour standard of biscuits is
determined by the individual company. Colour quality control in
most biscuit producers involved human inspectors while some rely
on a more objective measurement by using a colorimeter. How-
ever, this equipment is not suitable when the sample has colour
variations. Moreover, there are many reasons for considering the
problem of touching or partial recognition in biscuit processing. For
instance it is not always possible to keep biscuits from touching or
overlapping on a moving conveyor belt. Example of colour varia-
tion in biscuits highlighting touched and untouched cases is shown
in Fig. 1. In both cases, the biscuits are categorised into four distinct
groups reecting four degrees of baking: under-baked, moderately
baked, over-baked and substantially over-baked. It can be seen in
Fig. 1 that the colour is not uniformly distributed. This is due to
the temperature variation inside the oven, causing colours of bis-
cuits to appear darker in some regions. Therefore, the challenge for
image processing software is to use this information as the basis
for colour inspection.
2.2. Machine vision system
The hardware used in this study consisted of a HP Worksta-
tion XW4600 with quad core CPU 2.5 GHz processor, 2 GB of RAM
and equipped with a colour frame grabber, an illumination sys-
tem, cable, a charge-coupled device camera, and conveyor. The
frame grabber is of the 32-bit Meteor II type manufactured by
Matrox Electronic System Limited, Canada. This device is mounted
into the PCI slot of the workstation. A high quality 3-CCD Sony
XC-003P camera was used as the image capturing device with a
sustainable speed of 25 frames per second, captured at a spatial
resolution of 640 480 pixels. The camera comes with stan-
dard C-mount type optical lens connected to a frame grabber
via 2-m external BNC cable. The camera was mounted to a test
station, which comprised of iron holder xed at the height of
45.5 cm and at an angle of 90 mounted from horizontal arm. The
test station was illuminated using a white, Stocker Yale (USA)
ultra-high-frequency uorescent ring light, model 13 plus high
frequency steady light with a maximum oscillating frequency of
85 kHz. The light bulb was tted with a continuous light inten-
sity control which allowed 10100% intensity adjustment. Standard
conveyor belt with adjustable speed was used in this study to sim-
ulate moving object detection. The schematic diagram of biscuit
inspection system showing all essential elements is depicted in
Fig. 2.
The proposed colour inspection system is divided into two main
steps: pre-processing and post-processing. In summary, the pre-
processing step includes calibration of machine vision and the
image processing part. In this work the machine vision was cali-
brated using four colour samples manufactured by Labsphere Inc.,
USA. The samples are SCS-RD-010, SCS-GN-010, SCS-BL-010, and
SCS-YW-010, corresponding to red, green, blue, and yellow colour
standards respectively. The image processing part involves image
acquisition and smoothing, RGB to HLS colour transformation, and
image segmentation. The post-processing step includes dimension-
ality reduction of the segmented objects by using Wilks analysis,
and applying support vector machine to achieve classication.
In this study the image processing was performed using the
Matrox Image Library (MIL) version MIL 9.0 with visual C++ pro-
 S. Nashat et al. / Computers and Electronics in Agriculture 75 (2011) 147158 149

BNC Cable
Iron holder
1. Under-baked
2. Moderately baked
3-CCD
3. Over-baked
Camera 4. Substantially over-baked

Illumination Biscuit
system samples 1 2

3 4

Industrial HP Frame
Industrial conveyor Workstation grabber

Fig. 2. Schematic diagram of biscuit machine vision inspection system.

gramming language incorporated. The parallel-processing is based
on Matrox multiple-buffering and multithreading techniques.
2.3. Colour space transformation
Colour is one of the most important features for biscuits assess-
ment used in this recognition system. The images of biscuits were
taken by the CCD camera and represented in the three-dimensional
RGB colour space. Unfortunately, the RGB colour space used in com-
puter graphics is device dependent, which is designed for specic
devices, e.g. cathode-ray tube (CRT) display. Hence, RGB colour has
no accurate denition for a human observer. Therefore, the effec-
tive way is to transform the RGB information sensed by the machine
vision to the Hue-Lightness-Saturation (HLS) colour space. The HLS
space was selected since it denes colour not only in the sense of
perceptual uniformity, but more signicantly, it matches to the way
that the human perceives colour (Camastra and Vinciarelli, 2008).
In this space, only the hue component h was analysed, since this
attribute directly characterises the colour properties of an object
and thus the degree of doneness of the biscuit. Hence, colour con-
version process only involved transforming the RGB information to
a single hue buffer. Indirectly, this helped to compress information
for easier colour discrimination and manageable solution. For the 8-
bit machine vision, such a transformation is given mathematically
by Gonzalez and Wintz (1987):
  
0.5[(R G) + (R B)]
h=
360 cos 1

(R G)2 (R B)(G B)
255
If B G
360
or
  
0.5[(R G) + (R B)] 255
h= cos 1
 If B < G
(R G)2 (R B)(G B) 360
Eqs. (1) and (2) yield normalised hue values in the interval [0, 255].
The equation of hue can also be expressed using trigonometry iden-
tity, but Eqs. (1) and (2) are easier to visualise and superior in terms
of hardware implementation.
of the source image was internally generated then the threshold
value was set to a minimum value between the two most statis-
tically important peaks in the histogram, on the assumption that
these peaks represent the object and the background. If the his-
togram contains only two peaks, the threshold value was set to
the minimum value between these peaks. If the histogram contains
more than two peaks, then the threshold value was set between the
two most prominent peaks. In this way the biscuit images could be
separated from the background. The area of segmented image was
then calculated in order to distinguish between touching and non-
touching biscuits. This could easily be done since the surface area of
each biscuit was relatively constant. It was heuristically discovered
that the surface area of the biscuit lies in the range between 39,870
and 39,990 pixel square. Therefore, it was possible to distinguish
touched from the untouched cases using direct thresholding. If the
area was smaller than the threshold value, then, the biscuits were
considered untouched. In this case, image processing only involved
smoothing and the removal of artefacts or noise. This was achieved
by means of morphological erosion technique. For touched cases,
image processing required four more additional steps. Sequentially
they are (i) edge detection, (ii) watershed transformation, (iii) mor-
phological erosion, and (iv) distance transformation. For the biscuit
image that differs signicantly from the background, the Sobel
operator (Sobel, 1970) was applied to detect the edge of biscuit.
Then the watershed transformation was implemented to the gradi-
ent image (Vincent and Soille, 1991). The watershed transformation
considers the gradient magnitude of an image as a topographic sur-
face. If the objects have well-dened edges, an edge detection will
produce a maximum along the edges of each object. These maxima
will dene each object as a catchment basin since they produce
(1)
a minimum in each object. A watershed transformation will then
label the catchment basins, effectively segmenting the image. After
that morphological erosion was applied to smooth the image and
(2) remove artefacts. Then, watershed and the distance transformation
were implemented in order to separate touching objects and reduce
noise due to overlapping. The blob analysis was then used to com-
pute the centre of gravity of each object. Finally, image cropping
was applied to the original image in order to obtain the region of
interest. The owchart in Fig. 3 summarises the image processing
sequence of touching and non-touching biscuits.

2.4. Image segmentation and object detection

To separate an image of biscuit from the background, the
combination of auto thresholding and watershed transformation
techniques were proposed. The auto thresholding method locates
the two peaks in the histogram corresponding to the background
and object of the image (Otsu, 1979). In so doing, the histogram
2.5. Dimensionality reduction by Wilks  analysis
Similar to other image-based articial classiers the resulting
hue vectors from the above image processing algorithm are still
too large to allow fast and accurate detection. Therefore, it is very
desirable to use fewer hue values, preferably those with highest dis-
150 S. Nashat et al. / Computers and Electronics in Agriculture 75 (2011) 147158

Start Support vectors

Margin

Grab image

Auto thresholding

Calculate area for Class (+1) Class (-1)

each object

Touching
Yes objects
Area > T
Hyperplane
Non-touching No
Edge detection of
original image Fig. 4. SVM uses hyperplane margin to separate positive from negative classes.
objects

Watershed
Morphological vectors (h , y ), where h is the hue and y {+1, 1} indicates the
transformation i i i i
erosion class label. The classication of biscuit samples can be considered
as the task of determining a classication function f : h  y using
Morphological i i
training data. Subsequently, the classication function f is used to
erosion  ) > 0, the input vector h

classify the unseen test data set. If f (h i i
is assigned to the class yi = +1, otherwise class yi = 1. This tech-
Count number of
Separate touching using nique can be extended to classication involving more than two
objects and calculate
distance transformation groups using SVM algorithm such as the Directed Acyclic Graph
centre of gravity
(Platt et al., 2000). Essentially, this algorithm works by rstly col-
lecting the data from each classier, and secondly, assembling these
Image cropping data to form a graph or a tree comprising of several nodes with each
node representing result from each binary classier. Therefore, it is
theoretically plausible to perform more than 2-class classication
End by expanding and adding more nodes to the tree. The details are
described in Section 2.7.
Fig. 3. Image processing for touching and non-touching biscuits. The threshold For the linearly separable training vectors, the classication
value T was set to 40,000 pixel square. function f has the following form:

 ) = sgn(w
f (h  + b)
 Th (3)
criminant powers. Hence, a sequence of operations is needed before i

performing classication. The most commonly used technique is

where w is the normal to the hyperplane, and b is a bias term which
Wilks selection criteria, which is also known as the stepwise dis-
criminant analysis (Rencher, 2002). In summary, this method works should satisfy the following conditions:
by examining each hue value in the set containing all hue variables
yi (w  + b) 1,
 Th i = 1, . . . , l (4)
hp , iteratively updating and deleting each variable until a subset i

containing a best hm s is produced from the full available of hs . The

decision to enter or to delete each variable is based on F-statistic In operation, the SVM attempts to nd the optimal hyperplane
together with F-to-enter and F-to-remove values. The mathemat- that maximises the separation margin between positive and neg-
ative samples. The margin is 2/||w||, thus the optimal separating
ical derivations and technical description of Wilks analysis are
hyperplane is the one minimising ((1/2)w  subject to con-
 T w),
described in our earlier publication (Abdullah et al., 2001).
straints (4), which is a convex quadratic programming problem.
For the non-linear case when the hyperplane can only separate
2.6. Classication using SVM
the groups algebraically but not strictly separate them, a differ-
 are mapped
ent strategy is needed. In this case, the input vectors h
Support vector machine is a supervised learning algorithm i
into a high dimensional feature space H by using suitable kernel
developed by Vapnik and others (Cortes and Vapnik, 1995; Vapnik, ,h  ). Two popular kernels which are the polynomial
function k(h i j
1998). It has been used extensively for a wide range of applications
kernels and the Gaussian radial basis function (RBF) were used in
with excellent empirical performance (Karimi et al., 2006; Adankon
the study. Respectively, they are dened as:
and Cheriet, 2009). The fundamental idea of SVM is to construct a
hyperplane as the decision line, which separates the positive (+1) d
 ,h
 ) = (h
h
classes from the negative (1) ones in the binary classication with k(hi j i j + 1) (5)
the largest margin. This is illustrated in Fig. 4.
The classication of biscuits into four quality levels can be con- and
sidered as a collection of binary categorisation problems. Suppose   h
 ||2

there are l samples of biscuit in the training data corresponding  ) = exp
 ,h ||hi j
k(h (6)
to the two groups, and each sample is denoted by a vector h , i j
2 2
i
(i = 1, . . ., l), which represents the selected colour features of the
biscuit. Therefore the training data can be represented by the set of where d is the degree of polynomial kernel.
 S. Nashat et al. / Computers and Electronics in Agriculture 75 (2011) 147158
The classication function then has the following form in terms
of kernels:
 
l
 = sgn
f (h)  , h)
i yi k(h  +b
i
i=1
where k is a kernel function, b is a bias term, and i is the Lagrange
multiplier coefcient obtained by solving the quadratic program-
ming problem. Based on KarushKuhnTucker (KKT) conditions
of quadratic programming, only those h  with the corresponding
i
coefcients i > 0 in Eq. (7) are called support vectors (SVs), which
are used to make the decision in the classication problem. Platt
(1999) describes the techniques and procedures involved in KKT
programming. Interested readers are referred to this reference for
further details. Finding coefcients i is equivalent to maximise the
function as follows:
l
1
l l
W () = i  ,h
yi yj k(h  )
i j i j
2
i=1 i=1 j=1
Under the constraints:
0 i C, (i = 1, . . . , l) and yi i = 0
i=1
where C is a non-negative regularisation parameter.
For some data sets, the SVM may not be able to nd a separating
hyperplane in feature space. Therefore a regularisation parame-
ter C was used to control the trade-off between maximising the
margin and minimising the training error. Entirely specifying a
support vector machine requires setting two parameters: the ker-
nel function and the magnitude C for violating the soft margin.
The selections of these parameters are dependent on the specic
modelled data. In this study the sequential minimum optimisation
(SMO) algorithm was used to train the SVM algorithm and SVs are
analytically obtained by solving quadratic programming problem.
The details of SMO algorithm are described elsewhere (Platt, 1999).
2.7. Multi-class SVM with directed acyclic graph
For the problem of N-class SVM (N = 4), Platt et al. (2000) pro-
posed a novel algorithm for multiclass classication based on
placing two-class classiers into nodes of a directed acyclic graph
(DAG). The graph used a rooted binary DAG, a root DAG that has
a unique node with no arrows pointing into it, and other nodes
which have either 0 or 2 arcs leaving them, to be a class of function
in classication tasks, as shown in Fig. 5.
This algorithm constructs N(N 1)/2 internal nodes, each one
labelled with an element of a Boolean function. The DAG can be
performed using a list, where each node eliminates one class from
the list. The implementation is initiated with a list of all classes. A
test point is evaluated against the decision node that corresponds
to the rst and last elements of the list. If the node prefers one of
the two classes, the other class is eliminated from the list. The DAG
continues with testing until only one class remaining in the list, at
this stage, the algorithm will terminate and the state of the list is
the entire state of the system. The classication of biscuit samples
into four groups was implemented by using DAG algorithm. Here,
we refer to the decision node distinguishing groups i and j as the
i j node. Such a DAG is obtained by training each i j node only
on the subset of training points labelled by i or j. The nal class
decision is derived by using the DAG architecture as shown in Fig. 5.
The details of the DAG SVM are described elsewhere (Nashat and
Abdullah, 2010).
151
1
1. Under-baked
2 1vs4
2. Moderately baked
3
4
3. Over-baked
(7) not 1 not 4 4. Substantially over-baked
2 1
3 2vs4 2 1vs3
4 3
not 2 not 4 not 1 not 3
3 3vs4 2 2vs3 1 1vs2
4 3 2
3 1
(8)
4 2
Fig. 5. The acyclic graph for 4-group classication via DAG strategy. Here 1 indi-
cates under-baked, 2 moderately baked, 3 over-baked and 4 substantially
over-baked. The abbreviation vs stands for versus.
(9) 2.8. Experimental procedures
In this colour inspection system 400 samples of biscuit were
inspected and categorised to either training or independent test
sets. The rst 200 samples were allocated to the training set, com-
prising 50 samples for each group. These samples were then imaged
and latter used to train the machine vision system for colour inspec-
tion and recognition. Another 200 samples were inspected by the
inspectors and categorised as the independently set, comprising 50
samples for each group. This set was used to independently assess
the accuracy of the machine vision system. Initially, the captured
training set was converted from RGB to HLS colour space using Eqs.
(1) and (2). After that, image segmentation algorithm described
above was implemented to crop 80% of biscuit sample producing a
region of interest. Then the resulted hue population ranging from 0
to 59 was used as input variables to the machine vision system. In
this stage Wilks algorithm was invoked with several threshold
options in order to extract the maximum number of potent vari-
ables. Once the optimum or potent hue subset has been located,
the system established the SVM classier by scaling each input
hue value and applying SMO algorithm. The colour inspection sys-
tem rst mapped the scaled hue vector to high dimensional feature
space by using one of the kernel functions in Eqs. (5) and (6). Then
the SMO algorithm was implemented to train the system and estab-
lished criteria from the image multiple samples by calculating all
variables useful in classication. Finally, the classication output
is then processed using DAG algorithm which uniquely classied
sample into one of the four groups.
After the training was completed, the machine vision system
was switched from training to test mode. In this mode, the system
was used to reclassify the training samples in order to assess its
accuracy, repeatability and consistency. After this, the system was
used to classify all samples belonging to the independent test set.
In this way, the variability in grading between inspectors could be
compared, and the accuracy between machine vision system and
inspectors could also be investigated. All biscuit samples were man-
ually classied and sorted by hand using semi-trained inspectors
following the same methods and procedures discussed previously.
In this study, results obtained from DAG algorithm were com-
pared with DA based classier using both full and reduced features.
After that, the accuracy of the system was assessed on moving test
samples using different speeds in real-time.
152 S. Nashat et al. / Computers and Electronics in Agriculture 75 (2011) 147158
2.9. Parallel image processing
To achieve real-time object recognition with considerably lower
power, we considered the multi-core architecture comprising of
4 processing units with 6 MB of level 2 cache memory. The basic
idea is to partition the image into specic parts and simultane-
ously processing them on different processors. Respectively, the
multiple-buffering technique and the multithreading strategy are
implemented to manage the image capturing cycle and synchronise
the execution of the program in parallel threads. These methods are
described in the following sub-sections briey.
2.9.1. Multiple-buffering technique (MBT)
The key component in reaching real-time implementation is the
Matrox dynamic simulator featuring the multiple-buffering strat-
egy. This technique involves grabbing images and storing them into
buffers while processing previously grabbed images. In this way
MBT optimises wasted CPU cycles and reduces the memory cost
due to managing inconsistent frame complexity. This technique
allows images to be grabbed and processed concurrently. It also
permits for robustness when there is variability in processing time
from image to image (Hornberg, 2006).
In this study, multiple buffering is performed by rst allocating
list of buffers used to hold series of sequentially grabbed images,
and to allow for the processing of the buffers concurrently as they
are being grabbed. This grabbing is initiated by a start signal and
the process continued until the stop signal is activated. In this case,
grabbing is implemented asynchronously which allows other pro-
cessing functions to be executed while images are being acquired.
The technique goes round-robin through the list of buffers in a con-
tinuous acquisition. However, care must be taken to ensure that the
average time to process a frame is not greater than the frame rate of
the camera. Therefore, the design in Fig. 6 is adopted showing the
sequence of image grabbing and storing. To avoid memory access
conicts between the video interface and the worker threads, two
buffers of at least one for grabbing and one for processing are
required. In this experiment, n (2 n 22) buffers are used to hold
the images and the processing function is called n times to process
them. In this way complex image processing tasks can be realised
as speed equals to the frame rate of the acquisition hardware.
2.9.2. Multithreading technique
Multithreading technique and multi-core processing are two
powerful strategies to take advantage of parallelism in the appli-
cations in order to improve the systems performance. Computer
vision applications have the potentials to run much faster by
utilising more powerful multi-core systems. On the other hand,
algorithms and applications must be tuned to allow multi-core
processor to exploit their inherent parallelism. Multithreading
technique has the ability to perform multiple operations simulta-
Grab 1 Grab 2
Processing 1 Processing 2
Task 1 Task 2 Task 3 Task 4
Thread 1 Thread 2 Thread 3 Thread 4
Fig. 7. Structure of queuing tasks of the inspection system and with four SVMs threads.
Frame Buffer 1
grabber
Processing 1
Camera
Buffer n-1
Processing n-1
Buffer n
Processing n
Fig. 6. Block diagram of the multiple-buffering strategy for real-time grabbing and
image processing.
neously in the same process. This can be done by creating different
threads to ensure sequential execution of operations within the
same thread while allowing simultaneous yet independent exe-
cution of operations in other threads. Here, the multithreaded
programming is implemented using MIL tools for multiprocess-
ing.
In this work, multithreading was implemented by assuming that
there are maximum of four samples in the eld view. It is possible
to have more than four samples but the image resolution would
be signicantly reduced and the classication would be less accu-
rate. The algorithm works by rstly extracting important features
from each sample in the image, and secondly, passing this infor-
mation simultaneously to all distinct SVM classication threads in
parallel. After the classication has completed, each thread gave
a result corresponding to each sample in the image. Fig. 7 shows
the structure of queuing tasks of four-threads SVM used in this
study.
Prior to multithreading, the image is segmented into few sub-
images corresponding to the number of biscuit sample in the eld
of view. Resultantly, the background is also ltered leaving the
desired features only. The multi-core processor used the feature
space to count the number of sample, and resultantly, created
the number of compute threads in the node. One of the compute
threads is assigned as a master, which allocates work to all other
compute threads. Each thread independently performs the image
Grab n
Processing n
Task 1 Task 2 Task 3 Task 4
Thread 1 Thread 2 Thread 3 Thread 4
 S. Nashat et al. / Computers and Electronics in Agriculture 75 (2011) 147158 153

the multi-threaded data ow structure implemented in this

Start study.

3. Results and discussion

Grab image
3.1. Colour inspection

The edge images corresponding to non-touching and touching

Store image into buffer biscuits in Fig. 1 are shown in Fig. 9. The formation of four dis-
jointed regions corresponding uniquely to four biscuits in the scene
is clearly visible in this gure. A cross-hair indicates the centre of
each biscuit in the image. This result indicates that auto threshold-
Image segmentation and object
ing performs well for non-touching biscuits while the watershed
detection transformation is effective in dealing with touching biscuits. Fol-
lowing segmentation, the hue values of each biscuit are extracted
and then plotted as a bar graph or histogram. Example of the hue
If no. of histogram is shown in Fig. 10. This gure is plotted by averaging
No
over 50 samples of biscuits for each group category. The histogram
object 1
shows a typical hue distribution of biscuits which contains many
local maxima and minima. It was also heuristically discovered that
Yes the frequency distribution of hue depended strongly on the back-
Master thread crops the ground colour, producing an image with optimal contrast when
grabbed image into sub-images background hues ranged from green to blue. Hence, the cyan back-
ground corresponding to the hue value 180 was used to capture
the biscuits images.
Master thread forks several Careful examination of Fig. 10 reveals some unique characteris-
threads one for each object tics of biscuits. First, they are not truly brown, based on quantitative
hue values, but rather show colours that tend toward yellow and
light or dark brown. Clearly, the colour of the under-baked group
Each thread picks a task and is skewed towards the yellow and light brown region, peaking at
processes the sub-image 41; whereas the colours of the over-baked and substantially over-
baked groups are skewed towards the dark-brown region. For these
two groups, the hue reached maximum values at 21 and 16, respec-
Each thread writes result into tively. The hue distribution of moderately baked group lies between
memory light and dark brown region, peaking at approximately 20. Sec-
ondly it can be seen that the hue distributions for different groups
of biscuits overlap each other; no single threshold exists that can
Each thread joins Master thread distinctively separate one group from another. Thirdly, the hue
values for the four types of biscuits are approximately normally
distributed, ranging from 16 to 43 for the under-baked group, from
Master thread 14 to 43 for the moderately baked group, from 12 to 30 for the over-
baked group, and from 4 to 23 for the substantially over-baked
displays the results
group. This is in agreement with expected values, given that hue
values that range from 0 to 60 fall between the red and yellow
regions.
Free buffer However, not all these values are necessarily independent or
uncorrelated variables. In fact, some of them are redundant and do
not have signicant discriminating power. Therefore, these vari-
ables need to be eliminated, since such an elimination will not only
No Stop reduce the dimensionality of independence of the variables among
themselves, but also, and more importantly, it produces a best sub-
grading
set of variables leading to an optimal rate of correct classication
(Smith and Nakai, 1990; Abdullah et al., 2004).
Yes
End 3.2. Wilks  analysis

Fig. 8. Flowchart of a real-time colour inspection system based on multithreading
technique for parallel processing.
post-processing like dimensionality reduction and object recogni-
tion. Results are assembled by the master thread and displayed on
the monitor screen or written into a text le if necessary. After
completion of the image inspection task, the master thread frees
all buffers and the threads. Then, it enters a stand-by mode and
ready to process next image. The owchart in Fig. 8 summarises
The important element in this colour inspection system is to
locate a hue subset that contains signicant input variables. Refer-
ring to Fig. 10, the hue values of four different groups of biscuit
samples fall in the range from 0 to 59. Hence, altogether there are 60
variables, which are statistically independent and useful for analy-
sis. These variables were submitted to Wilks analysis in order to
detect both the insignicant and highly signicant hue variables. In
the analysis, the F-to-remove and the F-to-enter variables were set
to default values of 2.71 and 3.84, respectively. The algorithm iter-
atively took eight steps to converge, producing a subset containing
154 S. Nashat et al. / Computers and Electronics in Agriculture 75 (2011) 147158

Fig. 9. Segmented biscuits and their centres of gravity corresponding to image in Fig. 1: (a) non-touching biscuits and (b) touching biscuits. Here a(i) and b(i) are edge images
after auto thresholding; b(ii) is the edge image after watershed-based segmentation; a(ii) and b(iii) are the centres of gravity of a(i) and b(ii), respectively.

8 principal hues. This corresponds to a loss of 86.67% in variation. Using these potent variables, the machine vision system was
The 8 principal components produced here were: trained using biscuit samples in the training set. The classication
efciency of the system using 8 principal hues was examined by
hs = (19, 20, 23, 29, 39, 40, 42, 43) (10) studying the Mahalanobis distances, shown canonically in Fig. 11.
 S. Nashat et al. / Computers and Electronics in Agriculture 75 (2011) 147158 155

Fig. 10. Hue distributions of four different groups of biscuits averaged over 50 samples for each group category.

Table 1
Real time classication results of independent test samples corresponding to different conveyor speeds using SVM-R classier with Wilks method.

Conveyor speeds Classication results

Under-baked Moderately baked Over-baked Substantially over-baked Average accuracy

10 m/min 94% 98% 90% 100% 95.5%

9 m/min 96% 98% 88% 100% 95.5%
8 m/min 96% 98% 90% 100% 96%

Table 2
Total processing time of non-touching biscuits comparing sequential and parallel processing for different classiers.

Classiers Sequential processing Parallel processing

Direct method Wilks method Direct method Wilks method

No. of samples Processing time (ms) No. of samples Processing time (ms) No. of samples Processing time (ms) No. of samples Processing time (ms)

SVM-R 4 29.44 4 28.26 4 9.68 4 9.34

SVM-P 4 29.51 4 28.19 4 9.70 4 9.30
DA 4 24.14 4 24.10 4 8.87 4 8.25

Functions 1 and 2 in this gure refer to the canonical discrim-
inant functions derived from the canonical correlation analysis
which were used to examine the relationship between the variables
(Morison, 2005).
This graph clearly demonstrates the formation of four major
groups. It shows the existence of the hyperplane which strongly
separates under-baked and substantially over-baked groups. In
contrast, the hyperplane separation is weak for over-baked and
moderately baked groups. It can also be seen that these groups
are disjointed and convex, and the hyperplane separation is alge-
braically non-linear. It was also observed that the non-linearity
increased when independent test samples were used.

Table 3
Total processing time of touching biscuits comparing sequential and parallel processing for different classiers.

Classiers Sequential processing Parallel processing

Direct method Wilks method Direct method Wilks method

No. of samples Processing time (ms) No. of samples Processing time (ms) No. of samples Processing time (ms) No. of samples Processing time (ms)

SVM-R 4 62.44 4 61.35 4 36.70 4 36.50

SVM-P 4 62.48 4 61.30 4 36.86 4 36.43
DA 4 58.28 4 58.22 4 36.20 4 35.93
156 S. Nashat et al. / Computers and Electronics in Agriculture 75 (2011) 147158
Fig. 11. Discrimination separations by four biscuit groups produced by using Wilks
method. Data are averaged from 50 samples of each group.
Fig. 12. Classication results of training samples comparing SVM and DA classiers
by using Wilks and direct methods.
3.3. Classiers analysis
The experimental results on colour recognition of the SVM and
DA algorithms are summarised in Figs. 12 and 13, corresponding
to the results from classication of the training and independent
test samples, respectively. In these gures, results from the direct
application of SVM and DA are also presented, serving as reference
to Wilks analysis. In direct method, all hue variables ranging from
Fig. 13. Classication results of independent test samples comparing SVM and DA
classiers by using Wilks and direct methods.
0 to 59 are used for classication, whereas 8 principal hue values
are selected with Wilks method. In this experiment, polynomial
and RBF SVM classiers are trained and tested using the kernels
dened in Eqs. (5) and (6).
A range of parameters for the polynomial and RBF SVM classi-
ers are selected and tested to eliminate any biased performance of
the SVM that may be caused by inappropriate choice of parameters.
In this experiment, the best parameter of polynomial SVM
(SVM-P) is selected to be degree d = 3 for direct and Wilks meth-
ods in all trials. In this work, the sigma () and penalty (C) variables
were chosen experimentally through a search algorithm. In so
doing, the seed values of 0.001 and 1 respectively was used for C and
. These seed points were heuristically decided. During searching,
the algorithm solved Eq. (7) repeatedly and the C and  values that
resulted in the best accuracy were selected. In this case the sigma
values selected for RBF SVM (SVM-R) were 58 and 2 for direct and
Wilks methods, respectively. Meanwhile the best penalty vari-
able C was 1000. This penalty parameter improved the results of the
system and solved non-separable data set by using a soft margin.
Results are shown in Figs. 12 and 13.
The results using training samples produced high performance
accuracy which is clearly seen in Fig. 12. Here SVM-R and SVM-P
classiers yielded best classication rate of 100% with direct and
Wilks methods, whereas DA classiers yielded 99.5% and 99% by
using the same methods, respectively. Results using test samples
produced the same trend. It can be seen from Fig. 13, that the SVM-
R classier with Wilks method, resulted in the best classication
rate of 96.5%, compared 95.5% SVM-P and 94% DA. The results of
direct method in Fig. 13, presented a best performance for SVM-R
with correct rate 95%, comparing with the SVM-P and DA classiers,
which yielded 92% and 93%, respectively. In general the classica-
tion rates for SVM-R are consistently higher compared to SVM-P or
DA.
These results also indicated that discarding of variables, which
are correlated highly with those retained, guarantees that the
deleted variables are, in fact, redundant in the variable space.
Hence, the classiers after Wilks analysis is more precise in
classication compared with direct application of SVM and DA algo-
rithms.
Results from these experiments also demonstrated the difcul-
ties in machine vision distinguishing over-baked from moderately
baked samples. Referring to Fig. 13, the average success rate for
the over-baked samples with Wilks method is 90% compared to
96.5% for other classes. These results are consistent with the canon-
ical plot in Fig. 11 which showed that the hyperplanes separating
over-baked samples to other samples particularly the moderately
baked ones are relatively small. Considering this and the fact that
misclassication of the vision system varied from 0 to 6% and aver-
age on the order of 3% for individual class, the above results may
therefore suggest that machine vision classication may be equal or
better than semi-trained judgement in accuracy and consistency.
Comparing Figs. 12 and 13, it can also be seen that the good
classication results of the training samples are not reproduced in
classication of the test samples. Therefore the 100% or 99% average
success rate of training samples is articial, since the rate of cor-
rect classication is based against the best subset in this system.
In other words, the machine vision system seemed to memorise
the colour patterns in the training set instead of generalising them.
This observation is also in agreement with results on classication
and quantication of adulterants in marple syrup by multivariate
analysis of Fourier transforms infrared and near-infrared proles
(Paradkar et al., 2002).
Based on the stationary classication results, SVM-R classier
with Wilks analysis was chosen to inspect moving biscuits placed
on the conveyor belt. The experiment was conducted on biscuits
moving at three different speeds: (i) 8 m/min, (ii) 9 m/min and
 S. Nashat et al. / Computers and Electronics in Agriculture 75 (2011) 147158
(iii) 10 m/min. In this inspection, the machine vision system was
trained and tested using the same samples and illumination set-
up as those used in the stationary classication. Here, the biscuit
images were acquired continuously from the centre of the eld of
view by detecting the centre of each moving sample.
Table 1, shows the results of the dynamic classication using
independent test samples corresponding to the different conveyor
speeds. Expectedly, it can be seen from this table that there is a
slight decrease in the accuracy compared to stationary inspection.
The average correct classication attained by the system for mov-
ing samples is consistently higher than 95.5%. There is no signicant
change in the accuracy when the speed of the conveyor is varied
from 8 m/min to maximum adjustable speed of 10 m/min. Further-
more, the same correct classication is observed when the system
is used to inspect touching biscuits. In addition to accuracy, the
total processing times were also considered. These are tabulated in
Tables 2 and 3 for non-touching and touching biscuits, respectively.
It can be seen from these tables that the multi-threaded program
is more efcient when processing non-touching biscuits, resulting
in the speedup of more than 3.0. The average speedup of touching
biscuits is less than 1.7. The maximum speedup attainable by this
system is approximately 4. The slightly less than the ideal speedup
is attributed to the fact that the image segmentation procedure is
still performed in sequential. On average the utilisation of CPU cores
is measured as 65%. This result indicates multithreading improves
parallelism and the multiple processing cores are highly utilised.
Results comparing single and multi-core processing are also sum-
marised in Tables 2 and 3. In both cases, the DA is slightly faster than
SVM even though SVM is more superior in terms of accuracy and
performance. On average the accuracy of SVM-R stands at 96.5%
compared to 94% for DA. Therefore, the slight reduction in speed
of the former is compromised with its increase in accuracy. It can
also be seen from these tables that the image processing of touch-
ing biscuits is at least 4 times longer than non-touching biscuits. In
addition to the auto thresholding, image processing of touching bis-
cuits involved the watershed-based segmentation technique which
is wellknown for its extreme computational demand (Vincent and
Soille, 1991). Hence, the image processing for touching biscuits is
more intensive compared to non-touching biscuits. Based on SVM-
R classication results running on four samples, the vision system
should have a theoretical throughput of 428 biscuits/s or 110 bis-
cuits/s for non-touching and touching cases, respectively. These
results indicate that the total inspection process of several samples
can be realised as speed less than the frame rate of the acquisition
hardware. In summary the algorithm is able to perform inspec-
tion at 10 m/min with correct classication of more than 95.5%.
This image processing capacity has the potential to fully satisfy the
general requirements of a fast biscuit manufacturing.
4. Conclusion
Quality evaluation of bakery products has a major role in food
industry. In this study a colour-based inspection system has been
proposed and shown to have a good potential to classify biscuit
samples using SVM and DA classiers. Results from this study indi-
cate that SVM generally performs better than DA for both direct
and Wilks classications. It was discovered that the SVM-R
after Wilks resulted in correct classication of 96.5% which is
the highest compared to other classiers. Parallel processing was
implemented based on Matroxs multiple-buffering technique and
multithreading programming. These features combined provide
an efcient communication scheme, resulting in the increased in
image processing speedups by a factor of 3. There was no notice-
able change in the accuracy when the system was used to inspect
moving biscuits. Like the stationary case, the highest classication
157
was recorded by SVM-R at a speed of 10 m/min. This articial classi-
er has a potential for use in routine inspection of biscuits and other
bakery products. This articial classier has a potential to increase
inspection speeds beyond human operators when implemented on
a multi-core processor. It is suitable for use in high volume produc-
tion like biscuit processing considered in this example.
Acknowledgements
This work has been supported by the Universiti Sains Malaysia
Research University Grant 814012. The authors also would like to
acknowledge partial support of this research through the AUN-Seed
Net Program 449/USM/2009.
References
Abdullah, M.Z., Guan, L.C., Mohd-Azemi, B.M.N., 2001. Stepwise discriminant anal-
ysis for colour grading of oil palm using machine vision system. Transactions of
IChemE 79 (C), 223231.
Abdullah, M.Z., Guan, L.C., Mohamed, A.M.D., Noor, M.A.M., 2002. Colour vision
system for ripeness inspection of oil palm elaeis guineensis. Journal of Food
Processing and Preservation 26 (3), 213235.
Abdullah, M.Z., Rahman, A.S.A., Shakaff, A.Y.M., Noor, A.M., 2004. Discrimination
and classication of Eurycoma longifolia Jack in medical food by means of a
DSP-based electronic taste sensor. Transactions of the Institute of Measurement
and Control 26 (1), 1939.
Abdullah, M.Z., 2008. Quality evaluation of bakery products. In: Sun, D.-W. (Ed.),
Computer Vision Technology for Food Quality Evaluation. Elsevier, London, pp.
481522.
Ach, R., Luth, N., Techmer, A., 2008. Real-time detection of trafc signs on a multi-
core processor. In: Proceedings of the IEEE Intelligent Vehicles Symposium,
Eindhoven, The Netherlands, pp. 307312.
Adankon, M.M., Cheriet, M., 2009. Model selection for the LS-SVM application to
handwriting recognition. Pattern Recognition 42 (12), 32643270.
Blasco, J., Aleixos, N., Molt?, E., 2007. Computer vision detection of peel defects in
citrus by means of a region oriented segmentation algorithm. Journal of Food
Engineering 81 (3), 535543.
Camastra, F., Vinciarelli, A. (Eds.), 2008. Machine Learning for Audio Image and Video
Analysis Theory and Applications. SpringerVerlag, London.
Carter, R.M., Yan, Y., Tomlins, K., 2006. Digital imaging based classication and
authentication of granular food products. Measurement Science and Technology
17 (2), 235240.
Chen, X., Xun, Y., Li, W., Zhang, J., 2009. Combining discriminant analysis and neural
networks for corn variety identication. Computers and Electronics in Agricul-
ture 71 (1), S48S53.
Chen, K., Sun, X., Qin, Ch., Tang, X., 2010. Colour grading of beef fat by using computer
vision and support vector machine. Computers and Electronics in Agriculture 70
(1), 2732.
Cortes, C., Vapnik, V., 1995. Support-vector network. Machine Learning 20 (3),
273297.
Gonzalez, R.C., Wintz, P.A. (Eds.), 1987. Digital Image Processing. Addison-Wesley,
Reading, MA, USA.
Hornberg, A. (Ed.), 2006. Handbook of machine vision. Wiley-VCH Verlag GmbH &
Co. KGaA, Germany.
Karimi, Y., Prasher, S.O., Patel, R.M., Kim, S.H., 2006. Application of support vector
machine technology for weed and nitrogen stress detection in corn. Computers
and Electronics in Agriculture 51 (12), 99109.
Kim, J.-Y., Oh, S.-J., Lee, S.-J., Kim, M.-S., Oh, J.-W., Yoo, H.-J., 2010. An attention
controlled multi-core architecture for energy efcient object recognition. Signal
Processing: Image Communication 25 (5), 363376.
Kyo, S., Okazaki, S., Koga, T., Hidano, F., 2008. A 100 GOPS in-vehicle vision processor
for pre-crash safety systems based on ring connected 128 4-way VLIW process-
ing elements. In: Proceedings of the IEEE Symposium on VLSI Circuits, Hawaii,
USA, pp. 2829.
Moore, C.A. (Ed.), 1991. Automation in the Food Industry. Glasgow, Blackie, New
York.
Morison, D.F. (Ed.), 2005. Multivariate Statistical Methods. Thomson Brooks, Cali-
fornia.
Nashat, S., Abdullah, M.Z., 2010. Multi-class colour inspection of baked foods featur-
ing support vector machine and Wilks analysis. Journal of Food Engineering
101 (4), 370380.
Otsu, N., 1979. A threshold selection method from gray-level histograms. IEEE Trans-
actions on Systems Man and Cybernetics 9 (1), 6266.
Paradkar, M.M., Sivakesra, S., Irudayaraj, J., 2002. Discrimination and classication of
adulterants in marple syrups with the use of infrared spectroscopic techniques.
Journal of Science of Food and Agriculture 83 (7), 714721.
Platt, J.C., 1999. Fast training of support vector machines using sequential mini-
mal optimisation. In: Schkopf, B., Burges, C.J.C., Smola, A.J. (Eds.), Advances in
Kernel Methods: Support Vector Learning. MIT Press, Cambridge, MA, pp. 185
208.
158 S. Nashat et al. / Computers and Electronics in Agriculture 75 (2011) 147158
Platt, J.C., Christiani, N., Shawe-Taylor, J., 2000. Large margin DAGs for multiclass
classication. Advances in Neural Information Processing Systems, vol. 12. MIT
Press, Cambridge, pp. 547553.
Rencher, A.C. (Ed.), 2002. Methods of Multivariate Analysis. John Wiley & Sons,
Canada.
Smith, M.A., Nakai, S., 1990. Classication of cheese varieties by multivariate analysis
of HPLC proles. Canadian Institute of Food Science and Technology Journal 23
(1), 5358.
Sobel, I.E., 1970. Camera Models and Machine Perception, AIM-21. Stanford Articial
Intelligence Lab, Palo Alto, USA.
Tao, Y., Heinemann, P.H., Varghese, Z., Morrow, C.T., Sommer, H.J., 1995. Machine
vision for colour inspection of potatoes and apples. Transactions of the American
Society of Agricultural Engineers 38 (5), 15551561.
Vapnik, V. (Ed.), 1998. Statistical Learning Theory. John Wiley & Sons, New York.
Vincent, L., Soille, P., 1991. Watersheds in digital spaces an efcient algorithm based
on immersion simulations. IEEE Transactions on Pattern Analysis and Machine
Intelligence 13 (6), 583598.
Zhang, Q., Chen, Y., Zhang, Y., Xu, Y., 2008. SIFT implementation and optimisation
for multi-core systems. In: Proceedings of the IEEE International Symposium on
Parallel and Distributed Processing IPDPS, FL, USA, pp. 18.
Zhao, J., Yang, Y.M., Li, G., 2009. Real-time image processing system based on
multi-core processor. In: Proceedings of the IEEE 3rd International Symposium
on Intelligent Information Technology Application IITA, Nanchang, China, pp.
329332.