Escolar Documentos
Profissional Documentos
Cultura Documentos
DETECTION AND
TRACKING
HAND DETECTION,
TRACKING AND
GESTURE
RECOGNITION
SUBMITTED TO-
( 2k14/EC/013 )
Miss Shikha
I.
INTRODUCTION
Thus by performing this all the noise and a few spots inside a
region are all cleaned there by we get a much clearer motion
change image. The resulting image that we get in our case after
performing these image processing techniques is shown in figure
6.
D. Contour Extraction
Contour extraction is performed using OpenCVs inbuilt
edge extraction function. It uses a canny filter to get a list
of contours. It works by convoluting a filter with the image so
that gradually changing components gets cancelled out while
sudden changing components like at the edge or borders are
enhance. Thus the edges become visible. It does the following
steps to extract the edges.
Filter out any noise. The Gaussian filter is used for this
purpose.
Apply a pair of convolution masks (in x and y directions)
Find the gradient strength and direction. The direction is
rounded to one of four possible angles (namely 0, 45, 90
or 135)
Non-maximum suppression is applied. This removes
pixels that are not considered to be part of an edge.
Hence, only thin lines (candidate edges) will remain.
Canny does use two thresholds (upper and lower):
If a pixel gradient is higher than the upper threshold,
the pixel is accepted as an edge
If a pixel gradient value is below the lower threshold,
then it is rejected.
If the pixel gradient is between the two thresholds,
then it will be accepted only if it is connected to a
pixel that is above the upper threshold.
This gives us a list of set of points, each set representing
a contour. We can filter out small contours as they will be
noise. So we set a threshold for the area for the contour about
which we consider for the following steps. The image
showing the marked contours of my moving hand is shown
on the next page.
Fig. 7. Contours
points. Thus we get the center of the palm. Due to noise, this
center keeps jumping, so to stabilize it, we take an average
over a few iterations. Thus the radius of the palm is an
indication of the depth of the palm and we know the center of
the palm. Using this we can track the position of the palm in
real time and even know the depth of the palm using the radius
using which we can even estimate the distance of the palm from
the camera. The next challenge is detecting the no of fingers.
We use a couple of observations to do this. For each maxima
defect point which will be the finger tip, there will be 2
minimal defect points to indicate the valleys. Hence the maxima
and the 2 minimal defects should form a triangle with the
distance between the maxima and the minima to be more or
less same. Also the minima should be on or pretty close to
the circumference of the palm. We use this factor too. Also
the ratio of the palm radius to the length of the finger triangle
should be more or less same. Hence using these properties, we
get the list of maximal defect points that satisfy the above
conditions and thus we find the no of fingers using this. If no
of fingers is 0, it means the user is showing a fist.
III. RESULT
Thus we can detect the no of fingers, the location of palm
and its depth. Using this we can construct systems that detect
gestures, track moving objects in sensitive areas or make a
camera follow the moving object. Due to the heavy noise
involved in regular cameras, the location of the palm and the
radius are very jittery and varies continuously, but the average
is the same and is suitable for use. Thus we need to average
it over a few frames. Thus rapid gestures are not suited for the
system that is built, so the system suffers from lag but at the
same time suitable for many applications as these lags do not
matter much in practical applications. Also the background
needs to be stationary (slight to moderate motion is tolerable
due to the filtering techniques used) and of contrast colour (to
distinguish the foreground from the background). The hand
needs to be facing straight at the camera, as tilts cause the
palm to obtain elliptical shape which is not well detected by
the system. The system is very well suited for performing
gestures like pause, play, grab, drop, based on finger
location, etc.
IV. CONCLUSION
Thus a program was created that was able to detect our
hands, track them in real time and perform some gesture
recognition with simple signal processing performed on
images obtained from a regular laptop web-camera.
Reasonable accuracy and stability was obtained which can
be used for steady and simple gestures to perform tasks.
REFERANCES
[1] P. KaewTraKulPong and R. Bowden, An Improved Adaptive Background
Mixture Model for Real time Tracking with Shadow Detection, d European
Workshop on Advanced Video Based Surveillance Systems, AVBS01.
Sept 2001.
[2] Aniket Tatipamula, Hand Gesture
using
OpenCV,
http://anikettatipamula.blogspot.in/2012/02/hand-gesture-usingopencv.html
[3] Structural Analysis and Shape Descriptors, OpenCV documentation.
http://opencv.itseez.com/2.4/modules/imgproc/doc/structural_analysis_an
d_shape_descriptors.html
[4] www.stackoverflow.com
[5] Beginner's guide to understand Fingertips counting using convexity
defects in OpenCV.
http://www.codeproject.com/Articles/782602/Beginners-guide-tounderstand-Fingertips-counting
[6] Arduino and C++ (for Windows)
http://playground.arduino.cc/Interfacing/CPPWindows
6
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
7
123
ch[i]='1';
124
else if(x==2)
125
ch[i]='2';
126
else if(x==3)
127
ch[i]='3';
128
else if(x==4)
129
ch[i]='4';
130
else if(x==5)
131
ch[i]='5';
132
else if(x==6)
133
ch[i]='6';
134
else if(x==7)
135
ch[i]='7';
136
else if(x==8)
137
ch[i]='8';
138
else if(x==9)
139
ch[i]='9';
140
angle/=10;
141
i--;
142
}
143
angle=(180*palm_center.x)/frameSizeX;
144
ch[3]='\0';
145
//cout<<ch<<endl;
146
if(time(NULL)-timeS>=1)
147
{
148
SP->WriteData(ch,5);
149
timeS=time(NULL);
150
}
151
ignore++;
152
if(ignore>2)
153
ignore=5;
154
char t = (char)waitKey(20);
155
if( t == 27 )
156
{
157
SP->~Serial();
158
break;
159
}
160
if( t == ' ' )
161
{
162
update_bg_model = !update_bg_model;
163
if(update_bg_model)
164
printf("Background update is on\n");
165
else
166
printf("Background update is off\n");
167
}
168
}
169
return 0;
170 }
171
172
173 Mat mirror(Mat frame)
174 {
175
int X = frame.rows;
176
int Y= frame.cols;
177
for(int i=0;i<X;i++)
178
{
179
for(int j=0;j<=Y/2;j++)
180
{
181
Vec3b colorA = frame.at<Vec3b>(i,j);
182
Vec3b colorMir=frame.at<Vec3b>(i,Y-j);
183
frame.at<Vec3b>(i,j)=colorMir;
184
frame.at<Vec3b>(i,Y-j)=colorA;
185
}
186
}
187
return frame;
8
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
}
Mat Frame2Bin(Mat frame)
{
int X = frame.rows;
int Y= frame.cols;
Mat binaryImg(X,Y,CV_8U,Scalar(0));
for(int i=0;i<X;i++)
{
for(int j=0;j<Y;j++)
{
Vec3b color = frame.at<Vec3b>(i,j);
int c1=color[2];
int c2=color[1];
int c3=color[0];
if((c1<=20&&c2<=20&&c3<=20))
{
binaryImg.at<uchar>(i,j)=0;//All blue & black colour is converted to pure white%
}
else
{
binaryImg.at<uchar>(i,j)=255;//rest are converted to black
}
}
}
if(!GaussSmooth)
{
//morphological opening (remove small objects from the foreground)
erode(binaryImg, binaryImg, getStructuringElement(MORPH_ELLIPSE, Size(8,8)) );
dilate( binaryImg, binaryImg, getStructuringElement(MORPH_ELLIPSE, Size(8,8)) );
//morphological closing (fill small holes in the foreground)
dilate( binaryImg, binaryImg, getStructuringElement(MORPH_ELLIPSE, Size(8,8)) );
erode(binaryImg, binaryImg, getStructuringElement(MORPH_ELLIPSE, Size(8,8) ));
}
return binaryImg;
}
Mat convex_Hull(Mat src,Mat x)
{
Mat thresh;
x.copyTo(thresh);
vector<vector<Point> > contours;
//vector<Vec4i> hierarchy;
findContours( thresh, contours, CV_RETR_LIST, CV_CHAIN_APPROX_SIMPLE, Point(0, 0) );
//Find the contour with maximum area
int maxarea=5000;
int indexOfBiggestContour=-1;
for( int i = 0; i < contours.size(); i++ )
{
if(contourArea(contours[i])>maxarea)
{
maxarea=contourArea(contours[i]);
indexOfBiggestContour=i;
}
}
// Find the convex hull object for each contour
vector< vector<Point> > hull(1);
vector<vector<int> > hullsI(1);
Scalar color = Scalar(0,255,0);
Scalar color1= Scalar(0,0,255);
9
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317 }
if(indexOfBiggestContour!=-1)
{
convexHull(contours[indexOfBiggestContour],hull[0],false);
convexHull(contours[indexOfBiggestContour],hullsI[0],false);
vector<Vec4i> defects;
drawContours( src, contours, indexOfBiggestContour, {0,0,255}, 2, 8, vector<Vec4i>(),0, Point() );
drawContours( src, hull, 0, color, 2, 8, vector<Vec4i>(), 0, Point() );
convexityDefects(contours[indexOfBiggestContour],hullsI[0],defects);
//vector< vector<Point> > polygon(1);
//Rect ab;
RotatedRect rect;
rect=minAreaRect(contours[indexOfBiggestContour]);
//ab=boundingRect(contours[indexOfBiggestContour]);
Point2f rect_points[4];
rect.points( rect_points );
for( int j = 0; j < 4; j++ )
{
//draw rotated rectangle for the biggest contour
line( src, rect_points[j], rect_points[(j+1)%4], CV_RGB(255,255,0), 2, 8 );
}
//rectangle(src,rect,{0,255,255},2,8,0);
//approxPolyDP(contours[indexOfBiggestContour],polygon[0],3,true);
//drawContours(src,polygon,0,{255,255,0},2,8,vector<Vec4i>(),0,Point());
Point center;
int sumX=0;
int sumY=0;
countFingers(defects);
lowDepthDefects(defects);
for(int j=0;j<defects.size();j++)
{
int startidx=defects[j][0]; Point ptStart( contours[indexOfBiggestContour][startidx] );
int endidx=defects[j][1]; Point ptEnd( contours[indexOfBiggestContour][endidx] );
int faridx=defects[j][2]; Point ptFar( contours[indexOfBiggestContour][faridx] );
sumX+=ptFar.x;
sumY+=ptFar.y;
float depth=defects[j][3]/256.0;
//cout<<depth<<endl;
//line(src,ptStart,ptEnd,{255,0,0},2,8,0);
if(depth>70&&depth<130)
circle(src,ptFar,5,{255,255,0},2,8,0);
}
if(fingers==1&&lowDepth>=16)
{
average();
cout<<" Fingers = 0 ---- Angle = "<<angle<<endl;
}
else
{
average();
cout<<" Fingers = "<<fingers<<" ---- Angle = "<<angle<<endl;
}
if(defects.size()>3)
{
int cenX=sumX/defects.size(); int cenY=sumY/defects.size();
center={cenX,cenY};
Palm_Center(src,center);
circle(src,palm_center,15,{0,0,0},3,8,0);
}
}
return src;
10
10
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
11
11
Serial.cpp
1 #include "SerialClass.h"
2
3 Serial::Serial(char *portName)
4 {
5
//We're not yet connected
6
this->connected = false;
7
8
//Try to connect to the given port through CreateFile
9
this->hSerial = CreateFile(portName,
10
GENERIC_READ | GENERIC_WRITE,
11
0,
12
NULL,
13
OPEN_EXISTING,
14
FILE_ATTRIBUTE_NORMAL,
15
NULL);
16
17
//Check if the connection was successfull
18
if(this->hSerial==INVALID_HANDLE_VALUE)
19
{
20
//If not success full display an Error
21
if(GetLastError()==ERROR_FILE_NOT_FOUND){
22
23
//Print Error if neccessary
24
printf("ERROR: Handle was not attached. Reason: %s not available.\n", portName);
25
26
}
27
else
28
{
29
printf("ERROR!!!");
30
}
31
}
32
else
33
{
34
//If connected we try to set the comm parameters
35
DCB dcbSerialParams = {0};
36
37
//Try to get the current
38
if (!GetCommState(this->hSerial, &dcbSerialParams))
39
{
40
//If impossible, show an error
41
printf("failed to get current serial parameters!");
42
}
43
else
44
{
45
//Define serial connection parameters for the arduino board
46
dcbSerialParams.BaudRate=CBR_9600;
47
dcbSerialParams.ByteSize=8;
48
dcbSerialParams.StopBits=ONESTOPBIT;
49
dcbSerialParams.Parity=NOPARITY;
50
//Setting the DTR to Control_Enable ensures that the Arduino is properly
51
//reset upon establishing a connection
52
dcbSerialParams.fDtrControl = DTR_CONTROL_ENABLE;
53
54
//Set the parameters and check for their proper application
55
if(!SetCommState(hSerial, &dcbSerialParams))
56
{
57
printf("ALERT: Could not set Serial Port parameters");
58
}
59
else
60
{
12
12
61
//If everything went fine we're connected
62
this->connected = true;
63
//Flush any remaining characters in the buffers
64
PurgeComm(this->hSerial, PURGE_RXCLEAR | PURGE_TXCLEAR);
65
//We wait 2s as the arduino board will be reseting
66
Sleep(ARDUINO_WAIT_TIME);
67
}
68
}
69
}
70
71 }
72
73 Serial::~Serial()
74 {
75
//Check if we are connected before trying to disconnect
76
if(this->connected)
77
{
78
//We're no longer connected
79
this->connected = false;
80
//Close the serial handler
81
CloseHandle(this->hSerial);
82
}
83 }
84
85 int Serial::ReadData(char *buffer, unsigned int nbChar)
86 {
87
//Number of bytes we'll have read
88
DWORD bytesRead;
89
//Number of bytes we'll really ask to read
90
unsigned int toRead;
91
92
//Use the ClearCommError function to get status info on the Serial port
93
ClearCommError(this->hSerial, &this->errors, &this->status);
94
95
//Check if there is something to read
96
if(this->status.cbInQue>0)
97
{
98
//If there is we check if there is enough data to read the required number
99
//of characters, if not we'll read only the available characters to prevent
100
//locking of the application.
101
if(this->status.cbInQue>nbChar)
102
{
103
toRead = nbChar;
104
}
105
else
106
{
107
toRead = this->status.cbInQue;
108
}
109
110
//Try to read the require number of chars, and return the number of read bytes on success
111
if(ReadFile(this->hSerial, buffer, toRead, &bytesRead, NULL) )
112
{
113
return bytesRead;
114
}
115
116
}
117
118
//If nothing has been read, or that an error was detected return 0
119
return 0;
120
121 }
122
123
124 bool Serial::WriteData(char *buffer, unsigned int nbChar)
125 {
13
13
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
DWORD bytesSend;
//Try to write the buffer on the Serial port
if(!WriteFile(this->hSerial, (void *)buffer, nbChar, &bytesSend, 0))
{
//In case it don't work get comm error and return false
ClearCommError(this->hSerial, &this->errors, &this->status);
return false;
}
else
return true;
}
bool Serial::IsConnected()
{
//Simply return the connection status
return this->connected;
}