Through the Interface

September 1 9, 201 2

Creating a face-recognising security cam with a Raspberry Pi Part 2

In the first post in this series, we introduced the idea behind the Facecam, a Facebook-enabled security camera running on a Raspberry Pi dev ice. Ov er the nex t three posts were going to look in more detail at the sy stems components. In this post, were going to start by looking at the component the only one that doesnt actually run on the Raspberry Pi that downloads the information from Facebook to build the facial recognition database to be used by the dev ice. As mentioned last time, Iv e coded this as a .NET application actually by ex tending the WinForms sample prov ided with the Facebook C# SDK in order to dedicate the Raspberry Pi to performing the security cam function. Im not actually going to share the code for this application, as its really a work in progress. The code itself isnt v ery complicated the Facebook piece is mostly ex ecuting FQL, which allows y ou to use a SQL-like sy ntax to query the Facebook Graph API and Ill be describing the techniques it uses should people want to go ahead and roll their own. Before looking at the behav iour of the application itself, its important to understand more about this database that its going to help build. The facial recognition code Iv e adopted is based on work by Sherv in Emami and uses the Eigenface algorithm to compare detected faces against a database. To understand more about Eigenface, this article prov ides a thorough ex planation. Sherv ins OnlineFaceRec sample makes use of OpenCV for this process. Iv e borrowed code from this sample for use on the Raspberry Pi, but today s application simply builds a file and folder structure that can be used by the OnlineFaceRec application to train the database (and essentially build the facedata.xml file well transfer across to the Raspberry Pi). There are two main parts of this process: downloading the friend data from Facebook and then parsing that data on disk in order to create a training script to be passed to OnlineFaceRec. These two functions are reflected by the basic UI of the .NET application:

Clicking on the first button lets us log in to Facebook and accept the permissions requested by the .NET

application. Which reminds me: to create an application that makes use of the Facebook API, y oull need to register it with Facebook and include the AppId in y our code. As y ou can see below, I called my app FriendEx tractor:

Once were logged in, we see the main dialog which will help us driv e the download process:

Clicking on the Populate Friends button pulls down the list of the current users friends from Facebook and populates the list on the left:

Y ou can see the list has checkbox es associated with each item. Originally the implementation didnt allow selection of friends to include, which ended in a huge amount of data being downloaded (it turns out I hav e friends who seem to do nothing other than upload their pictures to Facebook and hav e 1 000+ tagged images up there). Running OnlineFaceRec to train the database with the results ended up in a 400MB facedata.x ml file being created. Giv en the fact the R-Pi has 256MB of RAM< I figured that was a non-starter. :-) I therefore enabled selection of friends that are actually likely to v isit our home or that I may end up demoing this to at AU which greatly reduces the amount of probably redundant data that gets downloaded and processed. The list of selected friends gets sav ed in a simple tex t file and reloaded automatically . I also added the ability to start downloading from a particular friend, to allow incremental additions/updates. The first time its run, its certainly worth starting from the beginning, of course, but y ou may want to restart from a friend further down the list a later point. When y ou select a friend on the left, the Start Download from Selected button should come aliv e. Pushing it does whats written on the box :

Once completed, y oull hav e a folder full of photos of y ou and y our friends. The folder structure contains one lev el of indirection to use the user ID as friends can v ery well hav e the same name (theres a teenager named Kean Walmsley who liv es in Canada and friended me on Facebook some time ago, for instance) and inside that folder y oull find a collection of images:

Looking closely , abov e, y oull see there are both .jpg and .pgm images. The .jpg files are those downloaded directly from Facebook nothing surprising there. The .pgm files are created by the .NET application using following algorithm: For each tagged photo of a user Use OpenCV face detection to find the faces in the image For this I used Emgu CV a .NET wrapper for OpenCV along with OpenCV s face detection feature that looks for haar-like features Compare the tag location with the faces returned. For the best match Take a grey scale copy of the cropped area Resize it to 50 x 50 Equalize the histogram to get consistent brightness and contrast Sav e that file to the .pgm format Regarding the quality of the photos downloaded from Facebook: frankly , it v aries. Some photos are poorly tagged, and some are tagged well enough but the face cannot easily be ex tracted (which sometimes means someone elses gets picked up in its place). And then some photos simply arent well suited to face detection/ex traction, so the results end up being just plain strange. Iv e found that some kind of manual scrubbing process between the download and the script creation/database training helps the quality of the data a great deal. Iv e used a tool called IrfanV iew to do this: y ou can search a set of sub-folders for *.pgm files and transfer the results across into Thumbnail V iew, and can then proceed to delete the ugly pics from there. Of course y ou could do the same thing in Ex plorer if y ou had a shell ex tension that can prev iew .pgm files, but I wasnt able to find one. And IrfanV iew seems to work pretty well. Heres a v iew of some shots of Scott Sheppard and Shaan Hurley , for instance:

In time I ex pect a bespoke tool one that remembers the pics y ouv e deleted before, to sav e y ou hav ing to repeat that task each time is the way to go. But thats for another time. Once this is done, y ou should hav e a fairly clean set of normalised (i.e. of the same size and with similar brightness/contrast) files that can be used to generate a training script for the OnlineFaceRec tool. The second button on our main dialog runs some code that copies the .pgm files to a separate folder and creates a training script listing them all:
p r i v a t ev o i db t n C r e a t e S c r i p t _ C l i c k ( o b j e c ts e n d e r ,E v e n t A r g se ) { c o n s ts t r i n gr o o t=" Z : \ \ f b _ p h o t o s \ \ " ; c o n s ts t r i n gd e s t R o o t=" Z : \ \ r p _ p h o t o s \ \ " ; S o r t e d L i s t < l o n g ,l o n g >c h e c k e d F r i e n d s= I n f o D i a l o g . L o a d C h e c k e d F r i e n d s ( ) ; / /G e tt h el i s to fP G Mf i l e si no u rs o u r c ed i r e c t o r y D i r e c t o r y I n f od i=n e wD i r e c t o r y I n f o ( r o o t ) ; v a rf i l e s=d i . G e t F i l e s ( " * . p g m " ,S e a r c h O p t i o n . A l l D i r e c t o r i e s ) ; S t r i n g B u i l d e rs b=n e wS t r i n g B u i l d e r ( ) , c u r=n e wS t r i n g B u i l d e r ( ) ; / /W e ' l lm a i n t a i na ni n d e xf o rt h ep e r s o n sa n da n / /o l du s e rI D ,s ow ec a nt e l lw h e nw eh a v ec h a n g e d / /t oan e wp e r s o n ' sd a t a i n tp e r s o n=0 ; s t r i n go u i d=" " ; / /M a i n t a i nac o u n t e ro ft h en u m b e ro fp h o t o sf o r / /t h ec u r r e n tp e r s o n( w eo n l yw r i t eo u t2 +p h o t o s , / /a so t h e r w i s et h e yd o n ' tt r a i nt h ed a t a b a s e ) , / /a sw e l la sal i s to ft h e i rp a t h s

i n tp h o t o s F o r C u r r e n t=0 ; L i s t < s t r i n g >f i l e s F o r C u r r e n t=n e wL i s t < s t r i n g > ( ) ; f o r e a c h( F i l e I n f of ii nf i l e s ) { v a rp a t h=f i . D i r e c t o r y N a m e ; v a ri n f o=p a t h . S p l i t ( n e w [ ]{' \ \ '} ) ; i f( i n f o . L e n g t h= =4 ) { s t r i n gn a m e=i n f o [ 2 ] ,u i d=i n f o [ 3 ] ; i f( o u i d! =u i d ) { / /W eh a v ean e wu s e r o u i d=u i d ; / /I ft h el a s tu s e rh a dm o r et h a n2p h o t o s . . . i f( p h o t o s F o r C u r r e n t>2 ) { / /L e t ' sa d dt h e mt ot h et r a i n i n gs c r i p ta n d / /c o p yt h ef i l e st ot h ed e s t i n a t i o n p e r s o n + + ; s b . A p p e n d ( c u r . T o S t r i n g ( ) ) ; f o r e a c h( s t r i n gf i l ei nf i l e s F o r C u r r e n t ) { s t r i n gd e s t F i l e= d e s t R o o t+ U t i l s . R e m o v e A c c e n t s ( f i l e . S u b s t r i n g ( r o o t . L e n g t h ) ) ; D i r e c t o r y . C r e a t e D i r e c t o r y ( P a t h . G e t D i r e c t o r y N a m e ( d e s t F i l e ) ) ; F i l e . C o p y ( f i l e ,d e s t F i l e ,t r u e ) ; } } / /R e s e tt h es t r i n g ,p h o t o sa n di m a g ef i l ei n f o / /f o rt h ec u r r e n tu s e r c u r . C l e a r ( ) ; p h o t o s F o r C u r r e n t=0 ; f i l e s F o r C u r r e n t . C l e a r ( ) ; } l o n gi d=I n t 6 4 . P a r s e ( u i d ) ; i f( c h e c k e d F r i e n d s . C o n t a i n s K e y ( i d ) ) { / /A d dt h ei n f o r m a t i o no nt h ec u r r e n tp h o t ot o / /t h ec u r r e n tu s e r ' ss t r i n g c u r . A p p e n d ( S t r i n g . F o r m a t ( " { 0 }{ 1 }{ 2 } \ r \ n " , p e r s o n ,U t i l s . R e m o v e A c c e n t s ( n a m e ) , " . / "+ U t i l s . R e m o v e A c c e n t s ( f i . F u l l N a m e ) . S u b s t r i n g ( r o o t . L e n g t h ) . R e p l a c e ( ' \ \ ' ,' / ' ) ) ) ;

/ /D ot h es a m ef o rt h ep h o t o st oc o p ya c r o s st o / /t h ed e s t i n a t i o n p h o t o s F o r C u r r e n t + + ; f i l e s F o r C u r r e n t . A d d ( f i . F u l l N a m e ) ; } } } / /D o n ' tf o r g e tt ot i d yu pa n dc o p yt h el a s tu s e r ' s / /t r a i n i n gs t r i n ga n df i l e sa c r o s s s b . A p p e n d ( c u r . T o S t r i n g ( ) ) ; f o r e a c h( s t r i n gf i l ei nf i l e s F o r C u r r e n t ) { s t r i n gd e s t F i l e= d e s t R o o t+f i l e . S u b s t r i n g ( r o o t . L e n g t h ) ; D i r e c t o r y . C r e a t e D i r e c t o r y ( P a t h . G e t D i r e c t o r y N a m e ( d e s t F i l e ) ) ; F i l e . C o p y ( f i l e ,d e s t F i l e ,t r u e ) ; } / /F i n a l l yw ew r i t eo u tt h es t r i n gt ot h et r a i n i n gf i l e F i l e . W r i t e A l l T e x t ( d e s t R o o t+" t r a i n i n g . t x t " ,s b . T o S t r i n g ( ) ) ; }

In case y oure wondering, Iv e kept the two main pieces of the process separate primarily so I can combine my wifes friends with my own before training the database. :-) Now its a simple matter of running OnlineFaceRec with the training script:

The output of this tool is pretty interesting: aside from the v ery important facedata.xml file (the database well use on the Raspberry Pi), we also two files that are created mainly to demonstrate the principle at work. Firstly we hav e the av erage face which is the absolute av erage of all the pictures of y our friends, and tends to be a bland, slightly androgy nous and some might say idealised image of a human face:

And seondly we hav e an image of the Eigenfaces themselv es, which map the differences from this av erage image:

Neither of these images needs to be transferred across to the Raspberry Pi their data is captured in the XML database (which stands at around 28 MB for the selected subset of my friends). In the nex t post, well look at the implementation of a component that makes use of this XML data to perform facial recognition on images captured by our motion detector. Y ou m ight also like:

Creating a facerecognising

Creating a facerecognising

Waiting for my Raspberry Pi

Creating a motiondetecting security