Facial Expression Recognition

Using PCA (EigenFaces)

Team Members:
1. RIT2006012 - Vivek Kumar Bharti
2. RIT2006035 - Jitendra Kumar
3. RIT2006046 – Deepak B. Mudgal

Supervised By:
Dr. Pavan Chakraborty
Indian Institute of Information Technology
Allahabad
 Contents

Introduction
Objective of the Project
Motivation
Problem Definition

Background
Literature Survey

Implementation
Plan of Work
Modules

References
References
 Introduction

Objective

The main objective of the project is to recognize facial expression from static frontal
facial image. Our goal is to create a system which can be trained from
training images for the facial expressions and used to recognize the
facial expression in the test images.

Motivation

FACIAL image analysis is an active research in the field of machine vision.

Psychological studies have suggested that facial motion is fundamental to the
recognition of facial expression. Fully automatic and real-time facial expression
system helped for many applications such as human computer interaction, behavioral
research, video conferencing and various vision systems. Facial expression analysis is
the part of facial image analysis. It has been indicated that the verbal part of a message
contributes only for 7% to the effect of the message as a whole, the vocal part
contributes for 38%, while facial expression of the speaker contributes for 55% to the
effect of the spoken message. This implies that the facial expressions form the major
modality in human communication.

Nowadays, Facial Expression Recognition is getting more and more attention from the
research community. Facial expression plays an important role in smooth
communication among individuals. The extraction and recognition of facial
expression has been the topic of various researches subject to enable smooth
interaction between computer and their users. In this way, computers in the future will
be able to offer advice in response to the mood of the users. Recently, a number of
new technologies for Facial Expression Recognition have been developed.

Problem Definition

Given the image of a human face. We are required to process the image and try to
extract such information from it that can help us classify the image as having one of
several possible expressions like happy, sad, angry etc.
For this, we have to create a system which can be trained for a number of facial
expressions and that image be tested for the facial expression using this system.
 Background

Literature Survey

A number of techniques have been proposed in this field and are being used. But they
have an inherent complexity which makes them opaque and are computationally
expensive. Thus, a person who tries implementing these algorithms finds the same
very difficult and time consuming.
Techniques for expression recognition can be divided roughly into two classes:
appearance-based methods, and feature-point methods.
Appearance-based techniques analyze the individual pixels of the face, possibly after
performing a dimension reduction or frequency filtering.
Feature-point methods instead track the locations of key points of the face (e.g.,
corners of the eyes) and classify expressions based on the spatial relationship between
these points.
Best results for each approach are similar.
We have tried to study a number of techniques on facial expression recognition which
include Gabor Wavelet Transform, Feature Fusion Algorithm, Independent
Component Analysis, Method of Optical Flow Analysis, Method Potential Field
Analysis etc.

The method we have used is a Appearance-based technique. The basic idea involved
following steps.

Pre-processing

Given the image of a human face. We are required to process the image and try to
extract such information from it that can help us classify the image as having one of
several possible expressions like happy, sad, angry etc.
The information that needs to be extracted is formally called a set of features or
feature vector.
The feature vector should fulfill some important considerations, like it should only
provide the necessary information and it should be functionally complete, i.e. it should
provide all the information.
Training

Having extracted the features from an image, it can be used for one of the following
two purposes: (i) training, (ii) testing.
By training we mean that we use the feature vector as a sample in a set of (say) m
samples and then apply some algorithm on this sample for finding out the various
expression groups present in this sample set. These groups may also be called clusters.

Testing

When we talk of testing we mean that having this sample in our hands, we try to test it
against a set of know clusters obtained from a given sample set. By doing so we wish
to classify this vector as belonging to one of the expression groups.

Implementation

Picture Database

Since the main purpose of this project is facial expression recognition (Not face
detection), therefore, the sample pictures are taken under special consideration to ease
up the face detection process. Each picture is taken under the condition that, only face
is the largest skin colored continuous object in the frame.

There are two sets of pictures:

One is used for training purpose and another is used for testing.

The training pictures are classified in the following expressional classes(in an

ascending order of expressional intensity):
1. Image001 to Image013 = Happy
2. Image014 to Image024 = Disgust
3. Image025 to Image034 = Anger
4. Image035 to Image043 = Unhappy
5. Image044 to Image050 = Neutral
Another image set is used for testing purpose. These images are taken in quite an
arbitrary fashion. It also includes some expressions that are not contained in the
training set like "Surprise" (Image004) and "Confused" (Image005).
Algorithm

Create a low dimensional face space using train images.

This is done by performing Principal Component Analysis (PCA) in the training
image set and taking the principal components (i.e. Eigen vectors with greater Eigen
values).
In this process, projected versions of all the train images are also created.

Project test images on the face space – as a result, all the test images are represented
in terms of the selected principal components.

Euclidian distance from all the projected train images of a projected test image are
calculated and the minimum value is chosen in order to find out the train image which
is most similar to the test image. The test image is assumed to fall in the same class
that the closest train image belongs to.

In order to determine the intensity of a particular expression, its Euclidian distance

from the mean of the projected neutral images is calculated.

The more the distance - according to the assumption – the far it is from the neutral
expression.

Face Detection
 Since the main purpose of this project is facial expression recognition (Not face
detection), therefore, the sample pictures are taken under special consideration to
ease up the face detection process. Each picture is taken under the condition that,
only face is the largest skin colored continuous object in the frame.

Matlab Function Used for Face Detection:

• imadjust :
This function is used for the lighting compensation.
This is a pre-processing step.

• rgb2ycbcr
This function is used for the changing the color space from RGB to YCbCr.
This color space is used for the efficient face detection.

• edge(image, ‘canny’)
This function is used for edge detection using the canny edge detector.

• strel, imdilate
strel is used to create a square element which is used in to convert the edge-
image to the dilate image

• imfill
Using this the dilate image is converted to the image with regions separated
from each other, i.e. skin color region separated from non-skin region

• bwlabel
Used to label the separated regions, which can further be used to bound the face
in the image

• imcrop
Crops the face from the image

Principal Component Analysis

A N x N pixel image of a face,represented as a vector occupies a single point in N2-

dimensional image space.

Images of faces being similar in overall configuration, will not be randomly

distributed in this huge image space.

Therefore, they can be described by a low dimensional subspace.

Main idea of PCA :

To find vectors that best account for variation of face images in entire image space.
These vectors are called eigen vectors.
Construct a face space and project the images into this face space(eigenfaces).

Recognition

The test image, Γ, is projected into the face space to obtain a vector, Ω:
Ω = UT(Γ – Ψ)

The distance of Ω to each training image is defined by

Єk2 = ||Ω-Ωk||2; k = 1,…,M

Training image at minimum distance is used as reference to recognize the expression

of the current test image

Inputs/Output

Inputs:
Training image set.
Label File for information about training images.
Testing image set.

Output:
Result file with
Expression for the test images
Best match
Distance from Neutral

Tool Used

Matlab

Toolboxes Used:
1. Image Processing Toolbox (For Resizing Image)
2. Statistics Toolbox (For PCA)
 References

References :

1. Turk, M., Pentland, A.: Eigenfaces for recognition. J. Cognitive Neuroscience 3

(1991) 71–86

2. Belhumeur, P., P.Hespanha, J., Kriegman, D.: Eigenfaces vs. fisherfaces:

recognition using class specific linear projection. IEEE Transactions on Pattern
Analysis and Machine Intelligence 19 (1997) 711–720

3. M. Turk and A. Pentland, "Eigenfaces for Recognition", Journal of Cognitive

Neuroscience, March 1991

4. Tolga Birdal, Simple Face Detection,

http://www.mathworks.com/matlabcentral/fileexchange/23382