AUGMENTED REALITY

BASED PRODUCT
IDENTIFICATION AND
ADVERTISING SYSTEM
- UDARA ABHILASH ALWIS (CB003982)

INTRODUCTION

PROBLEM BACKGROUND
Problem Statement
There are huge product varieties in the market and
advertising has a great impact in todays life. Customers
are always looking for extended information when they buy
products or even regarding an advertisement, whereas it is very
crucial to provide users with updated and accurate
information regarding them. QR Codes are being
deprecated in modern end consumerism due to its
disadvantages and real time visual recognitions technologies
are coming up with powerful mobile devices. Augmented
Reality is becoming the next mass media advertising in
todays world.

PROBLEM BACKGROUND

Product information could change anytime

Advertisements can not display all the information at
once
Customers are looking for more extended information
No proper method to look for information, time waste
Augmented Reality as the next mass advertising media
Extinction of QR Codes from end user consumerism

PROBLEM OBJECTIVES
Identify Product or Advertisement
View the extended information, and related
advertisements or products
Implement Augmented Reality based concepts in the
application
Insert Products and Advertisements information as
Administrators
Simulate Product or Advertisement Identification
process in admin system.

SOLUTION OUTLINE
Capture image when the user points at a product or
an advertisement.
Identify the product or the advertisement from the
server side.
Provide extended information about the product or
the advertisement to the user.
Present the information in according to Augmented
Reality principles.
Allow admins to add new information to the System.

MAIN CHALLENGES

Implementing Augmented Reality into the Mobile

Application.
Getting the identification results from the server from the
fastest way possible.
Implementing an Image Matching Algorithms.
Identifying a product or an advertisement by its Brand Logo.
Adding and maintaining Product and Advertisement
information

RESEARCH

RESEARCH
User Requirements Investigation
Customers
Difficulty of Finding information
Having to wait for a Salesperson to show up
QR Codes, useless
A New Experience
Advertisers
The search for new creative advertising strategies
Modern Adverts focus simplicity, therefore less
information
Visual Recognition over QR Code Recognition

RESEARCH
Similar Systems Study

Conclusions
Used for Advertising and real

Aurasma
Vuforia Qualcomm

time Image target

identification
Only available for US and

Layar
The IBM Augmented Reality
Personal Shopping Assist

Europe based countries

Cloud technologies
No Local implementation

RESEARCH
Augmented Reality
Mobile Augmented Reality
Mobile Application Development Platform
Apple
Android
Windows Phone
Mobile Video Stream Frame Monitoring
Image Processing

RESEARCH
Image Enhancements
Spatial Domain Enchantement
Image Negatives
Contrast Stretching
Histogram Processing
Frequency Domain Enchantement
Smoothing Frequency-Domain Filters
Low pass Filtering
High pass Filtering
Computer Vision

RESEARCH
Feature Extraction
SIFT - Scale Invariant Feature Transforms
SURF Speeded Up Robust Features
Matching
3D Object Detection in SURF
FAST - Corner Detection
FREAK - Fast Retina Key point Detection
BRISK - Binary Robust Invariant Scalable Key-points
Character Recognition and Reading

RESEARCH
Selection of APIs and Tools
Image Processing and Image Matching APIs
OpenCV
MatLab
Simple CV

Character Recognition APIs

Tesseract
ABBY OCR

RESEARCH
Augmented Realty APIs and SDKs
Vuforia
FastCV
Aurasma
Layar

DEVELOPMENT METHODOLOGY
Feature Driven Development
Optimized for Scalable Systems
Executes on iterative development
cycles
Suited for time intensive application
Features list that is being built at
the beginning of the development
method cycle
Feature by Feature
Each feature gets much better
focus

ANALYSIS
Augmented Realty APIs and SDKs
Mobile Augmented Reality
Mobile Application Development Platform
Apple
Android
Windows Phone
Mobile Video Stream Frame Monitoring
Image Processing

RESEARCH

RESEARCH
Similar Systems Study Analysis
Augmented Reality
Image Processing and Recognition
Cloud Computing
User Experience
Availability for 3rd Parties
Not available for 3rd World Countries
No actual implementation in Sri Lanka (Experimental)

RESEARCH

Mobile Application Development Analysis Android

Mobile Platform Usage
Developer Support and Features
Multitasking and Heavy Weight Processing Support
Image Processing
User Friendliness and UI Features

RESEARCH
Computer Vision Analysis
Robust
Faster
High Accuracy
Feature Extraction Analysis - SIFT, SURF, FREAK and
BRISK
SURF
Speed and Accuracy
Implementation Support

REQUIREMENT SPECIFICATION

REQUIREMENT SPECIFICATION
Functional Requirements
Ability to identify Products or Advertisements
Augmented Reality should be implemented
Provide Related Products and Advertisements
Automatically Sort and categorize related Products
and Advertisements

REQUIREMENT SPECIFICATION
Non-functional requirements
User friendly and easy to use Interface.
An Interface suitable for Augmented Reality based UI
designs.
Mobile application should be smooth and quickly
responsive.
Recognize Products and advertisements from the
fastest way possible.

DESIGN

DESIGN
Use case Mobile Application

DESIGN
Use case Admin Server System

DESIGN

Class Diagram Admin Server System

DESIGN

Activity Diagram SURF Image Matching

DESIGN

Sequence Diagram Brand Logo Identification

DESIGN

Sequence Diagram Product Identification

DESIGN

Sequence Diagram Mobile Application

IMPLEMENTATION

Image matching using SURF

Character Recognition and key words identification
Mobile Application UI development
Mobile Application Backend
Hosting a Web Service

IMPLEMENTATION
SURF Feature Extraction and Matching
Extract Feature Points and Descriptors

Brute Force Match and Vote for Feature Point Matches

IMPLEMENTATION
Further Voting and Filtering

Draw Matching Feature Points and Return Results

IMPLEMENTATION
SURF Feature Extraction and Matching Sample

IMPLEMENTATION
Experimentations
In need of Experimentation and Modifications
Could use mask Matrix
Started to give even greater matched points count
of incorrect images

IMPLEMENTATION
Experimentation Conclusions
Get the number of matched feature points count
Calculate the Area of the formed Polygon
Identify whether the formed Polygon is a proper
Polygon
Calculate the percentage Ratio of the Detected Key
points of both images
Adjust the Uniqueness threshold

IMPLEMENTATION
Determining Best Match

IMPLEMENTATION
Threshold of an Image Matching Process

Image Pre-Processing Usage

Character Recognition and key words identification

IMPLEMENTATION
Mobile Image Processing
Image Resizing and Scaling

IMPLEMENTATION
Image Encoding and Decoding Base64

IMPLEMENTATION
Web Service Connectivity

IMPLEMENTATION
Hosting a Web Service

Runtime Exception Heavy EmguCV Libraries

Cv.Invoke Exceptions
Server Out Of Memory Exceptions

IMPLEMENTATION
Dummy Web Service

IMPLEMENTATION
Overall Admin System Implementation
Object Oriented Methodologies

Surf Execution Class

IMPLEMENTATION
Product Identification and Advertisement Identification
Process

Find the Brand Logo Information

Load the Product Images under that Brand Logo
Take the given image through that Product Images
list to find the best match and retrieve information.

TESTING
Unit Testing
Functional Testing
Performance Testing

TESTING
Unit Testing - Server System
Test Case 1
Test Criteria
Input
/

Test Name : Recognize Given Product Images Brand Logo at

Server Simulation
Check if the Given Products Brand type is identified.
Action Drag and Drop a Product Image and Click on Search and Detect

Performed
Expected Results
Actual Results

System should identify the brand logo and Display the Brand Information
The System successfully identified the Brand Type and showed the

Test Case 2

information of the Brand.

Test Name : Recognize Given Product Image at Server Simulation

Test Criteria

Check if the given Products front view Image is Recognized and Show the

Input

information of the Product

Action Drag and drop Product Image and click on Search and Detect button.

Performed
Expected Results

System should recognize the Product and show information about it as

Actual Results

well as Related Products.

Product was Successfully Recognized and information were showed.

TESTING
Unit Testing - Server System
Test Case 3

Test Name : Determine when a given Image of a Product or an

Advertisement cannot be Identified

Test Criteria

Check if the given Threshold condition is able to determine whether a given

image can be Recognized or not.

Input

Action Insert a random unrelated Product/advertisement Image to the Matching

Performed

Algorithm and execute matching algorithm

Expected Results

The algorithm should determine that the given Image cannot be

Recognized based on the Threshold condition.

Actual Results

The Matching process successfully determines the given Image cannot be

identified.

TESTING
Unit Testing Server System
Test Case 4

Test Name : Automatically Recognize the Brand Type of the

Product/Advert image when adding a new Product/Advert info

Test Criteria

Check if the System is able to Automatically Recognize the Brand Type of a

given Product or Advertisement Image when Inserting their information.

Input

Action Drag and drop the Product/Advert Image and click on Auto Detect

Performed
Expected Results

The matching algorithm will detect the Brand logo of the Product/Advert
Image and identify the Brand.

Actual Results

The system successfully identifies the Brand Type by detecting the brand
logo.

TESTING
Unit Testing Mobile Application
Test Case 5
Test Criteria
Input

Test Name : Retrieve Recognition results from the Web Service

Check if the Application is able to retrieve the identification result of a
capture image
Action Click on Scan and Detect button

Performed
Expected Results

After sending the image to server application should wait for the

Actual Results

response and retrieve the results.

The Application send the image to the server and waits till the response
is being given, while the other UI Controls are being disabled. When the
results are retrieved, they are being displayed successfully.

TESTING
Performance Testing
Case
Capturing

Image

Expected Accuracy
and 100%

Actual Result
100%

Sending to Web service

Retrieving
Product/Advert 100%

100%

Recognition Results
Capturing
product/advert 100%

50%

image from a different angle

and recognizing it
Overall Recognition Results

90%

70%

CRITICAL EVALUATION
Efficiency
Execute all the heavy processing in Server
Holding the Pre-Processed SURF Features Extracted
Images in the Memory
Mobile Application to Server Communication Data
compression and Base64 Encoding
Utilizing the SURF based Image Matching process to
achieve more accuracy and Speed
Utilizing the Recognition Process
Automated Brand Logo detection Process

CRITICAL EVALUATION
Ease of Use
Accuracy
Increased User Satisfaction
Maintainability and Extendibility
The Centralized Database
Web Service Implementation

CRITICAL EVALUATION
System Utility and Usability
Consumers, Advertisers and Marketers
Providing Access for Extended information
Information are updated and centralized
implement creative strategies into Augmented Reality
intuitive way to recognize products for end users rather
than QR Codes
easily upgraded and extended due to the architecture
The interface has been designed in a very minimalistic,
simple, and user friendly

CRITICAL EVALUATION
Further Enhancements
Further Developing the Image Matching Process from
ground up to use even more efficient Algorithms such
as FREAK or BRISK Feature Extraction algorithms
Adopt an Augmented Reality API such as Vuforia
High end Cloud Service which will radically increase
the processing power.
Adopt a new lightweight API Library for Image
Processing

DEMONSTRATION

Q&A

THANK YOU !