Content-based Image

Retrieval
Presentation by
Charlie Neo
 Introduction
 Why
 Digital image database growing rapidly in size

 Professional needs – Logo Search

 Difficulty in locating images on the web

 Example
 Find a picture of me and jack on the bus
 Application CBIR
 Search for one specific image.

 General browsing to make an interactive choice.

 Search for a picture to go with a broad story or search to

illustrate a document.

 Search base on the esthetic value of the picture.

 Two Classes of CBIR
Narrow vs. Broad Domain
 Narrow
 Medical Imagery Retrieval

 Finger Print Retrieval

 Satellite Imagery Retrieval

 Broad
 Photo Collections

 Internet
 Challenges
 Semantic gap
 The semantic gap is the lack of coincidence between the
information that one can extract from the visual data and the
interpretation that the same data have for a user in a given
situation.
 User seeks semantic similarity, but the database can only
provide similarity by data processing.

 Huge amount of objects to search among.

 Incomplete query specification.

 Incomplete image description.

 Description Of Content:
Image Processing

 Color

 Local Shape

 Texture
 Color Image Processing

 Problems with color variances

 Surface Orientation
 Camera Viewpoint
 Position of Illumination
 Intensity of the Light
 Color Image Processing

 Approaches
 Opponent color axes

• Advantage of isolating the brightness information on the third axis.

• Invariant to changes in illumination intensity and shadows.

 HSV-representation
• Invariant under the orientation of the object with respect to the
illumination and camera direction.

 Search for clusters in a color histogram to identify which pixels

in the image originate from one uniformly colored object.
 Image Processing for Local Shape

 Problems
 Occlusion
 Different Viewpoint

 Approaches
 Collect all properties that capture geometric
details in the image.
 Invariant Descriptors.
 Image Texture Processing

 Problems
 Offer little semantic referent.

 Approaches
 Markovian analysis
 Wavelets
• Generated by groups of dilations or dilations and rotations
• Some semantic correspondent.

 Great For
 Satellite images
 Images of documents
 Description Of Content using Features

 Grouping Data
 Strong Segmentation

• Region T = 0 (object)
• Shape and Object features
 Weak Segmentation
• T subset of 0
• Salient features
 Sign Detection
• Signs Probabilities
 Partitioning
• Global feature
 Global and Accumulating Features

 Accumulation of features using a histogram

 The set of features F(m) ordered by histogram index m.

 64-bin histogram, has discriminating power up to 25,000

images
 Salient Features

 Weak segmentation. (grouping of data into

homogeneous regions.)

 Salient feature calculations lead to sets of regions or

points with known location and feature values.

 Innovation of content-based retrieval.

 Expected to receive much attention in the further

expansion of the field.
 Signs

 Typical signs are an icon, a character, a traffic light, or a

trademark.

 Strong semantic interpretation is within grasp

 Analysis tends to become application-oriented.

 Shape and Object Features

 Object segmentation is hard.

 Possible for narrow domain.

 Fortunately, for our purpose it only requires detection of

the object’s presents.
 Description of Structure and Lay-Out

 Structural feature
 Feature values

 Relationships between object sets.

 captured in a graph

 Lay-out descriptions
 characterized by locations, size, and features.
 Interpretation And Similarity

 Semantic Interpretation
 Derive interpretation from feature set.

 Features generate a probability distribution.

 MAVIS2-system: four semantic layers.

 Similarity
 Similarity measure S
q,d between the images q and d

 Sq,d = s(Fq,Fd).
 s(Fq,Fd) = g( d(Fq,Fd) )
 e.g. d can be just the Euclidean distance.
 User Interaction

 Query Space: Definition and Initialization

 Q = {IQ,FQ,SQ,ZQ}

 IQ is a selection of images from the large image archive I.

 FQ is a selection of features from feature set F.

 SQ similarity function.

 ZQ is a set of labels to capture goal dependent semantics.

Query Specification
 Query Space Display

 Besides just showing the images that match the query …

 Images are placed in such a way that distances between

images in the display reflect SQ.

 Highlight parts indicating which parts of the image fulfill

the criteria. (exact query)
 Interacting with Query Space

 The process of query specification and display is iterated, where, in

each step, the user revises the query.

 user feedback leads to an update of query space:

 Both positive and negative examples is used.

 Each iteration, the probability of being the target for an image in IQ

is increased or decreased
 SYSTEM ASPECTS:
Storage and Indexing

 Standard, Linear File System

 O(N)

 Three classes of indexing methods

 Space partitioning

 Data partitioning

 Distance-based technique

 Varies tree structure

 O(log N)
 SYSTEM ASPECTS:
System Architectures

 Separate indexing and retrieval.

 Image retrieval as a plug-in module to an existing

database system.

 Analysis, indexing and training as modules

 SYSTEM ASPECTS:
System Evaluation

 Relevance is subjective.

 Human subjects to produce idea ordering for a query.

 Cortina: A System for large scale,
content based Web Image Retrieval

 Built for Web Image Retrieval

 3 million images
 Clusters
 Data Mining for Semantics
 Cortina: A System for large scale,
content based Web Image Retrieval
 Cortina: A System for large scale,
content based Web Image Retrieval
 Four global feature descriptor
 Homogeneous Texture Descriptor

 Edge Histogram Descriptor

 Scalable Color Descriptor

 Dominant Color Descriptor

 Linear combination of the 4 features, as the distance for

K-NN search.
 Cortina: A System for large scale,
content based Web Image Retrieval
 Conclusion

 There is a need for CBIR

 Image retrieval does not entail solving the general image
understanding problem. It may be sufficient that a
retrieval system present similar images, similar in some
user-defined sense.
 Interaction
 The need for database
 The semantic gap