Emre Celebi

Introduction
Color Spaces
Color Image Quantization
Color Image Filtering
Color Feature Extraction
Color Image Segmentation
Fundamentals of Color Image Processing
M. Emre Celebi
Department of Computer Science

Louisiana State University in Shreveport
ecelebi@lsus.edu
http://www.lsus.edu/emre-celebi
February 21, 2012
M. Emre Celebi Fundamentals of Color Image Processing

Introduction
Color Spaces
Table of Contents
1 Introduction
2 Color Spaces
3 Color Image Quantization
4 Color Image Filtering
5 Color Feature Extraction
6 Color Image Segmentation

Introduction
Color Spaces
Color and Colorimetry
Color: A phenomenon that relates to the physics of light,

chemistry of matter, geometric properties of objects, and
human visual perception.

Introduction
Color Spaces

Color is a purely psychological phenomenon.

Introduction
Color Spaces

Color is a purely psychological phenomenon.
Colorimetry: Science of color measurement and description

Introduction
Color Spaces
Fields Related to Color
Study of color theory and its applications spans various fields:

Physics

Introduction
Color Spaces

Physics
Chemistry

Introduction
Color Spaces

Physics
Chemistry
Optics

Introduction
Color Spaces

Physics
Chemistry
Optics
Colorimetry

Introduction
Color Spaces

Physics
Chemistry
Optics
Colorimetry
Physiology

Introduction
Color Spaces

Physics
Chemistry
Optics
Colorimetry
Physiology
Psychology

Introduction
Color Spaces

Physics
Chemistry
Optics
Colorimetry
Physiology
Psychology
Vision science

Introduction
Color Spaces

Physics
Chemistry
Optics
Colorimetry
Physiology
Psychology
Vision science
Image processing

Introduction
Color Spaces

Physics
Chemistry
Optics
Colorimetry
Physiology
Psychology
Vision science
Image processing
Computer vision

Introduction
Color Spaces
Scope of This Tutorial
This tutorial is about Color Image Processing (CIP) and

not Color Imaging.

Introduction
Color Spaces

not Color Imaging.
We will not explore topics such as

Introduction
Color Spaces

not Color Imaging.
Physics of light

Introduction
Color Spaces

not Color Imaging.
Physics of light
Chemistry of matter

Introduction
Color Spaces

not Color Imaging.
Physics of light
Chemistry of matter
Human vision

Introduction
Color Spaces

not Color Imaging.
Physics of light
Chemistry of matter
Human vision
Color perception

Introduction
Color Spaces

not Color Imaging.
Physics of light
Chemistry of matter
Human vision
Color perception
Color image acquisition

Introduction
Color Spaces

not Color Imaging.
Physics of light
Chemistry of matter
Human vision
Color perception
Color appearance models

Introduction
Color Spaces

not Color Imaging.
Physics of light
Chemistry of matter
Human vision
Color perception
Color management

Introduction
Color Spaces

not Color Imaging.
Physics of light
Chemistry of matter
Human vision
Color perception
Color management
History of color imaging

Introduction
Color Spaces
CIP: Motivation
Advances in imaging and computer hardware have led to an

explosion in the use of color images in a variety of applications
including
Medical imaging

Introduction
Color Spaces
CIP: Motivation

including
Medical imaging
Content based image retrieval

Introduction
Color Spaces
CIP: Motivation

including
Medical imaging
Digital inpainting

Introduction
Color Spaces
CIP: Motivation

including
Medical imaging
Digital inpainting
Digital multimedia

Introduction
Color Spaces
CIP: Motivation

including
Medical imaging
Digital inpainting
Digital multimedia
Visual quality inspection

CIP: Challenges
Multivariate nature of color image data presents numerous

challenges for researchers:
Color is a surprisingly complex phenomenon.
CIP: Challenges

Methods developed since 1960s are mostly inapplicable.
CIP: Challenges

There is no universal method for ordering multivariate data
(order-statistics filters, morphology, etc.)
CIP: Challenges

Processing 3-D data sets is computationally expensive (in
many applications, LUTs are of limited use).
CIP: Challenges

Processing 3-D data sets is computationally expensive (in
many applications, LUTs are of limited use).
Many problems become intractable with increasing
dimensionality, e.g. for D = 1 quantization can be performed
in O(N) time; otherwise the problem is NP-hard.
Introduction
Color Spaces
Coverage of CIP in Textbooks
Traditional IP textbooks have inadequate coverage of CIP.

Introduction
Color Spaces

Most textbooks touch upon CIP by including a chapter on
color spaces and some outdated color segmentation methods.

Introduction
Color Spaces

Some authors even suggest applying monochrome techniques
to color channels independently.

Introduction
Color Spaces

Some authors even suggest applying monochrome techniques
to color channels independently.
Good books dedicated to CIP are often of edited type.

Introduction
Color Spaces
The Colour Image Processing Handbook
Edited by S.J. Sangwine and R.E.N. Horne, published by Chapman

& Hall (1998). Good but relatively outdated content.

Introduction
Color Spaces
Color Image Processing and Applications

Authored by K.N. Plataniotis and A.N. Venetsanopoulos, published
by Springer (2000). Very strong on certain topics, but relatively
weak on others.

Introduction
Color Spaces
Digital Color Imaging Handbook
Edited by G. Sharma, published by CRC Press (2003). Mostly

focused on imaging, but the chapters on CIP are excellent.

Introduction
Color Spaces
Color Image Processing: Methods and Applications
Edited by R. Lukac and K.N. Plataniotis, published by CRC Press

(2006). Authoritative and relatively recent reference.

Introduction
Color Spaces
Digital Color Image Processing

Authored by A. Koschan and M. Abidi, published by Wiley (2008).
Application-oriented book based mostly on the authors own
research experiences.

Introduction
Color Spaces
Color Image Processing with Biomedical Applications
Authored by R.M. Rangayyan, B. Acha, and C. Serrano, published

by SPIE Press (2011). The only book that comes close to being a
textbook.

Introduction
Color Spaces
Advanced Color Image Processing and Analysis
Edited by C. Fernandez-Maloigne, to be published by Springer

(2012). Look forward to reading this one.

Introduction
Color Spaces
Table of Contents
1 Introduction
2 Color Spaces

Introduction
Color Spaces
Color Spaces
3-dimensional arrangements of color sensations.

Introduction
Color Spaces
Color Spaces

Each color perception is represented as a point.

Introduction
Color Spaces
Color Spaces

Dozens of color spaces in existence.

Introduction
Color Spaces
Color Spaces

No single color space suits all possible applications.

Introduction
Color Spaces
Color Spaces

No single color space suits all possible applications.
Fusion of results from multiple color spaces is often useful.

Introduction
Color Spaces
Criteria for Evaluating Color Spaces
Device independence (crucial for heterogeneous image sets)

Introduction
Color Spaces

Perceptual uniformity (can perceived color differences be
quantified using numerical formulae?)

Introduction
Color Spaces

Linearity of the transformation (computational convenience)

Introduction
Color Spaces

Intuitiveness (important for manual color selection)

Introduction
Color Spaces

Intuitiveness (important for manual color selection)
Invariance (viewpoint, object shape, highlights, illumination
direction, illuminant intensity/color)

Introduction
Color Spaces
RGB Color Space
Default color space in most applications.
(a) RGB color cube (wire) (b) RGB color cube (solid)
Adapted from L. Busin, N. Vandenbroucke, and L. Macaire

Introduction
Color Spaces
Properties of RGB
Not device independent

Introduction
Color Spaces
Properties of RGB

Not perceptually uniform

Introduction
Color Spaces
Properties of RGB

No transformation necessary

Introduction
Color Spaces
Properties of RGB

Not intuitive

Introduction
Color Spaces
Properties of RGB

Not intuitive
Not invariant

Introduction
Color Spaces
Gamma Correction
Image acquisition devices apply gamma correction to transform

linear-light L = {R, G , B} to nonlinear L = {R , G , B }:

4.5L L 0.018
L = 1/
1.099L 0.099 otherwise
Typically, = 2.2.

Introduction
Color Spaces
Inverse Gamma Correction
Display devices apply inverse gamma correction to nonlinear R ,

G , B values:
( L
L= 4.5 L 0.081
L +0.099
1.099 otherwise

(Inverse) Gamma Correction
Introduction
Color Spaces
rgb Color Space
Normalized r, g, b are obtained by factoring out illumination

intensity, R + G + B, from R, G, B values:
r = R+GR +B
G
g = R+G +B
B
b = R+G +B = 1 (r + g )

Introduction
Color Spaces
Properties of rgb

Introduction
Color Spaces
Properties of rgb


Introduction
Color Spaces
Properties of rgb

Not linear (unstable when R + G + B is small)

Introduction
Color Spaces
Properties of rgb

Not intuitive

Introduction
Color Spaces
Properties of rgb

Not intuitive
Invariant to viewpoint, object shape, illumination intensity

Introduction
Color Spaces
XYZ Color Space
Intermediate color space linearly related to RGB:

X R
Y = M33 G
Z B
For ITU-R BT.709 (D65 illuminant):

0.4124564 0.3575761 0.1804375
M = 0.2126729 0.7151522 0.0721750
0.0193339 0.1191920 0.9503041

Introduction
Color Spaces
Properties of XYZ
Device independent

Introduction
Color Spaces
Properties of XYZ
Device independent

Introduction
Color Spaces
Properties of XYZ
Device independent
Linear

Introduction
Color Spaces
Properties of XYZ
Device independent
Linear
Not intuitive

Introduction
Color Spaces
Properties of XYZ
Device independent
Linear
Not intuitive
Not invariant

Introduction
Color Spaces
Hue-Saturation-Intensity (HSI) Color Space

Hue:

0.5(RG +RB)
H = arccos
(RG )2 +(RB)(G B)
if (B > G )
H = 2 H
Saturation:
min (R, G , B)
S =13
R +G +B
Intensity:
R +G +B
I =
3
Introduction
Color Spaces
Alternative Hue Expression
if R > B and G > hB i

3(G R)
H = 3 + arctan G B+RB
else if G > R h i
3(BG )
H = + arctan BR+G R
else if B > G h i
5 3(RB)
H= 3 + arctan RG +BG
else if R > B
H=0
else
H = undefined
Reference: J.R. Kender, Instabilities in Color Transformations, Proc. of the IEEE Int. Conf. on Pattern
Recognition and Image Processing, pp. 266274, 1977.

Introduction
Color Spaces
Problems with HSI
It is not clear whether linear or nonlinear RGB should be used.

Introduction
Color Spaces
Problems with HSI

Hue becomes unstable when Saturation or Intensity is 0.

Introduction
Color Spaces
Problems with HSI

Hue is discontinuous at 360 (arithmetic mean should not be
used on Hue values).

Introduction
Color Spaces
Problems with HSI

Saturation becomes unstable when Intensity is 0.

Introduction
Color Spaces
Problems with HSI

Saturation becomes unstable when Intensity is 0.
Intensity does not capture the lightness perception of HVS.

Introduction
Color Spaces
Properties of HSI

Introduction
Color Spaces
Properties of HSI


Introduction
Color Spaces
Properties of HSI

Not linear

Introduction
Color Spaces
Properties of HSI

Not linear
Intuitive

Introduction
Color Spaces
Properties of HSI

Not linear
Intuitive
Hue not invariant to illuminant color

Introduction
Color Spaces
Properties of HSI

Not linear
Intuitive
Saturation not invariant to highlights and illuminant color.

Introduction
Color Spaces
Properties of HSI

Not linear
Intuitive
Saturation not invariant to highlights and illuminant color.
Intensity not invariant

Introduction
Color Spaces
Confusion around Hue-Based Color Spaces
There are many hue-based color spaces:

HSL: Hue-Saturation-Lightness
HSI: Hue-Saturation-Intensity
HSV: Hue-Saturation-Value
HSB: Hue-Saturation-Brightness
HVC: Hue-Value-Chroma
HMMD: Hue-Max-Min-Diff
HWB: Hue-Whiteness-Blackness
What is worse is that the literature contains different transforms
for the same color space !

Introduction
Color Spaces
Why are Hue-Based Color Spaces so Popular?
They are more intuitive than most other color spaces (manual
color selection).

Introduction
Color Spaces
color selection).
They decouple luminance from color information
(segmentation can be done on Hue and Saturation only).

Introduction
Color Spaces
color selection).
Hue is invariant to viewpoint, object shape, highlights, and
illuminant intensity.

Introduction
Color Spaces
color selection).
Hue is invariant to viewpoint, object shape, highlights, and
illuminant intensity.
Computationally efficient transformations exist between RGB
and these spaces.

Introduction
Color Spaces
Acceleration of RGB to HSI Transformation
arctan(.) computation can be accelerated using the following

minimax polynomial:
arctan(x) = 1.801283e-04 + 1.739333x 2.039848e-02x 2

2.065512x 3 + 2.052837x 4 6.591729e-01x 5
The approximation error over [1.0, 1.0) is 1.801311e-04.

Introduction
Color Spaces
I1I2 I3 Color Space
Channels of the RGB space are often highly correlated.
Reference: Y.I. Ohta, T. Kanade, and T. Sakai, Color Information for Region Segmentation, Computer Graphics
and Image Processing, vol. 13, no. 3, pp. 222241, 1980.

Introduction
Color Spaces
I1I2 I3 Color Space

Based on 8 color images, Ohta et al. determined that
I1 = R+G3 +B
I2 = R B
I3 = 2G RB
2
are roughly uncorrelated.

Introduction
Color Spaces
I1I2 I3 Color Space

Based on 8 color images, Ohta et al. determined that
I1 = R+G3 +B
I2 = R B
I3 = 2G RB
2
are roughly uncorrelated.

Segmentation can be performed on I1 and I2 only.

Introduction
Color Spaces
Properties of I1I2 I3

Introduction
Color Spaces


Introduction
Color Spaces

Linear

Introduction
Color Spaces

Linear
Not intuitive

Introduction
Color Spaces

Linear
Not intuitive
Not invariant

Introduction
Color Spaces
CIELAB Color Space
L = 116f (Y /Y0 ) 16
a = 500 [f (X /X0 ) f (Y /Y0 )]
b = 200 [f (Y /Y0 ) f (Z /Z0 )]
(
t 1/3 t > 0.008856
f (t) =
7.787t + 16/116 t 0.008856
where X0 , Y0 , and Z0 are the tristimulus values of the reference
white, e.g. for D65: X0 = 0.950456, Y0 = 1.0, and Z0 = 1.089058.

Introduction
Color Spaces
Properties of CIELAB
Device independent

Introduction
Color Spaces
Device independent
Approximately perceptually uniform (Euclidean distance
correlates well with perceived color differences)

Introduction
Color Spaces
Device independent
Not linear

Introduction
Color Spaces
Device independent
Not linear
Not intuitive

Introduction
Color Spaces
Device independent
Not linear
Not intuitive
Not invariant

Introduction
Color Spaces
Acceleration of RGB to CIELAB Transformation
cbrt(.) computation can be accelerated using the following

minimax rational:
P(x) = 3.927283e-05 + 1.392318e-02x+

4.114739e-01x 2 + 1.734853x 3 + 8.679223e-01x 4
Q(x) = 4.022100e-04 + 5.414536e-02x+
8.221526e-01x 2 + 1.800167x 3 + 3.513617e-01x 4
The approximation error of P(x)/Q(x) over [0.008856, 1] is

3.856930e-05.

Introduction
Color Spaces
Table of Contents
1 Introduction
2 Color Spaces

(a) Input (129, 059 colors) (b) Output (256 colors)
Introduction
Color Spaces
Color Quantization (CQ)
Typical color images contain tens of thousands of colors.

Introduction
Color Spaces

Problems with too many colors:

Introduction
Color Spaces

Display (device restrictions)

Introduction
Color Spaces

Storage (space)

Introduction
Color Spaces

Storage (space)
Processing (time)

Introduction
Color Spaces

Storage (space)
Processing (time)
Analysis (complexity)

Introduction
Color Spaces

Storage (space)
Processing (time)
Analysis (complexity)
CQ is an important problem in IP/CV.

Introduction
Color Spaces
Classification of CQ Methods
Adapted from Z. Xiang

Introduction
Color Spaces
Anatomy of a CQ Method
1 Palette design: Construct a palette that represents the

original image colors.

Introduction
Color Spaces

2 Pixel mapping: Assign each original image pixel to one of the
palette colors.

Introduction
Color Spaces

2 Pixel mapping: Assign each original image pixel to one of the
palette colors.
Palette design often dominates the computational time.

Introduction
Color Spaces
Image-Independent CQ Methods
Methods that use a universal (fixed) palette.

Introduction
Color Spaces

Computationally cheap (palette design is done offline)

Introduction
Color Spaces

Computationally cheap (palette design is done offline)
Poor quality results (palette independent of image contents)

Universal Palette Example
(a) Input (183, 525 colors) (b) CGA mapped output
(c) CGA palette (16 colors)

Introduction
Color Spaces
Preprocessing for Image-Dependent CQ
5 5 5 uniform quantization is commonly employed to reduce

CPU time.
Adapted from W. Burger and M.J. Burge
Preprocessing (cont.)
(a) Input (b) Output (4 4 4)
Disadvantages of Uniform Quantization

Not possible to control the number of colors (K ).

Does not consider the nonlinearity of the HVS.

Does not consider the nonlinearity of the HVS.
Leads to false contouring.
Introduction
Color Spaces
Image-Dependent, Context-Sensitive CQ Methods
Methods that consider the neighborhood of a pixel

Introduction
Color Spaces

Quantization and dithering are performed simultaneously

Introduction
Color Spaces

Computationally very expensive

Introduction
Color Spaces

Computationally very expensive
Very few researchers have pursued this direction

Introduction
Color Spaces
Image-Dependent, Context-Free CQ Methods
Majority of the CQ methods fall into this category

Introduction
Color Spaces

19822000: Fast methods based on approximate hierarchical
divisive clustering algorithms

Introduction
Color Spaces

19822000: Fast methods based on approximate hierarchical
divisive clustering algorithms
2000Present: Slow, optimization based methods (KM, FCM,
KHM, ANN, GA, SA, PSO, ACO, VNS, . . . )

Popularity: Simplest Context-Free CQ Method
(a) Input (230, 427 cols.) (b) Output (256 cols.)
Disadvantages
Performs poorly on images with a wide range of colors.
Disadvantages
Disregards colors in sparse regions of the color space.
Disadvantages
Disregards colors in sparse regions of the color space.
Performs poorly when K is small.
Introduction
Color Spaces
Hierarchical Context-Free CQ Methods
Two variants: Agglomerative and Divisive

Introduction
Color Spaces
Hierarchical Context-Free CQ Methods
Two variants: Agglomerative and Divisive

Mostly adaptations of existing data clustering algorithms

Introduction
Color Spaces
Agglomerative Context-Free CQ Methods
Start with N singleton clusters each containing a single color.

Introduction
Color Spaces

Merge these clusters repeatedly until K colors remain.

Introduction
Color Spaces

Computationally very expensive, typically O(N 2 )

Introduction
Color Spaces

Computationally very expensive, typically O(N 2 )
Few examples in the literature

Introduction
Color Spaces
Divisive Context-Free CQ Methods
Start with a single cluster that contains all N colors.

Introduction
Color Spaces

Recursively subdivide this cluster until K clusters remain.

Introduction
Color Spaces

Computationally cheap, typically O(N)

Introduction
Color Spaces

Computationally cheap, typically O(N)
Numerous variants in the literature

Classification of Divisive Context-Free CQ Methods
Adapted from L. Brun and A. Tremeau

Median-Cut: Most Common Divisive Context-Free CQ
Method
(a) Median-cut procedure (Adapted from L. Velho,

A. Frery, and J. Gomes)
(b) Input (183, 525 cols.) (c) Output (8 cols.)

Introduction
Color Spaces
K-Means: Best Trade-Off Between Quality and Speed
Most popular non-hierarchical clustering algorithm

Introduction
Color Spaces

Minimizes the SSE (Sum of Squared Error) measure

Introduction
Color Spaces

SSE minimization is NP-hard even for K = 2 or D = 2

Introduction
Color Spaces

SSE minimization is NP-hard even for K = 2 or D = 2
Lloyds heuristic is commonly used to obtain an approximate
solution

K-Means: Step 1
Set the target number of clusters (K )

Adapted from A. Moore
K-Means: Step 2
Randomly initialize K cluster centers

K-Means: Step 3
Determine the closest center to each data point

K-Means: Step 4
Determine the centroid of each cluster

K-Means: Step 5
Iterate until termination criterion is satisfied

Introduction
Color Spaces
Advantages/Disadvantages of K-Means
Advantages
Simple, versatile, easy to implement
Disadvantages

Introduction
Color Spaces
Advantages
Time efficient: O(NK ) per iteration
Disadvantages

Introduction
Color Spaces
Advantages
Guaranteed to terminate (quadratic convergence rate)
Disadvantages

Introduction
Color Spaces
Advantages
Disadvantages
Easily gets stuck in local minima

Introduction
Color Spaces
Advantages
Disadvantages
Highly sensitive to initialization

Introduction
Color Spaces
Advantages
Disadvantages
Highly sensitive to initialization
Computationally expensive due to iterative nature, O(NKT )

Introduction
Color Spaces
Adapting K-Means to CQ
Apply 2 : 1 subsampling in horizontal and vertical directions.

Introduction
Color Spaces

Assign each sample a weight proportional to its frequency.

Introduction
Color Spaces

Exploit triangle inequality to reduce distance computations.

Introduction
Color Spaces

Exploit triangle inequality to reduce distance computations.
Initialize centers using a divisive context-free CQ method.

K-Means (KM) vs. Median-Cut (MC) [K = 8]
(a) Input (183, 525 colors)
(b) MC output (MSE = 1215, (c) KM output (MSE = 724,

time = 0.0 msec.) time = 21.5 msec.)
Introduction
Color Spaces
Table of Contents
1 Introduction
2 Color Spaces

Introduction
Color Spaces
Noise in Color Images
Noise in color images is often due to acquisition (sensor noise)

or transmission (channel noise).

Introduction
Color Spaces

Sensor noise is usually modeled using Gaussian or Possion
distributions.

Introduction
Color Spaces

Sensor noise is usually modeled using Gaussian or Possion
distributions.
Channel noise is typically characterized by impulsive sequences
(short-duration, high-energy spikes).

Introduction
Color Spaces
Common Types of Image Noise
Adapted from B. Smolka

Examples of Noisy Images 1
Digital image corrupted by transmission noise

Scanned image of an artwork

(a) Adequate light, short exposure (b) Low light, long exposure
(a) ISO 100, f/5.6, 1/350 s (b) Zoomed section of (a)
(c) ISO 1600, f/5.6, 1/4000 s (d) Zoomed section of (c)

Introduction
Color Spaces
Problems with Image Noise
Lowers the perceptual quality of the images

Introduction
Color Spaces

Complicates subsequent image analysis tasks (segmentation,
feature extraction, object detection, etc.)

Introduction
Color Spaces

Complicates subsequent image analysis tasks (segmentation,
feature extraction, object detection, etc.)
Reduces the compressibility of the images

Uncorrelated Impulsive Noise Model

ok with probability 1 ,
xk =
rk with probability
Symbols
o = {o1 , o2 , o3 }: original color vector
x = {x1 , x2 , x3 }: noisy color vector
r1 , r2 , r3 : random, uncorrelated variables
: channel corruption probability
Correlated Impulsive Noise Model

o with probability 1 ,
{r1 , o2 , o3 } with probability 1 ,

x= {o1 , r2 , o3 } with probability 2 ,
{o1 , o2 , r3 } with probability 3 ,

{r1 , r2 , r3 } with probability (1 (1 + 2 + 3 ))

Symbols
o = {o1 , o2 , o3 }: original color vector
x = {x1 , x2 , x3 }: noisy color vector
r = {r1 , r2 , r3 }: random noise vector
: sample corruption probability
1 , 2 , 3 : corruption probabilities for R, G, B, respectively
Marginal vs. Vector Processing
(a) Marginal processing (b) Vector processing

Marginal vs. Vector Processing Example 1
(a) Input (3 colors)
(b) Marginal median (c) Vector median fil-

filter output (4 colors) ter output (3 colors)
Marginal vs. Vector Processing Example 2
(a) Input (37, 552 colors) (b) 30% noisy (57, 434 col-
ors)
(c) Marginal median filter (d) Vector median filter out-

output (113, 230 colors) put (26, 220 colors)
Marginal vs. Vector Processing Example 2 (cont.)
(a) Marginal median filter output (b) Vector median filter output
Introduction
Color Spaces
Common Evaluation Measures
M N
1 XX
MAE (A, B) = d1 (A(r , c), B(r , c))
3MN
r =1 c=1
M X N
1 X
MSE (A, B) = d22 (A(r , c), B(r , c))
3MN
r =1 c=1
Symbols
A and B: M N color images being compared
d1 : L1 (City-Block) distance
d2 : L2 (Euclidean) distance

MSE and Human Perception 1

MSE and Human Perception 2
(a) Original (b) Luminance shift (c) Contrast stretch
(d) Salt & Pepper (e) Blur (f) JPEG
(b)(f) have an MSE of 144

Adapted from Z. Wang
Introduction
Color Spaces
Multivariate Ordering Methods
There is no universal method for ordering multivariate data:

Marginal (M) ordering: Vectors are ordered in each
component independently.

Introduction
Color Spaces

Conditional (C) ordering: Vectors are ordered based on M
ordering of one of the components.

Introduction
Color Spaces

Partial (P) ordering: Vectors are partitioned into smaller
groups that are then ordered.

Introduction
Color Spaces

Partial (P) ordering: Vectors are partitioned into smaller
groups that are then ordered.
Reduced (R) ordering: Vectors are first reduced to scalars
using a suitable distance measure. These scalars are then
ordered.

Vector Ordering (Original Samples)

Marginal Ordering

Conditional Ordering on x1

Conditional Ordering on x2

Partial Ordering

Reduced Ordering

Introduction
Color Spaces
Sliding Window Filtering
x1 x2 x3
x x6
4
x5
x 7 x8 x 9
Most images are nonstationary in nature

Introduction
Color Spaces
x1 x2 x3
x x6
4
x5
x 7 x8 x 9

Filters often operate over a small image region (sliding
window)

Introduction
Color Spaces
x1 x2 x3
x x6
4
x5
x 7 x8 x 9

Filters often operate over a small image region (sliding
window)
A square window is commonly used due to its efficiency

Introduction
Color Spaces
Vector Median Filter (VMF)

1 Calculate the cumulative distance Di for each vector xi :
X n
X
Di = d(xi , xj ) = dij
xj W j=1

Introduction
Color Spaces

1 Calculate the cumulative distance Di for each vector xi :
X n
X
Di = d(xi , xj ) = dij
xj W j=1
2 The ordered sequence of scalars
D(1) D(2) . . . D(n)
implies the same ordering in the corresponding vectors
x(1) x(2) . . . x(n)

Introduction
Color Spaces
Vectors with lower ranks are representative (typical) vectors in

their neighborhoods.

Introduction
Color Spaces

Vectors with higher ranks are often outliers.

Introduction
Color Spaces

Vectors with higher ranks are often outliers.
Most filters define the lowest ranked input vector, x(1) , as the
output vector:
y = arg min Di
xi W

Illustration of VMF

Introduction
Color Spaces
Choice of Distance Function in VMF
Typically L1 and L2 distances are employed.

Introduction
Color Spaces

L1 is not only much faster than L2 , but also it produces
higher quality output.

Introduction
Color Spaces

L22 is equivalent to marginal mean filter (very efficient, but
results in heavy blurring).

Introduction
Color Spaces

L22 is equivalent to marginal mean filter (very efficient, but
results in heavy blurring).
Several fuzzy distance functions outperform L1 and L2 .

Introduction
Color Spaces
Fuzzy Magnitude Similarity Function in VMF
3
K
Y min(xik , xjk ) + K
sfms (xi , xj ) =
max(xik , xjk ) + K
k=1
where K is a user-defined parameter.

Reference: S. Morillas, V. Gregori, G. Peris-Fajarnes, and P. Latorre, A Fast Impulsive Noise Color Image Filter
Using Fuzzy Metrics, Real-Time Imaging, vol. 11, no. 5/6, pp. 417428, 2005.

Introduction
Color Spaces
Spatio-Chromatic Fuzzy Similarity Function in VMF
C ,t C t
scfs (xi , xj ) =
C + d2 (xi , xj ) t + max (|ri rj | , |ci cj |)
where (ri , ci ) and (rj , cj ) are the spatial coordinates of xi and xj ,

respectively, and C and t are user-defined parameters.
S. Morillas, V. Gregori, G. Peris-Fajarnes, and A. Sapena, New Adaptive Vector Filter Using Fuzzy Metrics,
Journal of Electronic Imaging, vol. 16, no. 3, pp. 033007, 2007.

Introduction
Color Spaces
Cosine Distance Function in VMF

3
P
xik xjk
k=1
dcos (xi , xj ) = = arccos

kxi k kxj k

VMF + dcos (., .) Basic Vector Directional Filter (BVDF).

Introduction
Color Spaces

3
P
xik xjk
k=1

kxi k kxj k


BVDF considers only directional information.

Introduction
Color Spaces

3
P
xik xjk
k=1

kxi k kxj k


BVDF considers only directional information.
It can be calculated efficiently by first normalizing the color
vectors to unit length and then using L2 distance.

Comparison of Distance/Similarity Functions for VMF
(a) Input (b) 10% noisy (MAE=6, (c) dcos (MAE=5, MSE=83)
MSE=1041)
(d) d1 (MAE=4, MSE=41) (e) sfms (MAE=4, MSE=41) (f) scfs (MAE=1, MSE=14)
Introduction
Color Spaces
Switching Vector Filters
VMF and its variants tend to modify pixels that are not
corrupted by noise.

Introduction
Color Spaces
corrupted by noise.
Switching filters employ noise detectors to avoid replacing
noise-free pixels.

Introduction
Color Spaces
corrupted by noise.
Switching filters employ noise detectors to avoid replacing
noise-free pixels.
These filters are often quite efficient since the filtering
operation is performed only on the noisy pixels.
Introduction
Color Spaces
Fast Peer Group Filter (FPGF)

xvmf if |{xi 6=C W s.t. kxC xi k T }| < m
y=
xC otherwise
where m and T denote the peer group size and threshold,

respectively.
Reference: B. Smolka and A. Chydzinski, Fast Detection and Impulsive Noise Removal in Color Images,
Real-Time Imaging, vol. 11, no. 5/6, pp. 389402, 2005.

Introduction
Color Spaces
Fast Fuzzy Noise Reduction Filter (FFNRF)
n n
(x , x ) < (x , x )
P P
xk
if sfms C i sfms k i
y= i =1 i =1
i 6=C

xC
otherwise

n
(x , x )
P
xk = arg max sfms k i
xk W i =1
i 6=C
Reference: S. Morillas, V. Gregori, G. Peris-Fajarnes, and P. Latorre, A Fast Impulsive Noise Color Image Filter
Using Fuzzy Metrics, Real-Time Imaging, vol. 11, no. 5/6, pp. 417428, 2005.

Comparison of Switching Vector Filters
(a) Input (b) 20% noisy (MAE=13,

MSE=1848)
(c) VMF (MAE=3, MSE=36) (d) FFNRF (MAE=1, (e) FPGF (MAE=1, MSE=25)
MSE=33)
Introduction
Color Spaces
Table of Contents
1 Introduction
2 Color Spaces

Introduction
Color Spaces
Color vs. Shape
Color is an important cue for image understanding.

Introduction
Color Spaces
Color vs. Shape

As compared to shape features, color features are generally
more robust to

Introduction
Color Spaces
Color vs. Shape

more robust to
Rotation

Introduction
Color Spaces
Color vs. Shape

more robust to
Rotation
Scaling

Introduction
Color Spaces
Color vs. Shape

more robust to
Rotation
Scaling
Occlusion

Introduction
Color Spaces
Color vs. Shape

more robust to
Rotation
Scaling
Occlusion
Viewpoint

Introduction
Color Spaces
Color vs. Shape

more robust to
Rotation
Scaling
Occlusion
Viewpoint
In addition, color features do not require segmentation.

Introduction
Color Spaces
Color vs. Shape

more robust to
Rotation
Scaling
Occlusion
Viewpoint
In addition, color features do not require segmentation.
Color features are often sensitive to lighting (color constancy
methods might help).

Introduction
Color Spaces
Desirable Characteristics of a Color Descriptor
include
Uniquely identifies the image/region

Introduction
Color Spaces
include
Cheap to compute

Introduction
Color Spaces
include
Cheap to compute
Low in dimensionality

Introduction
Color Spaces
include
Cheap to compute
Invariant to occlusion

Introduction
Color Spaces
include
Cheap to compute
Invariant to various geometric transformations

Introduction
Color Spaces
include
Cheap to compute
Invariant to various geometric transformations
Invariant to lighting

Introduction
Color Spaces
Color Histograms
1-/3-D data structure that captures the color distribution of a

(uniformly) quantized image

Introduction
Color Spaces
Color Histograms
1-/3-D data structure that captures the color distribution of a

(uniformly) quantized image
Popularized by Swain and Ballards seminal work:
M.J. Swain and D.H. Ballard, Color Indexing, International
Journal of Computer Vision, vol. 7, no. 1, pp. 1132, 1991.
(4, 730 Google Scholar citations)

Color Histogram Examples
(a) 230, 427 colors (b) Histogram of (a)
(c) 256 colors (d) Histogram of (c)

Introduction
Color Spaces
Computation of a Color Histogram
Quantize the 3-D color space (RGB, CIELAB, etc.) uniformly

to n = n1 n2 n3 cells.

Introduction
Color Spaces

Record the number of colors that fall into each cell.

Introduction
Color Spaces

Typically, lightness/luminance axis is quantized more coarsely.

Introduction
Color Spaces

Typically, lightness/luminance axis is quantized more coarsely.
High resolution (large n) histograms are unreliable and
expensive to match.

Introduction
Color Spaces
Distance Between Two Color Histograms
Given two n-bin flattened histograms:

A = hA1 , A2 , . . . , An i,
B = hB1 , B2 , . . . , Bn i
Histogram Intersection is defined as:
Pn
min (Ai , Bi )
Dhi (A, B) = i =1Pn
i =1 Bi
When N = ni=1 Ai = ni=1 Bi , Dhi is equivalent to L1 :

P P
n
1 X
1 Dhi (A, B) = |Ai Bi |
2N
i =1

Introduction
Color Spaces
Advantages/Disadvantages of Color Histograms
Advantages
Cheap to compute
Disadvantages

Introduction
Color Spaces
Advantages
Cheap to compute
Invariant to rotation and translation
Disadvantages

Introduction
Color Spaces
Advantages
Cheap to compute
Disadvantages
Spatial information is discarded

Introduction
Color Spaces
Advantages
Cheap to compute
Disadvantages
Moderate-to-high dimensionality

Introduction
Color Spaces
Advantages
Cheap to compute
Disadvantages
Moderate-to-high dimensionality
Sensitive to lighting

Introduction
Color Spaces
Quadratic Histogram Distance
L1 and L2 disregard the correlation between different colors.
Reference: J. Hafner, H.S. Sawhney, W. Equitz, M. Flickner, and W. Niblack, Efficient Color Histogram Indexing
for Quadratic Form Distance Functions, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 17,
no. 7, pp. 729736, 1995.

Introduction
Color Spaces

A = hA1 , A2 , . . . , An i and B = hB1 , B2 , . . . , Bn i
and an n n perceptual similarity matrix S,
Quadratic Histogram Distance is defined as:
h i1/2
Dqhd (A, B) = (A B) S (A B)T
no. 7, pp. 729736, 1995.

Introduction
Color Spaces

A = hA1 , A2 , . . . , An i and B = hB1 , B2 , . . . , Bn i
and an n n perceptual similarity matrix S,
Quadratic Histogram Distance is defined as:
h i1/2
Dqhd (A, B) = (A B) S (A B)T
When S is diagonal, Dqhd is equivalent to L2 .

no. 7, pp. 729736, 1995.

Introduction
Color Spaces
Quadratic Distance vs. Euclidean Distance
Given a histogram distribution of three colors: red, orange, and

blue with
1.0 0.9 0.0
S = 0.9 1.0 0.0
0.0 0.0 1.0
and two histograms:
A = h1.0, 0.0, 0.0i (pure red) and B = h0.0, 1.0, 0.0i (pure orange)
2 = 0.2 kA Bk2 = 2.0 because despite their similarity, the
Dqhd
two colors populate distinct bins of the histogram.

Introduction
Color Spaces
Calculation of the Perceptual Similarity Matrix S
Similarity between two colors ci and cj is given by:
kci cj k
sij = 1
max (kca cb k)
a,b{1,2,...,n}
S is positive semidefinite and thus Dqhd 0.

Color Moments
Simple descriptor that captures the statistical moments of

each color channel.
Reference: M. Stricker and M. Orengo, Similarity of Color Images, Proc. of SPIE 2420, pp. 381392, 1995.
Color Moments
Simple descriptor that captures the statistical moments of

each color channel.
Low-order moments of color channel k of an image
X = {x1 , x2 , . . . , xN } are given by:
N
1 P
Mean : k (X) = xik ,
N
i =1
N 1/2
1 P 2
Stdev : k (X) = N (xik k (X)) ,
i =1
N 1/3
Skewness : k (X) = N1 (xik k (X))3
P
i =1
Reference: M. Stricker and M. Orengo, Similarity of Color Images, Proc. of SPIE 2420, pp. 381392, 1995.
Introduction
Color Spaces
Matching Images Based on Color Moments
Weighted City-Block distance can be used to calculate the

channelwise distance of two images X and Y:
Mk (X, Y) = wk1 |k (X) k (Y)| +

wk2 |k (X) k (Y)| +
wk3 |k (X) k (Y)|
where wkj 0 (1 k, j 3) are user-defined weights.

Total distance between X and Y is given by:
3
X
Dm (X, Y) = Mk (X, Y)
k=1

Introduction
Color Spaces
Advantages/Disadvantages of Color Moments
Advantages
Cheap to compute
Disadvantages

Introduction
Color Spaces
Advantages
Cheap to compute
Very low in dimensionality
Disadvantages

Introduction
Color Spaces
Advantages
Cheap to compute
Disadvantages

Introduction
Color Spaces
Advantages
Cheap to compute
Disadvantages

Introduction
Color Spaces
Advantages
Cheap to compute
Disadvantages
Choice of wkj is not trivial

Introduction
Color Spaces
Advantages
Cheap to compute
Disadvantages
Choice of wkj is not trivial
Color distribution is treated as three independent 1-D
distributions rather than a true 3-D distribution

Color Histograms Lack Spatial Information
(a) 183, 525 colors (b) Histogram of (a)
(c) 183, 525 colors (d) Histogram of (c)

Introduction
Color Spaces
Color Coherence Vectors (CCVs)
Simple modification of color histograms that incorporates

spatial information.
Reference: G. Pass, R. Zabih, J. Miller, Comparing Images Using Color Coherence Vectors, Proc. of the 4th
ACM Int. Conf. on Multimedia, pp. 6573, 1996.

Introduction
Color Spaces

Pixels in histogram bins are classified as either coherent or
incoherent.

Introduction
Color Spaces

incoherent.
A pixel is coherent if the size of its connected-components
exceeds a threshold ; otherwise the pixel is incoherent.

Introduction
Color Spaces

incoherent.
A pixel is coherent if the size of its connected-components
exceeds a threshold ; otherwise the pixel is incoherent.
CCVs store the number of coherent versus incoherent pixels in
each bin.

Introduction
Color Spaces
An Example CCV ( = 4)
Input image Quant. image Label image

22 10 21 22 15 16 2 1 2 2 1 1 B C B B A A
24 21 13 20 14 17 2 2 1 2 1 1 B B C B A A
23 17 38 23 17 16 = 2 1 3 2 1 1 = B C D B A A
25 25 22 14 15 21 = 2 2 2 1 1 2 = B B B A A E
27 22 12 11 21 20 2 2 1 1 2 2 B B A A E E
24 21 10 12 22 23 2 2 1 1 2 2 B B A A E E
Conn. Comp. Table CCV

Label A B C D E Color 1 2 3
Color 1 2 1 3 2 = 12 20 0
Size 12 15 3 1 5 3 0 1

Introduction
Color Spaces
Distance Between Two CCVs
Given two n-bin CCVs:

A = h(1 , 1 ) , (2 , 2 ) , . . . , (n , n )i
B = h(1 , 1 ) , (2 , 2 ) , . . . , (n , n )i
distance between them is defined as:
n
X
i i + i i

Dccv (A, B) =
i =1
A andPB must have

Pn the same number of pixels, i.e.
n
N = i =1 Ai = i =1 Bi .

Color Correlogram
Informally, a correlogram is a table indexed by color pairs, where

the k-th entry for (i , j) specifies the probability of finding a pixel of
color cj at a distance k from a pixel of color ci in the image, i.e.
(k)
ci ,cj (I) = Pr x2 Icj |max {|r1 r2 | , |c1 c2 |} = k
x1 Ici ,
x2 I
where i , j { 1, 2, . . . , n} and k {d}.

Symbols
I: input image quantized to n colors c1 , c2 , . . . , cn .
(ri , ci ): spatial coordinates of pixel xi .
x Ic : pixel x is of color c.
Reference: J. Huang, S.R. Kumar, M. Mitra, W.J. Zhu, and R. Zabih, Spatial Color Indexing and Applications,
International Journal of Computer Vision, vol. 35, no. 3, pp. 245268, 1999.
Introduction
Color Spaces
Autocorrelogram
CCVs disregard the correlation between different colors.

Introduction
Color Spaces
Autocorrelogram

Correlograms address this drawback at the expense of O(n2 d)
space.

Introduction
Color Spaces
Autocorrelogram

space.
An autocorrelogram captures spatial correlation between
identical colors only:
(k) (k)
c (I) = c,c (I)

Introduction
Color Spaces
Autocorrelogram

space.
An autocorrelogram captures spatial correlation between
identical colors only:
(k) (k)
c (I) = c,c (I)
Therefore, autocorrelograms require only O(nd ) space.

Introduction
Color Spaces
Distance Between Two Correlograms

(k) (k)
X ci ,cj (I) ci ,cj (I )
D (I, I ) = (k) (k)
i,j{1,2,...,n}, 1 + ci ,cj (I) + ci ,cj (I )
k{d}

Introduction
Color Spaces
Table of Contents
1 Introduction
2 Color Spaces

Adapted from C. Fowlkes
Introduction
Color Spaces
Segmentation: Partitioning an image into disjoint regions

that are homogeneous w.r.t luminance, color, texture, etc.

Introduction
Color Spaces

Each of these regions should ideally correspond to a single
object or at least part of an object.

Introduction
Color Spaces

Despite half a decade of research, segmentation remains as
one of the most challenging problems IP/CV.

Introduction
Color Spaces

Despite half a decade of research, segmentation remains as
one of the most challenging problems IP/CV.
In most cases, there is no single correct segmentation.

Introduction
Color Spaces
Categorization of (Color) Image Segmentation Methods
Thresholding

Introduction
Color Spaces
Thresholding
Clustering

Introduction
Color Spaces
Thresholding
Clustering
Edge detection

Introduction
Color Spaces
Thresholding
Clustering
Edge detection
Region-based (shrinking/growing and/or splitting/merging)

Introduction
Color Spaces
Thresholding
Clustering
Edge detection
Morphological (watersheds)

Introduction
Color Spaces
Thresholding
Clustering
Edge detection
Active contours

Introduction
Color Spaces
Thresholding
Clustering
Edge detection
Active contours
Soft-computing (neural networks, fuzzy logic, evolutionary
computing, etc.)

Introduction
Color Spaces
Segmentation vs. Quantization
Segmentation methods aim to find homogeneous image

regions; quantization methods aim to find compact color
clusters.

Introduction
Color Spaces

clusters.
Segmentation methods incorporate spatial information;
quantization methods mostly do not.

Introduction
Color Spaces

clusters.
Segmentation methods determine the number of regions (K )
adaptively; quantization methods often require K as input.

Introduction
Color Spaces

clusters.
Segmentation methods determine the number of regions (K )
adaptively; quantization methods often require K as input.
Segmentation methods often incorporate cues other than
color (shape, texture, etc.); quantization methods consider
only color.

Introduction
Color Spaces
Color Image Segmentation Methods
Hundreds of methods proposed since the 1960s. We will explore

only three of the most prominent methods:
Normalized Cuts (Shi and Malik, 2000)

Introduction
Color Spaces

Mean Shift Clustering (Comaniciu and Meer, 2002)

Introduction
Color Spaces

Mean Shift Clustering (Comaniciu and Meer, 2002)
Statistical Region Merging (Nock and Nielsen, 2004)

Introduction
Color Spaces
Graph Based Image Segmentation
Represent the feature space as a weighted complete graph

G = (V, E).

Introduction
Color Spaces

G = (V, E).
Nodes of the graph are points in the feature space.

Introduction
Color Spaces

G = (V, E).
The weight on each edge, wij , is a function of the similarity
between nodes i and j.

Introduction
Color Spaces

G = (V, E).
The weight on each edge, wij , is a function of the similarity
between nodes i and j.
Objective: Partition V into disjoint sets V1 , V2 , . . ., Vm s.t.
similarity among the vertices in each set is high and across
different sets is low.

Introduction
Color Spaces
Graph Cuts
A graph G = (V, E) can be partitioned into two disjoint sets,

A and B by removing edges connecting the two parts.
Adapted from R. Szeliski

Introduction
Color Spaces
Graph Cuts

The cut between A and B is the sum of all weights being cut:
X
cut(A, B) = wij
i A,jB

Introduction
Color Spaces
Graph Cuts

X
cut(A, B) = wij
i A,jB
The optimal bipartitioning of G is the one that minimizes

this cut value.

Introduction
Color Spaces
Graph Cuts

X
cut(A, B) = wij
i A,jB
The optimal bipartitioning of G is the one that minimizes

this cut value.

Minimum Cut Criterion
Adapted from J. Shi and J. Malik

Introduction
Color Spaces
Normalized Cut Criterion
Alternative cut criterion:

1 1
Ncut(A, B) = cut(A, B) +
assoc(A, V) assoc(B, V)
where XX
assoc(A, V) = wij
i A jV

Introduction
Color Spaces
Normalized Association Criterion
assoc(A, A) assoc(B, B)
Nassoc(A, B) = +
assoc(A, V) assoc(B, V)
This criterion measures how tightly on average nodes within the
group are connected to each other. Since,
Ncut(A, B) = 2 Nassoc(A, B)
minimizing Ncut and maximizing Nassoc are in fact equivalent.

Introduction
Color Spaces
Minimization of the Ncut Criterion
Exact minimization of the Ncut criterion is NP-hard !
Symbols
W: |V| |V| symmetric matrix with W(i , j) = wij
D = diag(d1 , d2 , . . . , d|V| )
P
di = j wij : degree of a node i

Introduction
Color Spaces
Minimization of the Ncut Criterion
Exact minimization of the Ncut criterion is NP-hard !

The second smallest eigenvector of (D W) y = Dy gives
an approximate solution.
Symbols
W: |V| |V| symmetric matrix with W(i , j) = wij
D = diag(d1 , d2 , . . . , d|V| )
P
di = j wij : degree of a node i

Introduction
Color Spaces
Graph Representation of an Image
Construct weighted complete graph G = (V, E) by taking each

pixel as a node and connecting each pair of pixels by an edge.
Edge weights may be defined as:
!
kFi Fj k2 kxi xj k2
wij = exp
F2 s2
for pixels within a radius kxi xj k < r , where Fi and xi are pixel
i s feature vector and spatial location, respectively.

Introduction
Color Spaces
Segmentation Using Ncuts
Given the maximum number of regions, K :

1 Calculate the affinity matrix W and degree matrix D.

Introduction
Color Spaces

2 Solve (D W) y = Dy.

Introduction
Color Spaces

3 Use the second smallest eigenvector to bipartition the graph
s.t. Ncut is minimized.

Introduction
Color Spaces

4 Decide if the current partition should be subdivided by
checking the stability of the cut.

Introduction
Color Spaces

4 Decide if the current partition should be subdivided by
checking the stability of the cut.
5 Recursively divide the segmented partitions if necessary.

Ncuts Segmentation Example
Adapted from P. Arbelaez

Introduction
Color Spaces
Advantages/Disadvantages of Ncuts Method
Advantages
Flexibility (definition of W)
Disadvantages

Introduction
Color Spaces
Advantages
Good performance
Disadvantages

Introduction
Color Spaces
Advantages
Good performance
Disadvantages
O(N 3 ) time complexity (N: # data points)

Introduction
Color Spaces
Advantages
Good performance
Disadvantages
Max. number of regions, K , should be specified a priori

Introduction
Color Spaces
Advantages
Good performance
Disadvantages
Max. number of regions, K , should be specified a priori
Numerous other parameters

Introduction
Color Spaces
Feature Space Analysis
Definition
Feature Space: An D-dimensional abstract space where each
point corresponds to a parametric representation of a pixel or a set
of pixels.
Significant image features correspond to denser regions of

the feature space.

Introduction
Color Spaces
Definition
of pixels.

the feature space.
Histogram based techniques, Hough transform, etc. are
feature space analysis techniques.

Introduction
Color Spaces
Definition
of pixels.

the feature space.
Histogram based techniques, Hough transform, etc. are
feature space analysis techniques.
Ideally, a feature space should be continuous.

Feature Space Example
(a) 400 276 color image (b) Corresponding CIELUV color

space with 110, 400 data points
Adapted from D. Comaniciu and P. Meer

Introduction
Color Spaces
Non-Parametric Density Estimation
Feature space can be regarded as an empirical p.d.f.

Introduction
Color Spaces

Dense regions of the feature space correspond to local
maxima of the p.d.f. (modes of the unknown density)

Introduction
Color Spaces

Dense regions of the feature space correspond to local
maxima of the p.d.f. (modes of the unknown density)
A histogram is the simplest density estimation method.

Drawbacks of Histograms
They are not smooth
Two histograms with the same binwidth, but different end points
Adapted from T. Duong
They are not smooth
They depend on end points of the bins
They are not smooth
They depend on end points of the bins
They depend on width of the bins
Introduction
Color Spaces
Kernel Density Estimators (KDEs) vs. Histograms
When compared to histograms, KDEs are

Smooth

Introduction
Color Spaces

Smooth
Not dependent on end points

Introduction
Color Spaces

Smooth
Not dependent on end points
Dependent on bandwidth

Introduction
Color Spaces
Mean Shift (MS) Method
Modes of the underlying density f (x) are located among the

zeros of its gradient, i.e. f (x) = 0.

Introduction
Color Spaces

MS method is an elegant way to locate the zeros of the
gradient without estimating the density.

Introduction
Color Spaces

MS method is an elegant way to locate the zeros of the
gradient without estimating the density.
MS is the difference between the kernel weighted mean and
the kernel center:
Pn
xi G (x xi )
m(x) = Pi =1
n x
i =1 G (x xi )

Introduction
Color Spaces
Properties of MS
MS vector always points towards the direction of maximum

density increase.

Introduction
Color Spaces
Properties of MS

density increase.
MS procedure consists of the following successive steps:

Introduction
Color Spaces
Properties of MS

density increase.
1 Compute the MS vector

Introduction
Color Spaces
Properties of MS

density increase.
2 Translate the kernel by the MS vector

Introduction
Color Spaces
Properties of MS

density increase.
2 Translate the kernel by the MS vector
MS steps are large in sparse regions and small near local
maxima (adaptive gradient ascent)

Intuitive Description of the MS Procedure
Adapted from Y. Ukrainitz and B. Sarel







Introduction
Color Spaces
Mode Detection Using MS
Run the MS procedure to find stationary points of the density.

Introduction
Color Spaces

Determine saddle points by perturbing each stationary point
by a small random vector.

Introduction
Color Spaces

Fuse mode candidates that are closer than the bandwidth.

Introduction
Color Spaces

Fuse mode candidates that are closer than the bandwidth.
Points visited by MS procedures converging to a particular
mode represent a cluster of arbitrary shape.

Illustration of Mode Detection

Illustration of Mode Detection

MS Decomposition of the House Image
Next set of slides demonstrate meanshift decomposition of a 2-D

data set of 16, 000 points corresponding to the L u components
of the House image.
(a) Input (b) Input projected onto L u
Adapted from D. Comaniciu

Introduction
Color Spaces
Color Segmentation Using MS
Form 5-D feature vectors by concatenating a 2-D spatial

subvector and a 3-D range subvector for each pixel.
Reference: D. Comaniciu and P. Meer, Mean Shift: A Robust Approach Toward Feature Space Analysis, IEEE
Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 5, pp. 603619, 2002.

Introduction
Color Spaces

Multivariate kernel now becomes the product of a spatial
kernel and a range kernel.

Introduction
Color Spaces

Three intuitive parameters

Introduction
Color Spaces

Spatial kernel bandwidth (hs )

Introduction
Color Spaces

Range kernel bandwidth (hr )

Introduction
Color Spaces

Range kernel bandwidth (hr )
Minimum region size (M)

MS Color Segmentation Example 1
(hs , hr , M) = (16, 12, 100)

MS Color Segmentation Example 2
(hs , hr , M) = (16, 12, 100)

Introduction
Color Spaces
Advantages/Disadvantages of MS
Advantages
Generality
Disadvantages

Introduction
Color Spaces
Advantages
Generality
Noise robustness
Disadvantages

Introduction
Color Spaces
Advantages
Generality
Noise robustness
Ability to handle arbitrary clusters shapes/feature spaces
Disadvantages

Introduction
Color Spaces
Advantages
Generality
Noise robustness
Only one (intuitive) parameter: bandwidth
Disadvantages

Introduction
Color Spaces
Advantages
Generality
Noise robustness
Disadvantages

Introduction
Color Spaces
Advantages
Generality
Noise robustness
Disadvantages
Choice of kernel bandwidth is not trivial

Introduction
Color Spaces
Statistical Region Merging (SRM)
Very simple, extremely efficient segmentation method
Reference: R. Nock and F. Nielsen, Statistical Region Merging, IEEE Transactions on Pattern Analysis and
Machine Intelligence, vol. 26, no. 11, pp. 14521458, 2004.

Introduction
Color Spaces

Includes two components: merging predicate and order to test
region mergings

Introduction
Color Spaces

region mergings
No color space xforms, color quantization, or smoothing

Introduction
Color Spaces

region mergings
O(N) time complexity (N: # pixels in the input image)

Introduction
Color Spaces

region mergings
O(N) time complexity (N: # pixels in the input image)
Only one (intuitive) parameter, Q, that controls the
coarseness of the segmentation

Merging Predicate

true if a {R, G,
pB},
Ra Ra b 2 (R) + b 2 (R )

P R, R =
false otherwise

where s
ln(6 |I |2 R|R| )

b(R) = g
2Q |R|
Symbols
I : Input image with |I | pixels
g : Number of gray levels (g = 256 for 8-bit images)
R and R : The two regions being tested
Ra : Average for color channel a in region R
Rp : Set of regions with p pixels
Introduction
Color Spaces
SRM Algorithm
Symbols
SI : Set of all pairs of adjacent pixels
R(p): Region to which pixel p belongs at any given time
input : Image I
output: A segmentation of I
SI = counting sort(SI , f );
for (i = 0; i < |SI | ; i = i + 1) do
(pi , pi ) is the i -th pair of adjacent pixels in SI ;
if R(pi ) 6= R(pi ) and P(R(pi ), R(pi )) = true then
merge(R(pi ), R(pi ));
end
end
Introduction
Color Spaces
Choice of f (., .)
Color variations within the regions should be reasonably

smaller than those between the regions.

Introduction
Color Spaces
Choice of f (., .)
Color variations within the regions should be reasonably

smaller than those between the regions.
Simplest choice for f (., .):
f (p, p ) = max fa (p, p )

a{R,G,B}
where
fa (p, p ) = pa pa


Influence of Q on Segmentation
(a) Input (b) Output (Q = 2)
(c) Output (Q = 8) (d) Output (Q = 32)

More SRM Segmentation Examples
(a) Input (b) Output (Q = 256)
(c) Input (d) Output (Q = 256)

Introduction
Color Spaces
Acknowledgments
SIP 2011 organizers

National Science Foundation
(Grant Nos MRI-0959583 and RUI-1117457)
Prof. Bogdan Smolka

Emre Celebi

Enviado por

Dados do documento

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

Emre Celebi

Enviado por

Direitos autorais:

Formatos disponíveis

Introduction

Fundamentals of Color Image Processing

Department of Computer Science

February 21, 2012

M. Emre Celebi Fundamentals of Color Image Processing

3 Color Image Quantization

4 Color Image Filtering

5 Color Feature Extraction

6 Color Image Segmentation

M. Emre Celebi Fundamentals of Color Image Processing

Color and Colorimetry

Color: A phenomenon that relates to the physics of light,

M. Emre Celebi Fundamentals of Color Image Processing

Color and Colorimetry

Color: A phenomenon that relates to the physics of light,

M. Emre Celebi Fundamentals of Color Image Processing

Color and Colorimetry

Color: A phenomenon that relates to the physics of light,

M. Emre Celebi Fundamentals of Color Image Processing

Fields Related to Color

Study of color theory and its applications spans various fields:

M. Emre Celebi Fundamentals of Color Image Processing

Fields Related to Color

Study of color theory and its applications spans various fields:

M. Emre Celebi Fundamentals of Color Image Processing

Fields Related to Color

Study of color theory and its applications spans various fields:

M. Emre Celebi Fundamentals of Color Image Processing

Fields Related to Color

Study of color theory and its applications spans various fields:

M. Emre Celebi Fundamentals of Color Image Processing

Fields Related to Color

Study of color theory and its applications spans various fields:

M. Emre Celebi Fundamentals of Color Image Processing

Fields Related to Color

Study of color theory and its applications spans various fields:

M. Emre Celebi Fundamentals of Color Image Processing

Fields Related to Color

Study of color theory and its applications spans various fields:

M. Emre Celebi Fundamentals of Color Image Processing

Fields Related to Color

Study of color theory and its applications spans various fields:

M. Emre Celebi Fundamentals of Color Image Processing

Fields Related to Color

Study of color theory and its applications spans various fields:

M. Emre Celebi Fundamentals of Color Image Processing

Scope of This Tutorial

This tutorial is about Color Image Processing (CIP) and

M. Emre Celebi Fundamentals of Color Image Processing

Scope of This Tutorial

This tutorial is about Color Image Processing (CIP) and

M. Emre Celebi Fundamentals of Color Image Processing

Scope of This Tutorial

This tutorial is about Color Image Processing (CIP) and

M. Emre Celebi Fundamentals of Color Image Processing

Scope of This Tutorial

This tutorial is about Color Image Processing (CIP) and

M. Emre Celebi Fundamentals of Color Image Processing

Scope of This Tutorial

This tutorial is about Color Image Processing (CIP) and

M. Emre Celebi Fundamentals of Color Image Processing

Scope of This Tutorial

This tutorial is about Color Image Processing (CIP) and

M. Emre Celebi Fundamentals of Color Image Processing