Introduction of Saliency

Map
Presenter: Chien-Chi Chen
Advisor: Jian-Jiun Ding

Outline
Introduction of saliency map
Button-up approach

L. Ittis approach
Frequency-tuned
Multi-scale contrast
Depth of field
Spectral Residual approach
Global contrast based

Top-down approach
Context-aware

Information maximum
Measuring visual saliency by site entropy rate
2

Outline
Introduction of saliency map
Button-up approach

L. Ittis approach
Frequency-tuned
Multi-scale contrast
Depth of field
Spectral Residual approach
Global contrast based

Top-down approach
Context-aware

Information maximum
Measuring visual saliency by site entropy rate
3

Introduction of saliency map

Low-level(contrast)
Color
Importa
Orientation nt!
Size
Motion
Depth

High-level
People
Context

Lowlevel

With
face
detecti
on

Judd et
al,

Outline
Introduction of saliency map
Button-up approach

L. Ittis approach
Frequency-tuned
Multi-scale contrast
Depth of field
Spectral Residual approach
Global contrast based

Top-down approach
Context-aware

Information maximum
Measuring visual saliency by site entropy rate
5

L. Ittis approach

Architecture:
R,G,B,
Y

Gaussian
Pyramids
Gabor
pyramids
for = {0,
45, 90,
135}

L. Ittis approach
Center-surround Difference
Achieve center-surround difference through across-scale
difference

Operated denoted by Interpolation to finer scale and pointto-point subtraction

One pyramid for each channel: I(), R(), G(), B(), Y()
where [0..8] is the scale

L. Ittis approach
Center-surround Difference
Intensity Feature Maps

I(c, s) = | I(c) I(s)|

c {2, 3, 4}
s = c + where
So I(2, 5) = | I(2)
I(2, 6) = | I(2)
I(3, 6) = | I(3)

6 Feature Maps

{3, 4}
I(5)|
I(6)|
I(6)|

L. Ittis approach

Center-surround Difference

Center-surround Difference

Orientation Feature Maps

Color Feature Maps

Red-Green and Yellow-Blue

+B-Y
+R-G
+G-R

+G-R
+R-G

+B-Y
+Y-B

O (c, s , ) O ( c, ) O ( s , )
Same c and s as with
intensity
+Y-B
+B-Y

RG(c, s) = | (R(c) - G(c)) (G(s) - R(s)) |

BY(c, s) = | (B(c) - Y(c)) (Y(s) - B(s)) |

L. Ittis approach

Normalization Operator
Promotes maps with few strong
peaks
Surpresses maps with many
comparable peaks
1.
2.
3.
4.

Normalization of map to range [0M]

Compute average m of all local maxima
Find the global maximum M
Multiply the map by (M m)2

L. Ittis approach
Example of
Operation:

Inhibition of retu

Frequency-tuned
Image Average

L

I a

b

S ( x, y ) I Ihc ( x, y )

Gaussian blur

L
hc
Ihc ( x, y ) ahc

bhc

12

Multi-scale contrast
Saliency algorithm
Multiscale
contrast

Imag
e

Centersurround
histogra
m
Color
spatialdistributi
on

Conditiona
l
Random
Field

Salienc
y map

13

Multi-scale contrast
Multi-scale contrast
Local summation of
laplacian pyramid
L
l
l
2
f c ( x, I )
|| I ( x ) I ( x) ||
l 1 x N ( x )

Center-surround histogram

Distance between
histograms of RGB color:
1 ( Ri Rsi )2
( R, Rs ) i
2 ( R Rsi )
2

R* ( x) arg max 2 ( R( x), Rs ( x))

R ( x)

f h ( x, I )

xx

{ x| xR* ( x )}

( R* ( x), Rs* ( x))

14

Multi-scale contrast
Color spatial-distribution
Image(R
GB)

GMM

The
variance of
Coordinate
of pixel x
and y
Distance
from pixel x
to image
center

f s ( x, I ) p (c | I x )(1 V (c)) (1 D(c))

c

15

Multi-scale contrast
Energy term:
K
E ( A | I ) k Fk (a x , I ) S (a x , a x , I )
x k 1
x, x

Saliency object: Pairwise

feature:
f k ( x, I ), a x 0
Fk (a x , I )
1 f k ( x, I ), a x 1

S (ax , ax , I ) | ax ax | exp( d x, x )
d x, x || I x I x ||, L 2norm

(2 || I x I x ||2 ) 1
16

Multi-scale contrast
CRF:
1
P ( A | I ) exp( E ( A | I ))
Z

r*
n n r
arg max
r log P ( A | I ; )

The derivative of the log-likelihood

k to
with respect
17

Depth of field
As the spread of single lens reflex
camera, more and more low depth of
field(DOF) images are captured.
However, current saliency detection
methods work poorly for the low DOF
images.

18

Depth of field
Algorithm:

19

Depth of field
Classification:

Focal Point:

In a low

DOF image
DO
G

S (i,

Rectangle with the

highest edge density,
and center is initial
d
focal point

j ) S (i, j ) Ae 2

Composition
Analysis:
segmentati
Regio
on

Sr Sr e

Ar
n
d

Ai1 m 2 3
20

Spectral Residual Approach

First scaling image to 64x64.
Then we smoothed the saliency map
with a gaussianfilter g(x) ( = 8).

21

Global contrast-based
Histogram based contrast(Lab):
O( N 2 )

O( N ) O(n 2 )
Quantizati
on of Lab
Each channel to
have 12 different
value

123 1728

22

Global contrast-based
Region based contrast
Segment the Image
[Efficient graph-based image
segmentation]

23

Outline
Introduction of saliency map
Button-up approach

L. Ittis approach
Frequency-tuned
Center-surround
Depth of field
Spectral Residual approach
Global contrast based

Top-down approach
Context-aware

Information maximum
Measuring visual saliency by site entropy rate
24

Context-Aware
Goal:

Convey the image content

Liu et al, 2007

25

Context-Aware
Distance between a pair of patches:

d ( pi , p j )

d color ( pi , p j )
1 c d position ( pi , p j )
High

salient

Context-Aware
Distance between a pair of patches:

K
1
r
r
r
Si 1 exp d ( p i , q j )
K k 1

q kr K most similar patches at

scale r

High for K most

similar

Saliency

Context-Aware
Salient at:
Multiple scales foreground
Few scales background

1
Si
M

Scale 1

Scale 4

rM

S
r r1

r
i

Context-Aware
Foci = S 0.8
i

Include distance map

1 d foci (i )

Si Si 1 d foci (i )

Si
X

Outline
Introduction of saliency map
Button-up approach

L. Ittis approach
Frequency-tuned
Center-surround
Depth of field
Spectral Residual approach
Global contrast based

Top-down approach
Context-aware

Information maximum
Measuring visual saliency by site entropy rate
30

Measuring visual saliency by

site entropy rate
1

31

Measuring visual saliency by

site entropy rate
2

A fully-connected graph representation is adopted

for each

32

Sub-band graph
representation

33

Sub-band graph
representation

34

Measuring visual saliency by

site entropy rate
3

A random walk is adopted on each sub-band graph. And

Site entropy rate(SER) is measured the average information
from a node to the other

35

The site entropy rate

Wi
,Wi ij , W ij
2W
j
i , j: j i

Pij

ij
ij
j

36

Conclusion
Image processing is funny
Unusual in its neighborhood will
correspond to high saliency weight
Contrast is the key of saliency

37

Reference
[1] R. Achanta, F. Estrada, P. Wils, and S. Susstrunk. Salient region
detection and segmentation. In ICVS, pages 6675. Springer, 2008.
410, 412, 414
[2] R. Achanta, S. Hemami, F. Estrada, and S. Susstrunk. Frequencytuned salient region detection. In CVPR, pages 15971604, 2009. 409,
410, 412, 413, 414, 415
[3] L. Itti, C. Koch, and E. Niebur. A model of saliency based visual
attention for rapid scene analysis. IEEE TPAMI, 20(11):12541259,
1998. 409, 410, 412, 414
[4] X. Hou and L. Zhang. Saliency detection: A spectral residual
approach. In CVPR, pages 18, 2007. 410, 412, 413, 414
[5] S. Goferman, L. Zelnik-Manor, and A. Tal. Context-aware saliency
detection. In CVPR, 2010. 410, 412, 413, 414, 415
[6] MM Cheng, GX Zhang, N. J. Mitra, X. Huang, S.M. Hu. Global Contrast
based Salient Region Detect. In CVPR, 2011 .
[7] T. Liu, Z. Yuan, J. Sun, J.Wang, N. Zheng, T. X., and S. H.Y. Learning to
detect a salient object. IEEE TPAMI, 33(2):353367, 2011. 410
[8] W. Wang, Y. Wang, Q. Huang, W. Gao, Measuring Visaul Saliency by
Site Entropy Rate, In CVPR, 2010.
38