Content aware image resize

Johan Jansen
September 15, 2014
Abstract
Reducing the size of images often happens at
the cost of distortion or loss of aspect ratio.
Content-aware image resizing is methodology
that tries to intelligently analyze the image to
detect important features and minimize the
distortion to these parts when resizing the im-
age. With todays varying displays and reso-
lutions from cell phones, desktop computers,
tablet computers, etc. resizing and scaling
images is often a necessity. This article ex-
plores how content-aware resizing can be im-
plemented as an useful alternative to tradi-
tional image scaling or cropping.
1 Introduction
This article examines the implementation of
a content aware algorithm[1], and algorithm
that resizes images based on its contents.
The naive way of resizing images without con-
tent awarness, is usually simply scaling all
pixels with a Linear, Bicubic or similar lter.
This often results in images that are skewed
or loses its aspect ratio. This is especially
true when only either the height or width
is resized. My content aware algorithm im-
plementation will focus on reducing size of
images and minimizing the loss of important
information and with as little damage to the
important content as possible. When reduc-
ing the size of images information is always
lost, but with the help of the algorithm we
will determine which information we can re-
move without damaging the important con-
tent in the image.
A content-aware resize algorithm is usually
implemented in three steps:
1. Calculate the energy of every pixel in the
image
2. Find the path from one side of the image
to the other at minimal cost of energy
(this path is often called a seam)
3. Remove the seam and repeat this process
until image has reached the desired size
Figure 1 shows my seam carver in action
where the red pixels indicates the seam being
removed in this iteration.
1
Figure 1: Seam carving in action
2 Methodology
We need a way to measure how good my al-
gorithm resizes images. We are going to take
a look at two variables, image distortion and
performance. A good image resize algorithm
can quickly resize images with minimal dis-
tortion of the image contents or loss of im-
portant data. Image distortion can be dif-
cult to measure without human evaluation,
but performance of each method can be easily
measured.
There are two dierent modules in my con-
tent aware algorithm that we will experiment
with:
1. A function that calculates the energy of
each pixel
2. A function that connects a path from one
side of the image to the other with min-
imal cost
We will calculate the energy of the image us-
ing two dierent lters. These are the Sobel
Filter[3] and a Pixel Dierence lter. We will
compare the result of resizing an image us-
ing both energy functions and compare the
performance of using either lter. For the
path connection function we take a look at
Dynamic Programming and pathnding us-
ing Dijkstras Algorithm[2]. Both path con-
nectors should nd the optimal path so for
pathnding the only dierence we are taking
a note of is performance.
We are interested in performance since for
content aware resizing to be useful, it should
preferably be able to be done in real time. My
implementation of the algorithm is mainly to
show how the algorithm works and is not op-
timized for performance. We can however
compare the performance of the dierent l-
ters and pathnders that are implemented in
the algorithm.
Figure 2 shows the image we are going to
use for the testing. The original image has a
resolution of 315x420 pixels and in each of
my tests we are going to resize the image
to 215x220 using linear scaling and content
aware resize with both sobel and pixel dier-
ence gradient lters.
3 Results
First we are going to look at the visual re-
sults of resizing the images. Figure 3 shows
the image resized using a linear lter using
nearest neighbour. We can see that the image
has become smaller and all details in the im-
ages have become smaller as well. Now when
we use content aware ltering with the sobel
lter (shown in gure 4) we see the image re-
2
Figure 2: Original 315x420 image
duced to the same size but with much more
detail. All the details in the original image
have been kept and only unimportant infor-
mation such as the sky and non-dening parts
of the beach have been removed. Some details
have been even become easier to spot such as
the tower in the background or the waves in
the water. The image has been successfully
reduced in size without distorting anything
or by simply making everything smaller.
Next we are going to compare an image re-
sized using content-aware resizing using two
dierent lters. These are the Pixel Dier-
ence gradient lter (gure 5) and the Sobel
lter we rst compared with linear lter (g-
ure 5). The result of a content-aware resized
image using Pixel Dierence is shown in g-
ure 5. A rst look at the images might not
show much dierence. But if you take a closer
look at the Sobel ltered image you might
notice some artifacts where seams have been
removed from top to bottom. This is easi-
est to see on the rock on the bottom right
going up to the hill on the top of the im-
3
Figure 3: Resized using Linear lter
Figure 4: Resized with content-awarness us-
ing Sobel edge detection
Figure 5: Resized with content-awareness us-
ing Pixel Dierence gradient
age. Figure 6 enhances these artifacts with
the red squares. Note that in the Pixel Dif-
ference lter we have lost some details such
as the roundness of the rocks in the water
which are kept in the Sobel version. Luckly
this does not cause any problems for the con-
tents in this specic image. It seems for the
test image, the Nearest Pixel Dierence lter
produces better results than using Sobel Edge
Detection when reducing the image size.
We will also look on another image which
is dicult for my content-aware algorithm to
resize (shown in gure 7). Both my energy
functions detect the persons on the beach as
low energy while the waves in the sea are cal-
culated as high energy, causing the persons
to be removed and not the surroundings as
one might expect (shown in gure 8).
4
Figure 7: Original 405x800 image which my algorithm fails to reduce in size without distor-
tion (not to scale). Image source: [6]
Figure 8: The image in gure 7 after content-aware resizing (not to scale)
5
Figure 6: The artifacts in the image resized
using Sobel Edge Detection
Finally we are going to look at the perfor-
mance of each energy function and pathnd-
ing algorithm. Note that we do not compare
the results to Linear lter resizing. The rea-
son for this is as opposed to my content-aware
resize algorithm, Linear ltering is hardware
accelerated and optimized for speed. I have
also used two dierent images with dierent
sizes to see if any of the energy functions
or pathnders perform better with larger or
smaller data sets. The small image sized
315x420 is the one shown in gure 2, while
the 542x800 image is shown in gure 9.
The results of the performance tests in ta-
ble 1 and table 2 are unanimous and show
clearly that Nearest Pixel Gradient with Dy-
namic Programming resizes the image fastes
in both the small and large image. This is
not proof that this is always the case, but is
only true in my implementation of the energy
1
Photo by Newton2, available from Wikimedia
Commons under the CCA 2.5 Generic license.
Figure 9: Original 542x800 image
1
(not to
scale)
functions and pathnders. The time used to
resize a single image even with the fastest en-
ergy function and pathnder is still impracti-
cal (about 8 seconds) in real-time. This can
however be mitigated with optimization.
4 Futher Work
Optimizing the content-aware resize algo-
rithm to become real-time is an important
feature [4]. Currently my algorithm re-
calculates the energy image every iteration
which costs a lot of computing power. The
pathnder algorithms can be parallelized as
well, making it execute more eciently on
modern multi-core computers. Exploring ad-
ditional energy functions such as Canny Edge
Detection or Prewitt Operator is interesting
because each can produce a dierent resized
image and some may work better for specic
images than others.
We have only touched the surface of what
6
Table 1: Performance for resizing from 315x420 to 215x220
Energy Function Pathnding Performance
Sobel Filter Dynamic Programming 10.02 seconds
Nearest Pixel Dynamic Programming 8.21 seconds
Sobel Filter Dijkstras Algorithm 12.17 seconds
Nearest Pixel Dijkstras Algorithm 10.80 seconds
Table 2: Performance for resizing from 542x800 to 242x500
Energy Function Pathnding Performance
Sobel Dynamic Programming 61.64 seconds
Nearest Pixel Dynamic Programming 35.85 seconds
Sobel Dijkstras Algorithm 88.40 seconds
Nearest Pixel Dijkstras Algorithm 64.25 seconds
content-aware algorithms are capable of [1].
We can specify and area in the image to be
innite energy so that when resized, the algo-
rithm will not touch that part of the image.
One could for example mark a person in such
an image and resize all the surroundings with-
out aecting the person. This also works in
the opposite direction where we can mark an
are as low energy cost, meaning this section
will be removed with minimal distortion or
damage to the surroundings.
Furthermore, we have only looked at reduc-
ing the size of images, but how can content-
aware resize algorithms be used to increase
the size of images without distorting impor-
tant parts of the image? These are questions
that are out of the scope of this article, but
shows that content-aware resizing can be a
useful tool and becomes more and more ap-
plicable as the computing power of hardware
increases.
5 Conclusion
We have seen that content-aware image resiz-
ing can reduce the size of an image while not
distorting the contents of the image. This
allows us to see the image in a smaller size
but still grasp same the content with same
proportions as if were full sized. This thec-
nique is not applicable to all images how-
ever. Luckly we have dierent energy func-
tions where some might work better for cer-
tain types of images than others. It is dif-
cult to determine which energy function is
best for what type of image without human
interaction or evaluation.
Determining what energy function to use
for a specic image is a non-trivial matter and
might not be determinable without compar-
ing the results of resizes using dierent energy
functions. We have seen that some energy
functions can cause artifacts in the image by
inadvertently creating new high-energy seam
7
when removing a low-energy seam.
While being a useful tool, the content-
aware algorithm may need human input to
resize images correctly. The computation
power required to resize an image using a
content-aware algorithm increases exponen-
tionally with the image size. Traditional im-
age scaling often uses hardware acceleration
to mitigate this problem and this could also
be applicable to content-aware algorithms
which also are parallelizable[5].
References
[1] Shai Avidan, Ariel Shamir. Seam Carving
for Content-Aware Image Resizing. ACM
SIGGRAPH. 2007.
[2] Moshe Sniedovich. Dijkstras Algorithm
revisited: the OR/MS Connexion. The
University of Melbourne Parkville, VIC
3052, Australia.
[3] R. Fisher, S. Perkins, A. Walker and E.
Wolfart. Sobel Edge Detector. University
of Edinburgh, Edinburgh, UK. 2003.
[4] Hua Huang, TianNan Fu, Paul L. Ros-
ing and Chun Qi. Real-time content-aware
image resizing. Science in China Press.
2009.
[5] Jacob Stultz. Seam Carving: Paralleliz-
ing a novel new image resizing algorithm.
Massachusetts Institute of Technology.
2008.
[6] Photo is taken from MacWorld Ar-
ticle about Content Aware Resize
in Adobe Photoshop CS4. Link:
http://www.macworld.com/article/1135681/
photoshopcs4.html
8