International Journal of Computer Information Systems, Vol. 3, No. 5, 2011
Jenita Subash ^{1} ,Sasithradevi ^{2} ,Girish H ^{3}
^{1} Lecturer, Cambridge Institute of technology, Bangalore. ^{2} Lecturer, V.V College of Engineering, V V Nagar, Tisaiyanvilai ^{3} Assistant Professor, Cambridge Institute of technology, Bangalore.
Abstract— One of the key processes in digital camera pipeline is demosaicing or color filter array (CFA) interpolation. A way to reconstruct a full threecolor of color images by estimating the missing pixel components in each color plane is called a demosaicking algorithm. The purposes of this paper are twofold. First, a rigorous treatment of a classical demosaicking algorithm based on alternating projections (AP) is presented. Despite its good performances, a relative weakness of the AP algorithm is its high computational complexity. The precise analysis of the convergence property of the AP algorithm reveals that it is a contraction mapping which converges to a unique fixed point. Second, this theoretical analysis leads to the construction of a fast noniterative implementation of the AP algorithm. The quality of the reconstructed images is calculated by using two performance parameters PSNR and MSE values. Using multirate signal processing technique the results of the AP algorithm can be obtained in a single step, when implemented as linear filtering in polyphase domain. Numerical experiments show that the proposed onestep solution leads to substantial computational savings, by about an order of magnitude.
exploit the correlation between channels, they tend to cause large interpolation errors in the red and blue channels when green value abruptly changes. Directional filtering is the most popular approach to color demosaicking that produces competitive results in the literature. The best known directional interpolation scheme is, perhaps, the secondorder Laplacian filter proposed by Hamilton and Adams [7], [8], [9].
Fig. 1. Bayer pattern.
Keywords— Demosaicing, color filter array,
fixed point,
contraction mapping, alternating projections, polyphase representation
I.
INTRODUCTION
Most digital cameras capture an image with single sensor array. At each pixel, only one of the three primary colors is sampled. Fig.1 shows the commonly used Bayer color filter array (CFA) [1]. In order to reconstruct a full color image, the missing color samples need to be interpolated by a process called color demosaicking. The quality of reconstructed color images depends upon the image contents and the employed demosaicking algorithms [2].
An immense number of demosaicking methods have been proposed in the literature. The early demosaicking methods include nearest neighbor replication, bilinear interpolation and cubic spline interpolation [2], [3], [4]. These methods can be simply implemented, but they suffer from many artifacts such as blocking, blurring and zipper effect at edges. Under the assumptions that images have slowly varying hue, the smooth hue transition methods [4], [5], [6] interpolate the luminance and chrominance channels differently. After recovering the green channel by bilinear interpolation, the red and blue channels are recovered by bilinearly interpolating the red hue and blue hue. Although the smooth hue transition methods
They used the second order gradients of blue and red channels as the correction terms to interpolate the green channel. The smaller of the two second order gradients in the horizontal and vertical directions is added to the average of green samples along the chosen direction. Once the green samples are filled, the red and blue samples are interpolated similarly with the correction of the second order gradients of the green channel. Chang et al. [10] proposed a more complicated gradientbased demosaicking scheme. They computed a set of gradients in different directions in the 55 neighborhood centered at the pixel to be interpolated. A subset of these gradients is selected by adaptive thesholding. At last, the missing samples are estimated from the known samples located along the selected gradients. Recently, Ramnath and Snyder [11] proposed a bilateral filtering based scheme to denoise, sharpen and demosaick the image simultaneously. Another class of demosaicking technique is iterative schemes, which can also be combined with gradientbased methods. Kimmel developed a two step iterative demosaicking process consisting of reconstruction step and an enhancement step [12]. He calculated eight directional derivatives at each pixel based on its eight neighbors. Based on these edge indicators, the hue values are computed and the missing green, red and blue samples are corrected iteratively by the ratio rule.
November Issue
Page 38 of 59
ISSN 2229 5208
In this paper one well known influential demosaicking algorithm which is based upon the concept of alternating projections [13] is analyzed theoretically. Despite impressive performances, a limitation of the AP algorithm is its high computational complexity: The full resolution color images are reconstructed in an iterative fashion, with each iteration involving a 2D nonsubsampled subband decomposition and reconstruction of an entire image. The algorithm defines two constraint sets based on the observed color samples and prior knowledge about the correlation between the channels. It reconstructs the color channels by alternately projecting the initial estimates into these constraint sets.
The rest of the manuscript is organized as follows. A brief discussion of the AP algorithm in Section II sets the ground for all subsequent discussions. Section III provides a compact representation of the AP algorithm in polyphase domain and a precise analysis of its convergence based on the contraction mapping. Section IV presents a fast one step implementation of AP algorithm. Section V concludes the paper
Throughout
the
paper
the
grayscale
images
is
regarded, with n ≝ (n1, n2) , as vectors in linear space of
squaresummable x[n] is defined as,
sequences. The 2D Ztransform of
2
where
≝
. Specializing to the unit sphere, we
have
, hence
represents the
Fourier transform of x[n] .
II. BACKGROUND AND PROBLEM STATEMENT
In this section the original AP demosaicking algorithm is briefly discussed and the key problems that are to be addressed in this work are stated explicitly.
A. Iterative Demosaicking Algorithm Let s[n] denote the raw sensor image obtained from the
CFA, and let r[n], g[n], b[n] represent the fullresolution red,
International Journal of Computer Information Systems, Vol. 3, No. 5, 2011
pixels. For simplicity, the case of red pixels is considered throughout the paper. The processing steps of blue pixels can be inferred by symmetry. The first constraint set introduced by the AP algorithm exploits the correlation between the detail subbands of the green and red channels. Consider a generic 2D filter bank
which decomposes an input image x[n] into one lowpass
subband
and N different highpass subbands
.
The
set
of
analysis
filters
(z)
and
synthesis filters condition,
(z) satisfies the perfect reconstruction
(1)
One concrete example of such filter bank is the 2D undecimated wavelet transform used in [13]. With one level decomposition, we obtain one lowpass channel and three highpass channels. Meanwhile the channel filters are all separable products of 1D filters. Based on the observation that the highpass subbands of the green and red channels of a natural image tend to be very similar, the AP algorithm proposes to search for fullresolution red image within the following detail constraint set,
(2)
where
[n] is the i ^{t}^{h} highpass analysis filter, _{g}_{[}_{n}_{]} is the
previously estimated fullresolution green channel, and T is a threshold indicating how similar the two signals should be. To enforce the constraint defined in (2), the AP algorithm employs an update operator, defined in the transform domain as,
( 
z 
) 
X 
( 
z 
)) 

N S 
i 
( 
z 
)( 
Ai 
( 
)z G 
( 
z 
)) 

i 
1 
(3)
The above formula (3) can be simplified as follows. Let
green, and blue color channels, respectively. The goal of the Simplifying (3) we get, 
S 
0 
( 
z 
) 
A0 
( 
z 
) 
(4) 

AP algorithm starts by obtaining an estimate of the full resolution green channel, which is then used in the subsequent estimation of the missing red and blue pixels. This two step strategy is justified by the fact that the green channel in the Y(z) H(z)X (z) (1 H(z))G(z)(5) Compared with the original definition (3) for P _{d} , the proposed formula in (16) only requires the filtering operation of 

Bayer CFA has twice the sampling density as that of red or blue channels (Fig. 1), and, hence, is relatively easier to reconstruct. H(z) ≝ X (z) and _{G}_{(}_{z}_{)}Consequently the 
in the lowpass channel of the filter bank. 

computations 
previously 
needed 
in 

The ultimate goal of this section is to focus on how the AP algorithm iteratively interpolates the missing red and blue November Issue Page 39 of 59 
ISSN 2229 5208 
obtaining the three highpass channels of _{G}_{(}_{z}_{)} can be eliminated. The second constraint set that the AP algorithm employs based on the available sensor measurements s[n] . Most specifically, the interpolated red channel should belong to the
following ―observation‖ constraint set,
where r ≝
_{(}_{6}_{)}
represents the locations of the red pixels in the Bayer CFA as shown in Fig. 1. To enforce the above constraint, we can
define the observation update operator _{P}_{o} as follows
(7)
In words, the operator _{P}_{o} replaces the value of x[n] at the red pixel locations with the available sensor measurements, but leaves the rest of the pixels intact.
B. Problem Statement
We first recall the following facts about projection onto convex sets (POCS).
International Journal of Computer Information Systems, Vol. 3, No. 5, 2011
where
is the filter defined in (4), and 1 _{D} () is the
indicator function defined on the frequency domain support D
of
.
Remark 1. Proposition 1 states that, for P _{d} to be a projection
operator, the lowpass filter
must be ideal in the
frequency domain. Consequently, h[n] has to be a sinclike
filter in the spatial domain, and cannot have a finite impulse response (FIR).
Example 1. The separable filters used in AP algorithm are
a
0
[
n
]
1
16
[1 2 1]
[1 2 1]
T
,
and
s
o
[
n
]
1
64
[1 2 6
2 1]
[
1 2 6
2 1]
T
.
Algorithm 1 Interpolate Missing Red/Blue Pixels:
Input: The raw CFA image
and the estimated green
channel _{g} . Output: An estimated fullresolution red channel.
Definition 1: Let C be a closed convex set in Hilbert
space H , for any x such that
H there exists a unique element y _{x}
C
 x  y _{x} 
 x – z , for all z
C
(8)
The mapping P _{c} onto C.
: H → H, P _{c} x
y _{x} is the projection operator
Theorem 1: Let C _{1} and C _{2} be the closed convex sets in a
Hilbert space H, and
and
the corresponding projection
operators. Suppose that C _{1}
. For any x ^{(}^{0}^{)} H, the
sequence
_{.} converges
to
the projection of
onto C _{1}
C _{2} , i.e. ,


= 0
(9)
To invoke the above theorem in the context of the AP algorithm, one can easily verify that the two constraint sets C _{o} and C _{d} defined in Section II(A) are closed and convex. Meanwhile the observation projection operator P _{o} in (7) is indeed the projection onto C _{o} . However, this is in general not the case for the detail update operator P _{d} .
Proposition 1.
is a projection operator if and only if
= 1 _{D} (),
(10)
Begin
Obtain r ^{(}^{0}^{)} , b ^{(}^{0}^{)} , g ^{(}^{0}^{)} using edge directed interpolation. Initialize: i
IF
Then
Enforce the detail constraint: r ^{(}^{i}^{+}^{0}^{.}^{5}^{)} = P _{d} r ^{(}^{i}^{)} Enforce the observation constraint: r ^{(}^{i}^{+}^{1}^{)} = P _{o} r ^{(}^{i}^{+}^{0}^{.}^{5}^{)} i = i+1
Else
return r ^{(}^{i}^{)}
End
In
Fig.
2.
the
magnitude
frequency
response
≝
is shown. Evidently,
since
both a0 [n] and s0[n] are FIR filters, 

is non ideal
and does not satisfy (10), and thus P _{d} is not a projection operator. Nevertheless, we know from numerical experiments that the AP algorithm based on this filter still converges after a small number of iterations.
Example 2. Another interesting fact about the AP algorithm is that the final result of the iteration process appears to be independent of the starting point. To demonstrate this phenomenon, we apply the AP algorithm to the standard test
November Issue
Page 40 of 59
ISSN 2229 5208
MSE
image ―light house‖ from the Kodak set, with three different
interpolation as the starting point. The results are shown in Fig. 3. It is seen that the different choices of the initial estimates only affect the speed of convergence, but all three processes eventually converge to the same result r [n] .
.
III. CONVERGENCE OF AP ALGORITHM
In this section, a rigorous analysis of the convergence property of the AP algorithm is provided. The theoretical analysis will be heavily based on the polyphase representation [14], [15] of signals. To facilitate readers who are not familiar with this concept, we start our discussion by having a brief overview of this useful signal representation.
A. Polyphase Representation
The polyphase representation (transform) is a powerful tool in multirate signal processing. It also plays a critical role in the derivations presented in later parts of this paper. For a 2D
image x[n], we can decompose it into four polyphase
International Journal of Computer Information Systems, Vol. 3, No. 5, 2011 The polyphase components in (12) are simply downsampled versions of the original signal x[n] . The sampling locations of
all four polyphase components form a partition. Note that the mapping between the signal x[n] and its polyphase components is onetoone. To go back to the original signal from its polyphase components, we can easily verify that
(13)
The polyphase representation becomes especially handy when we try to describe the periodic sampling structures of color filter arrays. To describe the filtering operation in the polyphase domain, consider two signals x[n], y[n] such that
(14)
5
10
0
10
5
10
10
10
0
5
10
15
20
25
30
35
40
iteration number
Fig. 3. Convergence of the AP algorithm for different choices of initial estimates.
for some filter h[n] .Denote by
and
(15)
the vector of all four polyphase components of x[n] , and the
corresponding vector in the transform domain, respectively. Similarly, we can define y _{p} [n] andYp(z)) for y[n] ] as well.
The filtering operation (14) can be described in the polyphase domain as
(12)
(16)
At this point, it may seem that going to the polyphase domain only makes the filtering operation more complicated. As we shall see below however, the primary advantage in using the
November Issue
Page 41 of 59
ISSN 2229 5208
polyphase notation is that it can convert the shiftvariant operator P _{o} P _{d} used in the AP algorithm to a multipleinput multipleoutput system of shiftinvariant (filtering) operations. The added complexity of dealing with matrixvector multiplications are more than compensated for by the convenience of working with shiftinvariant operators.
Definition 2. A mapping T from a Hilbert space H to itself is called Lipschitz continuous, if there is some real number α > 0 such that
(17)
The smallest such value of α, denoted by
α _{T}
,
is called the
Lipschitz constant of T . Furthermore, if α _{T} < 1, then T is a
contraction mapping.
Theorem 3. [16] Let T : H H be a contraction mapping with Lipschitz constant 0 < α _{T} < 1.
1. The mapping T admits one and only one fixed point .
2. For arbitrary x ^{(}^{0}^{)} H, the iterated sequence x ^{(}^{k}^{+}^{1}^{)} = T (x ^{(}^{k}^{)} )
always converges to x .
3. The speed of convergence is bounded by the following
inequality
 x ^{(}^{k}^{)}  x  (α _{T} ) ^{k} / 1 α _{T}  x ^{(}^{k}^{)}  x .
Now we just need to check that the mapping T is indeed a contraction. If that is the case, then the convergence of the AP algorithm will be automatically guaranteed by the contraction mapping theorem stated above. The following proposition presents a simple way to obtaining the Lipschitz constant of T.
Proposition 2. The Lipschitz constant of the mapping T can be calculated as α _{T} =max( _{m}_{a}_{x} (T(e ^{j}^{} ))),
International Journal of Computer Information Systems, Vol. 3, No. 5, 2011
where T (z) is a matrix obtained from H(z) after removing its 2 ^{n}^{d} row and 2 ^{n}^{d} column, and
TABLE I. PERFORMANCE COMPARISON OF ORIGINAL AP ALGORITHM AND THE PROPOSED NONITERATIVE SCHEME
PSNR(dB) 

Red 
Green 
Blue 

AP(15 
38.47 


iterations) 

AP(5 
38.44 


iterations) 

Proposed 
38.43 


scheme 
where σ _{m}_{a}_{x} (·) denotes the largest singular value of a matrix. So far, we have used the polyphase representation and the contraction mapping theorem for the rigorous explanation of the convergent property, and the corresponding condition for the filter h[n] used in the iteration process. The convergence value of the iteration process appears to be unique and does not depend on the starting points. The following section shows that we can directly reach the convergence result of the AP algorithm without going through iterations
IV. NONITERATIVE IMPLEMENTATION OF THE AP ALGORITHM
A. 
General Schemes 
We know from Theorem 3 that, if AP algorithm is convergent, then it must converge to the unique fixed point of the mapping 

T. 
Let _{C}_{ˆ} _{v} _{(}_{z}_{)} denote the transform domain representation of the fixed point. Applying the definition of the mapping T we obtain,
Fig. 4. The block diagram of the proposed noniterative implementation of the AP algorithm.
The above equality implies
that
(18)
Consequently (18) means
In words, the final convergence result of the AP algorithm can be directly obtained by three filtering operations in the
November Issue
Page 42 of 59
ISSN 2229 5208
polyphase domain. Meanwhile, the remaining polyphase
component, C01(z) , is equal to
.The block
diagram of the proposed Noniterative implementation of the AP algorithm is shown in Fig. 4.
B. Numerical Experiments
To demonstrate the performance of the proposed scheme, both the proposed Noniterative algorithm and the original AP algorithm are applied to the standard Kodak test images. For the proposed algorithm, the three polyphase filters F _{0}_{0} (z), F _{1}_{0} (z), F _{1}_{1} (z) defined in (18) are generally not FIR. However these filters can be wellapproximated by their finitely truncated versions. For AP algorithm, we test two different options for maximum number of iterations 15 and 5. The former will ensure us to obtain the convergence value of the
International Journal of Computer Information Systems, Vol. 3, No. 5, 2011
V.
CONCLUSIONS
This paper presents a detail treatment of a classical color image demosaicking algorithm based on alternating projections. On the theoretical side, this analysis provides a rigorous foundation for the convergence of the AP algorithm, based on contraction mapping theorem. On practical side, an efficient Noniterative implementation of AP algorithm in polyphase domain is proposed. Numerical experiments confirm that the proposed algorithm can achieve the same results obtained by AP algorithm at convergence, but significantly faster than latter.
ACKNOWLEDGMENT
TABLE II. PERFORMANCE COMPARISON OF ORIGINAL AP ALGORITHM AND THE PROPOSED NONITERATIVE SCHEME
Time(second) 

Initialization 
Refinement 
Total 

stage 
stage 

AP(15 

2.526 

iterations) 

AP(5 

6.656 

iterations) 

Proposed 

0.286 

scheme 
procedure while the later is commonly used in practice due to its reduced computational load. Both the algorithms are implemented in Matlab. Table I and II summarizes the main results, averaged over 24 test images. The accuracy of the algorithm is measured in terms of peak signal to noise ratio (PSNR) of the demosaicked image. Note that the PSNR values for green channel are same for different algorithms, since both AP and the proposed scheme follow the same steps in estimating the green pixels. For the red and blue pixels, the PSNR values obtained by the proposed scheme are very close to those reached by the AP after 15 iterations. This indicates that this Noniterative algorithm can accurately compute the final convergence values of the AP procedure, even though the polyphase filters used in this implementation are truncated approximation. The main advantage of the proposed algorithm is its computational efficiency. As shown in table II, the total time is divided into two parts: initialization stage(estimating the missing green pixels), refinement stage (estimating the missing red or blue pixels). More substantial improvements can be found in refinement stage: the original AP algorithm approaches the reconstruction problem via iteration, whereas the proposed scheme achieves the same goal by direct filtering operations in polyphase domain. Correspondingly the running time can be dramatically reduced.
First and foremost we thank Almighty God for his grace to complete this work. We would like to thank Cambridge Institute of technology for providing permission and encouragement throughout this work.
REFERENCES
[1]
B. E. Bayer, ―Color imaging array,‖ U.S. Patent 3 971 065,
1975.Eastman Kodak Company. [2] P. Longère, X. Zhang, P. B. Delahunt, and D. H. Brainard,
―Perceptualassessment of demosaicing algorithm performance,‖ Proc.
IEEE, vol.90, pp. 123–132, 2002. [3] H. S. Hou et al., ―Cubic splines for image interpolation and digital filtering,‖ IEEE Trans. Acoust., Speech, Signal Process,, vol. ASSP26, no.3, pp. 508–517, Jun. 1987. [4] J. E. Adams, ―Intersections between color plane interpolation and other image processing functions in electronic photography,‖ Proc. SPIE, vol.2416, pp. 144–151, 1995. [5] D. R. Cok, ―Signal Processing method and apparatus for producing interpolated chrominance values in a sampled color image signal,‖ U.S.
[6] 
Patent 4 6 42 678, 1987. Eastman Kodak Company. J. A. Weldy, ―Optimized design for a singlesensor color electronic camera system,‖ Proc. SPIE, vol. 1071, pp. 300–307, 1988. 
[7] 
J. E. Adams and J. F. Hamilton Jr., ―Adaptive color plane interpolation in single color electronic camera,‖ U.S. Patent 5 506 619, 1996. 
[8] J. E. Adams, ―Design of practical color filter array interpolation algorithms for digital cameras,‖ Proc. SPIE, vol. 3028, pp. 117–125,
[9]
1997.
J. F. Hamilton Jr. and J. E. Adams, ―Adaptive color plane
interpolationin single sensor color electronic camera,‖ U.S. Patent 5 629 734, 1997. [10] E. Chang, S. Cheung, and D. Y. Pan, ―Color filter array recovery using a
thresholdbased variable number of gradients,‖ Proc. SPIE, vol. 3650, pp. 36–43, 1999. [11] R. Ramnath and W. E. Snyder, ―Adaptive demosaicking,‖ J.Electron.Imag., vol. 12, no. 4, pp. 633–642, 2003. [12] R. Kimmel, ―Demosaicing: Image reconstruction from CCD samples,‖IEEE Trans. Image Process., vol. 8, no. 11, pp. 1221–1228, Nov. 1999. [13] B. K. Gunturk, Y. Altunbasak, and R. M. Mersereau, ―Color plane interpolation using alternating projections,‖ IEEE Trans. Image Process.,vol. 11, no. 9, pp. 997–1013, Sep. 2002. [14] Vaidyanathan, P. P., Multirate Systems and Filter Banks, PrenticeHall, Englewood Cliffs, NJ (1993). [15] Vetterli, M. and Kovacevic, J., Wavelets and Subband Coding, Prentice Hall, Englewood Cliffs, NJ (1995).
November Issue
Page 43 of 59
ISSN 2229 5208
Authors
Jenita Subash 

She did 
her B.E 
in ECE from SCAD College of 
engineering and technology in 2004, Anna University ,
Chennai. She acquired her M.E in Communication System in PET Engineering College,Anna University of Technology, Thirunelveli in 2011.
She worked as
a Lecturer
in
ECE dept.
on
PSN
Engineering college, Thidiyoor during 20052009 Currently she is working as a Lecturer in ECE Dept in
Cambridge institute of technology, K.R Puram, Bangalore
Girish H
He did his B.E in ECE from Adichunchagiri institute of Technology, chikmagalur and M. Tech in United Technologies Limited ( VTU extension Center), Yeshwanthpur, Bangalore. He is currently working as a Assistant Professor in Cambridge Institute of technology, K.R.Puram, Bangalore16.
International Journal of Computer Information Systems, Vol. 3, No. 5, 2011
November Issue
Page 44 of 59
ISSN 2229 5208