Coding Theorems for Turbo-Like Codes

Dariush Divsalar, Hui Jin, and Robert J. McEliece

Jet Propulsion Laboratory and California Institute of Technology
Pasadena, California USA
E-mail: dariush@shannon.jpl.nasa.gov, (hui, rjm)@systems.caltech.edu
Abstract.
In this paper we discuss AWGN coding theorems for ensembles of coding systems which
are built from xed convolutional codes interconnected with random interleavers. We
call these systems turbo-like codes and they include as special cases both the classical
turbo codes [1,2,3] and the serial concatentation of interleaved convolutional codes [4].
We oer a general conjecture about the behavior of the ensemble (maximum-likelihood
decoder) word error probability as the word length approches innity. We prove this
conjecture for a simple class of rate 1/q serially concatenated codes where the outer
code is a q-fold repetition code and the inner code is a rate 1 convolutional code with
transfer function 1/(1 + D). We believe this represents the rst rigorous proof of a
coding theorem for turbo-like codes.
1. Introduction.
The 1993 discovery of turbo codes by Berrou, Glavieux, and Thitimajshima [1] has
revolutionized the eld of error-correcting codes. In brief, turbo codes have enough
randomness to achieve reliable communication at data rates near capacity, yet enough
structure to allow practical encoding and decoding algorithms. This paper is an attempt
to illuminate the rst of these two attributes, i.e., the near Shannon limit capabilities
of turbo-like codes on the AWGN channel.
Our specic goal is to prove AWGN coding theorems for a class of generalized con-
catenated convolutional coding systems with interleavers, which we call turbo-like
codes. This class includes both parallel concatenated convolutional codes (classical
turbo codes) [1, 2, 3] and serial concatenated convolutional codes [4] as special cases.
Beginning with a code structure of this type, with xed component codes and inter-
connection topology, we attempt to show that as the block length approaches innity,
the ensemble (over all possible interleavers) maximum likelihood error probability ap-
proaches zero if E
b
/N
0
exceeds some threshold. Our proof technique is to derive an
explicit expression for the ensemble input-output weight enumerator (IOWE) and then
to use this expression, in combination with either the classical union bound, or the
recent improved union bound of Viterbi and Viterbi [9], to show that the maximum
likelihood word error probability approaches zero as N . Unfortunately the di-
culty of the rst step, i.e., the computation of the ensemble IOWE, has kept us from
full success, except for some very simple coding systems, which we call repeat and ac-
cumulate codes. Still, we are optimistic that this technique will yield coding theorems
for a much wider class of interleaved concatenated codes. In any case, it is satisfying to
have rigorously proved coding theorems for even a restricted class of turbo-like codes.
Here is an outline of the paper. In Section 2 we quickly review the classical union
bound on maximum-likelihood word error probability for block codes on the AWGN
* Dariush Divsalars work, and a portion of Robert McElieces work, was performed
at JPL under contract with NASA. The remainder of McElieces work, and Hui Jins
work, was performed at Caltech and supported by NSF grant no. NCR-9505975,
AFOSR grant no. 5F49620-97-1-0313, and grant from Qualcomm.
1
channel, which is seen to depend on the codes weight enumerator. In Section 3 we
dene the class of turbo-like codes, and give a formula for the average input-output
weight enumerator for such a code. In Section 4 we state a conjecture (the interleaver
gain exponent conjecture) about the ML decoder performance of turbo-like codes. In
Section 5, we dene a special class of turbo-like codes, the repeat-and-accumulate codes,
and prove the IGE conjecture for them. Finally, in Section 6 we present performance
curves for some RA codes, using an iterative, turbo-like, decoding algorithm. This
performance is seen to be remarkably good, despite the simplicity of the codes and the
suboptimality of the decoding algorithm.
2. Union Bounds on the Performance of Block Codes.
In this section we will review the classical union bound on the maximum-likelihood
word error probability for block codes.
Consider a binary linear (n, k) block code C with code rate r = k/n. The (output)
weight enumerator (WE) for C is the sequence of numbers A
0
, . . . , A
n
, where A
h
de-
notes the number of codewords in C with (output) weight h. The input-output weight
enumerator (IOWE) for C is the array of numbers A
w,h
, w = 0, 1, . . . , k, h = 0, 1, . . . , n:
A
w,h
denotes the number of codewords in C with input weight w and output weight h.
The union bound on the word error probability P
W
of the code C over a memoryless
binary-input channel, using maximum likelihood decoding, has the well-known form
P
W

n

h=1
A
h
z
h
(2.1)
=
n

h=1
_
k

w=1
A
w,h
_
z
h
. (2.2)
In (2.1) and (2.2), the function z
h
represents an upper bound on the pairwise error
probability for two codewords separated by Hamming (output) distance h. For AWGN
channels, z = e
rE
b
/N
0
where E
b
/N
0
is the signal-to-noise ratio per bit.
3. The Class of Turbo-Like Codes.
In this section, we consider a general class of concatenated coding systems of the type
depicted in Figure 1, with q encoders (circles) and q 1 interleavers (boxes). The
ith code C
i
is an (n
i
, N
i
) linear block code, and the ith encoder is preceded by an
interleaver (permuter) P
i
of size N
i
, except C
1
which is not preceded by an interleaver,
but rather is connected to the input. The overall structure must have no loops, i.e., it
must be a graph-theoretic tree. We call a code of this type a turbo-like code.
Dene s
q
= {1, 2, . . . , q} and subsets of s
q
by s
I
= {i s
q
: C
i
connected to input},
s
O
= {i s
q
: C
i
connected to output }, and its complement s
O
. The overall system
depicted in Figure 1 is then an encoder for an (n, N) block code with n =

is
O
n
i
.
If we know the IOWE A
(i)
w
i
,h
i
s for the constituent codes C
i
, we can calculate
the average IOWE A
w,h
for the overall system (averaged over the set of all possible
interleavers), using the uniform interleaver technique [2]. (A uniform interleaver is
dened as a probabilistic device that maps a given input word of weight w into all
distinct
_
N
i
w
_
permutations of it with equal probability p = 1/
_
N
i
w
_
.) The result is
(3.1) A
w,h
=

h
i
:is
O
h
i
=h

h
i
:is
O
A
(1)
w
1
,h
1
q

i=2
A
(i)
w
i
,h
i
_
N
i
w
i
_
2
In (3.1) we have w
i
= w if i s
I
, and w
i
= h
j
if C
i
is preceeded by C
j
(see Figure 2.).
We do not give a proof of formula (3.1), but it is intuitively plausible if we note that
the term A
(i)
w
i
,h
i
/
_
N
i
w
i
_
is the probability that a random input word to C
i
of weight w
i
will produce an output word of weight h
i
.
For example, for the (n
2
+n
3
+n
4
, N) encoder of Figure 1 the formula (3.1) becomes
A
w,h
=

h
1
,h
2
,h
3
,h
4
(h
2
+h
3
+h
4
=h)
A
(1)
w
1
,h
1
A
(2)
w
2
,h
2
_
N
2
w
2
_
A
(3)
w
3
,h
3
_
N
3
w
3
_
A
(4)
w
4
,h
4
_
N
4
w
4
_
=

h
1
,h
2
,h
3
,h
4
(h
2
+h
3
+h
4
=h)
A
(1)
w,h
1
A
(2)
w,h
2
_
N
w
_
A
(3)
h
1
,h
3
_
n
1
h
1
_
A
(4)
h
1
,h
4
_
n
1
h
1
_ .
C
1
C
2
C
3
C
4
P
3
P
4
P
2
w w
1
w
2
w
3
w
4
h
1
h
2
h
3
h
4
n
3
n
4
N
3
N
4
n
1
N
2
N
1
N
n
2
input
output
output
output
Figure 1. A turbo-like code with
s
I
= {1, 2}, s
O
= {2, 3, 4}, s
O
= {1}.
P
j
C
j
C
i
(n
i
,N
i
)
(n
j
,N
j
)
h
i
w
j
n
i
N
j
Figure 2. C
i
(an (n
i
, N
i
) encoder) is connected to C
j
(an (n
j
, N
j
) encoder) by an interleaver of size N
j
. We
have the boundary conditions N
j
= n
i
and w
j
= h
i
.
4. The Interleaving Gain Exponent Conjecture.
In this section we will consider systems of the form depicted in Figure 1, in which
the individual encoders are truncated convolutional encoders, and study the behavior
of the average ML decoder error probability as the input block length N approaches
3
innity. If A
N
w,h
denotes the IOWE when the input block has length N, we introduce
the following notation for the union bound (2.2) for systems of this type:
(4.1) P
UB
W
def
=
n

h=1
_
N

w=1
A
N
w,h
_
z
h
.
Next we dene, for each xed w 1 and h 1,
(4.2) (w, h) = limsup
N
log
N
A
N
w,h
.
It follows from this denition that if w and h are xed,
A
N
w,h
z
h
= O(N
(w,h)+
) as N ,
for any > 0. Thus if we dene
(4.3)
M
= max
h1
max
w1
(w, h).
it follows that for all w and h,
A
N
w,h
z
h
= O(N

M
+
) as N ,
for any > 0. The parameter
M
, which we shall call the interleaving gain exponent
(IGE), was rst introduced in [2] and [3] for parallel concatenation and later in [4] for
serial concatenation. Extensive numerical simulations, and theoretical considerations
that are not fully rigorous lead to the following conjecture about the behavior of the
union bound for systems of the type shown in Figure 1.
The IGE Conjecture. There exists a positive number
0
, which depends on the q
component convolutional codes and the tree structure of the overall system, but not
on N, such that for any xed E
n
/N
0
>
0
,as the block length N becomes large,
P
UB
W
= O(N

M
) (4.4)
Eq. (4.4) implies that if
M
< 0, then for a given E
b
/N
0
>
0
the word error prob-
ability of the concatenated code decreases to zero as the input block size is increased.
This is summarized by saying that there is word error probability interleaving gain.
1
In [7], we discuss the calculation of (w, h) and
M
for a concatenated system of
the type depicted in Figure 1, using analytical tools introduced in [3] and [4]. For
example, for the parallel concatenation of q codes, with q 1 interleavers, we have

M
q + 2,
with equality if and only if each of the component codes is recursive. For a classical
turbo code with q = 2, we have
M
= 0, so there is no word error probability inter-
leaving gain. This suggests that the word error probability for classic turbo codes will
not improve with input block size, which is in agreement with simulations.
1
There is a similar conjecture for the bit error probability which we do not discuss in
this paper. Suce it to say that the interleaving gain exponent for bit error probability
is
M
1.
4
As another example, consider the serial concatenation of two convolutional codes.
If the inner code is recursive then,

M

_
d
o
free
+ 1
2
_
+ 1,
where d
o
free
is the minimum distance of the outer code. Therefore, for serial concate-
nated codes, if d
o
f
3 there is interleaving gain for word error probability. (If the inner
code is nonrecursive
M
0 and there is no interleaving gain.)
5. A Class of Simple Turbo-Like Codes.
In this section we will introduce a class of turbo-like codes which are simple enough
so that we can prove the IGE conjecture. We call these codes repeat and accumulate
(RA) codes. The general idea is shown in Figure 3. An information block of length
N is repeated q times, scrambled by an interleaver of size qN, and then encoded by
a rate 1 accumulator. The accumulator can be viewed as a truncated rate-1 recursive
convolutional encoder with transfer function 1/(1 +D), but we prefer to think of it as
a block code whose input block [x
1
, . . . , x
n
] and output block [y
1
, . . . , y
n
] are related
by the formula
(5.1)
y
1
= x
1
y
2
= x
1
+x
2
y
3
= x
1
+x
2
+x
3
.
.
.
y
n
= x
1
+x
2
+x
3
+ +x
n
.
P
P
rate 1
1/(1+D)
rate 1/q
repetition
N qN qN
qN x qN
permutation
matrix
qN
[w] [qw] [qw] [h]
LENGTH
[WEIGHT]
Figure 3. Encoder for a (qN, N) repeat and accumulate
code. The numbers above the input-output lines
indicate the length of the corresponding block, and
those below the lines indicate the weight of the block.
To apply the union bound from Section 2 to the class of RA codes, we need the
input-output weight enumerators for both the (qn, n) repetition code, and the (n, n)
accumulator code. The outer repetition code is trivial: if the input block has length n,
we have
(5.2) A
(o)
w,h
=
_
0 if h = qw
_
n
w
_
if h = qw.
5
The inner accumulator code is less trivial, but it is possible to show that (again assuming
the input block has length n):
(5.3) A
(i)
w,h
=
_
n h
w/2
__
h 1
w/2 1
_
.
It follows then from the general formula (3.1), that for the (qN, N) RA code represented
by Figure 3, the ensemble IOWE is
(5.4)
A
(N)
w,h
=
qN

h
1
=0
A
(o)
w,h
1
A
(i)
h
1
,h
_
qN
qw
_
=
_
N
w
__
qNh
qw/2
__
h1
qw/21
_
_
qN
qw
_ .
From (5.4) it is easy to compute the parameters (w, h) and
M
in (4.2) and (4.3).
The result is
(w, h) =
_
(q 2)w
2
_
(5.5)

M
=
_
(q 2)
2
_
. (5.6)
It follows from (5.6) that an RA code can have word error probability interleaving gain
only if q 3.
We are now prepared to use the union bound to prove the IGE conjecture for RA
codes. In order to simplify the exposition as much as possible, we will assume for the
rest of this section that q = 4, the extension to arbitrary q 3 being straightforward
but rather lengthy. For q = 4, (5.6) becomes
M
= 1, so the IGE conjecture is
P
UB
W
= O(N
1
) for E
b
/N
0
>
0
in this instance.
The union bound (2.2) for the ensemble of q = 4 RA codes is, because of (5.4),
P
UB
W
=
4N

h=2
h/2

w=1
_
N
w
__
4Nh
2w
__
h1
2w1
_
_
4N
4w
_ z
h
. (5.7)
Denote the (w, h)th term in the sum (5.7) by T
N
(w, h):
T
N
(w, h)
def
=A
w,h
z
h
=
_
N
w
__
4Nh
2w
__
h1
2w1
_
_
4N
4w
_ z
h
.
Using standard techniques (e.g. [8, Appendix A]), it is possible to show that for all
(w, h),
(5.8) T
N
(w, h) D2
h[F(x,y)+log
2
z]
,
where D = 4/

is a constant, x = w/4N, y = h/4N,

F(x, y) =

3
4
H
2
(4x) + (1 y)H
2
(
2x
1y
) +yH
2
(
2x
y
)
y
,
6
and H
2
(x) = xlog
2
(x) (1 x) log
2
(1 x) is the binary entropy function. The
maximum of the function F(x, y) in the range 0 2x y 1 2x occurs at (x, y) =
(0.100, 0.371) and is 0.562281, so that if log
2
z < 0.562281, the exponent in (5.8) will
be negative.
Let us therefore assume that log
2
z < 0.562281, which is equivalent to E
b
/N
0
=
(1/r) lnz = 4 lnz 4 ln2 0.562281 = 1.559 = 1.928 dB. If E is dened to be
E = log
2
z + 0.562281, it follows from (5.8) for all w and h,
(5.9) T
N
(w, h) D2
hE
.
What (5.9) tells us is that if E
b
/N
0
> 1.928 dB, most of the terms in the union bound
(5.7) will tend to zero rapidly, as N . The next step in the proof is to break the
sum in (5.7) into two parts, corresponding to those terms for which (5.9) is helpful,
and those for which it is not. To this end, dene
h
N
def
=
3
E
log
2
N,
and write
P
UB
W
=
4N

h=2
h/2

w=1
T
N
(w, h)
=
h
N

h=2
h/2

w=1
T
N
(w, h) +
4N

h=h
N
+1
h/2

w=1
T
N
(w, h)
= S
1
+S
2
.
Its easy to verify that when N is large enough, A
w+1,h
/A
w,h
< 1 for h h
N
and
w h/2 h
N
/2, which shows A
w,h
is a decreasing function of w for large N. Thus
the sum S
1
can be overbounded as follows (we omit some details):
S
1
=
h
N

h=2
h/2

w=1
T
N
(w, h)
=
h
N

h=2
T
N
(1, h) +
h
N

h=2
h/2

w=2
T
N
(w, h)
= O(N
1
) +
h
N

h=2
h/2

w=2
T
N
(w, h)
O(N
1
) +
h
N

h=2
h/2

w=2
A
2,h
z
h
= O(N
1
) +
h
N

h=2
h/2

w=2
O(h
3
/N
2
)z
h
= O(N
1
) +O(h
5
N
/N
2
)
= O(N
1
).
7
For the sum S
2
, we bound each term T
N
(w, h) by (5.9):
S
2
=
4N

h=h
N
+1
h/2

w=1
T
N
(w, h)

4N

h
N
+1
h/2

w=1
D2
hE
= D/2
4N

h
N
+1
h2
hE
D
2
Eh
N
(h
N
+ 1)
(1 2
E
)
2
= O(N
3
log
2
N)
= o(N
2
).
We have therefore shown that for the ensemble of q = 4 RA codes, if E
b
/N
0
>
1.928 dB,
(5.10) P
UB
W
= S
1
+S
2
= O(N
1
) +o(N
1
) = O(N
1
),
which as we saw above, is the IGE conjecture in this case.
Although the union bound gives a proof of the IGE conjecture for RA codes, the
resulting value of
0
is by no means the best possible. Indeed, if we use the recent
Viterbi-Viterbi improved union bound [9] to bound the sum S
2
, we can lower the value
of
0
considerably, e.g. for q = 4 from 1.928 dB to 0.313 dB. In Figure 4 and Table 1 we
display our numerical results on RA codes. There we compare the cuto threshold

0
for RA codes with q in the range 3 q 8 using both the classical union bound
and the Viterbi-Viterbi improved union bound to the cuto threshold for the ensemble
of all codes (i.e., random codes) of a xed rate. We believe that these values of

0
can be reduced still further, for example by using the bound of [6] instead of the
Viterbi-Viterbi bound.
q 3 4 5 6 7 8
RA Codes (Union Bound) 2.200 1.928 1.798 1.721 1.670 1.631
Random Codes (Union Bound) 2.031 1.853 1.775 1.694 1.651 1.620
RA Codes (Viterbi Bound) 1.112 0.313 0.125 0.402 0.592 0.731
Random Codes (Viterbi Bound) 0.214 0.224 0.486 0.662 0.789 0.885
Binary Shannon Limit 0.495 0.794 0.963 1.071 1.150 1.210
Table 1. Numerical data gleaned from Figure 4.
6. Performance of RA Codes with Iterative Decoding.
The results of this paper show that the performance of RA codes with maximum-
likelihood decoding is very good. However, the complexity of ML decoding of RA
8
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
2
1
0
1
2
3
4
5
6
7
8
Code Rate R

E
b
/
N
o
,

d
B

union bound random codes
viterbi bound random codes
shannon limit binary input
union bound RA codes
viterbi bound RA codes
Figure 4. Comparing the RA code cuto threshold to
the cuto rate of random codes using both the classical
union bound and the Viterbi-Viterbi improved union bound.
codes, like that of all turbo-like codes, is prohibitively large. But an important feature
of turbo-like codes is the availability of a simple iterative, message passing decoding
algorithm that approximates ML decoding. We wrote a computer program to imple-
ment this turbo-like decoding for RA codes with q = 3 (rate 1/3) and q = 4 (rate
1/4), and the results are shown in Figure 5. We see in Figure 4, for example, that
the empirical cuto threshold for RA codes for q = 3 appears to be less than 1 dB,
compared to the upper bound of 1.112 dB found in Table 1.
References.
1. C. Berrou, A. Glavieux, and P. Thitimajshima, Near Shannon limit error-
correcting coding and decoding: turbo codes, Proc. 1993 IEEE International
Conference on Communications, Geneva, Switzerland (May 1993), pp. 10641070.
2. S. Benedetto and G. Montorsi, Unveiling turbo codes: some results on parallel
concatenated coding schemes, IEEE Trans. on Inf. Theory, vol. 42, no. 2 (March
1996), pp. 409428..
3. S. Benedetto and G. Montorsi, Design of parallel concatenated convolutional
codes, IEEE Transactions on Communications, vol. 44, no. 5, (May 1996) pp. 591
600.
4. S. Benedetto, D. Divsalar, G. Montorsi, and F. Pollara, Serial concatenation
of interleaved codes: performance analysis, design, and iterative decoding, IEEE
Trans. on Information Theory, vol. 44, no. 3, (May 1998), pp. 909926.
9
3.0 2.5 2.0 1.5 1.0 0.5 0.0
10
-6
10
-5
10
-4
10
-3
10
-2
10
-1
10
0
E
b
/N
o
, dB
W
o
r
d

E
r
r
o
r

P
r
o
b
a
b
i
l
i
t
y
q=3
N=1024
10 it
20 it
30 it
20 it
30 it
q=3
N=16384
q=4
N=16384
20 it
30 it
10 it
q=4
N=1024
q=3
N=4096
q=4
N=4096
20 it
30 it
20 it
30 it
20 it
30 it
Figure 5. Simulated performance of iterative
decoding of RA codes on an AWGN channel.
5. D. Divsalar and F. Pollara, On the design of turbo codes, TDA Progress Report
vol. 42-123 (November 15, 1995), pp. 99121.
6. S. Dolinar, L. Ekroot, and F. Pollara, Improved Error Probability Bounds for
Block Codes for the Gaussian channel, Proc. 1994 ISIT, p. 243.
7. D. Divsalar and R. J. McEliece, On the design of concatenated coding systems
with interleavers, JPL TMO Progress Report vol. 2-134 (August 15, 1998), pp. 1
22. (http://tmo.jpl.nasa.gov/tmo/progress report/42-134/134D.pdf .)
8. W. W. Peterson and E. J. Weldon, Jr., Error-Correcting Codes, 2nd. ed. Cam-
bridge, Mass.: The MIT Press, 1972.
9. A. J. Viterbi and A. M. Viterbi, Improved union bound on linear codes for the
input-binary AWGN channel, with applications to turbo decoding, Proc. Winter
1998 Information Theory Workshop, San Diego California, Feb. 1998, p. 72.
10