THE STUDY OF MS USING MRI, IMAGE PROCESSING, AND VISUALIZATION

By

Jeremy Michael Nett
B.S.E.E., University of Louisville, 2000





A Thesis
Submitted to the Faculty of the
University of Louisville
Speed Scientific School
As Partial Fulfillment of the Requirements
For the Professional Degree





MASTER OF ENGINEERING





Department of Electrical and Computer Engineering



December, 2001

ii

THE STUDY OF MS USING MRI, IMAGE PROCESSING, AND VISUALIZATION



Submitted by:


Jeremy Michael Nett



A Thesis Approved on






By the Following Reading and Examination Committee:



Aly A. Farag, Thesis Director



Tom Cleaver



Kyung Kang



Robert Falk



Christina Kaufman



Stephen Hushek


iii

ACKNOWLEDGMENTS



The author would like to thank all those that made his education at the University
of Louisville possible, including the many contributors to the scholarship funds of which
he was profoundly honored to be a recipient.
The author would also like to thank Dr. Aly Farag and the CVIP Lab for an
interesting thesis topic, and support during the completion of this work. Additionally, the
author would like to thank Dr. Robert Falk of Jewish Hospital for spending much of his
valuable time guiding this work, commenting on its quality, and for many suggestions on
how to make the tools developed more applicable in the setting of medical research and
clinical use. The author would also like to thank the additional members of his thesis
committee for their review and critique of his work, including Dr. Stephen Hushek of
Norton Healthcare, Dr. Christina Kaufman of the Institute for Cellular Therapeutics of
the School of Medicine at the University of Louisville, Dr. Thomas Cleaver of the
Electrical and Computer Engineering Department, and Dr. Kyung Kang of the Chemical
Engineering Department.
Last but not least, the author would like to sincerely thank his parents, Michael
and Kathy Nett, for putting up with him, and his grandmother, Agnita Nett, for a place to
stay. Without their support, this work would have not been possible.



iv

ABSTRACT



Multiple sclerosis (MS), a well known disease of the nervous system in which the
myelin sheaths of axons are damaged, may be imaged using magnetic resonance imaging
(MRI) modalities. In this research, image analysis, volume registration, and scientific
visualization techniques are applied for quantitative and qualitative analysis of the
response of the disease to a proposed treatment regimen. Image analysis techniques are
applied, with the objective of automated and reliable quantitative evaluation of MS
lesions in the brain, through segmentation and classification of the brain and MS lesions.
To facilitate a time-series analysis of lesions, a volume registration technique is applied
to geometrically align MRI scans taken at different times over the course of treatment.
Additionally, scientific visualization techniques are utilized to facilitate three-
dimensional analysis of the disease pathology, and evaluation of changes in the structure
of lesions over a period of treatment. The result of these efforts is a preliminary system
for the study of MS using MRI.




v

TABLE OF CONTENTS



Page
APPROVAL PAGE..

ii

ACKNOWLEDGMENTS

iii

ABSTRACT. iv

NOMENCLATURE. viii

LIST OF TABLES xi

LIST OF FIGURES.. xii

I. INTRODUCTION... 1

A. General Introduction. 1

B. Introduction to Multiple Sclerosis 2

C. Introduction to Magnetic Resonance Imaging of MS.. 2

D. Introduction to Computer-Assisted Evaluation of MS 5

E. Previous Work in MS Studies Using MRI and Image
Processing Approaches..

7

F. Introduction to Lab Facilities and Data Acquisition. 9

G. Outline of the Components of the Researched Solution.. 11

H. Summary..

12
II. VOLUME REGISTRATION. 13

A. Introduction to Volume Registration... 13

B. Imaging Model. 19

C. Transformations... 26


vi

D. Volume Interpolation ... 30

E. Overview of Registration Approaches. 31

F. Registration Metric/Criteria. 32

G. Computation of the Mutual Information Metric.. 37

H. Search Algorithm. 43

I. Implementation.. 44

J. Results: MS Studies.. 44

K. Summary.. 50

III. BRAIN SEGMENTATION FROM THE HEAD. 51

A. Introduction and Necessity.. 51

B. Brain Segmentation Utilizing Registration.. 52

C. Results: Manual a priori Segmentation... 55

D. Results: Semi-Automatic a priori Segmentation. 59

E. Summary.. 62

IV. TISSUE SEGMENTATION. 63

A. Introduction and Necessity... 63

B. Feature Selection.. 64

C. Thresholding. 67

D. Segmentation by Image Enhancement. 69

E. Segmentation by Unsupervised Clustering.. 73

F. Bayesian Classification 77

G. Bayesian Classification: Gaussian Conditionals, Equal a priori
Probabilities...

80

vii


H. Bayesian Classification: Nonparametric Conditionals, a priori
Probabilities Modeled by Markov Random Fields

82

I. Quantification 89

J. Accuracy 90

K. Summary.. 91

V. VISUALIZATION. 92

A. Introduction.. 92

B. Visualization for the Study of Volume Registration 92

C. Visualization for the Study of Volume Segmentation. 96

D. Web-Based Presentation of Results..

98
E. Summary.. 99

VI. CONCLUSIONS AND RECOMMENDATIONS 101

REFERENCES.. 104

VITA. 109




viii

NOMENCLATURE



l = a volume coordinate vector in the image coordinate system
w = a volume coordinate vector in the world coordinate system
c =
coordinate vector locating the center of a volume
or
the number of states of nature
c(i) = a diffusion function
C =
transformation matrix for alignment of the center of a volume to
the origin in the world coordinate system
V =
transformation matrix for scaling a volume to account for
spatial sampling rates
= transformation matrix for accounting of a gantry angle
A
i,w
=
transformation matrix for conversion of image to world
coordinates
R = reference volume for registration
F =
floating volume for registration
or
a random field
T = transformation matrix for translation
R
x
, R
y
, R
z
= axis-angle rotation matrices
R = transformation matrix for rotation
T
FR
=
transformation matrix from image coordinates in the floating
volume to the reference volume
lower_left_N1,
lower_left_N2,
lower_left_N3
=
coordinates for alignment of a bounding cube for trilinear
volume interpolation

ix

V(sample
i
) =
interpolated sample from a volume for a sample coordinate of
sample
i

delta
x
, delta
y
, delta
z
=
translational differences between a coordinate to interpolate at,
and a point that forms an origin for trilinear interpolation
weight
i
= a weight and index i for trilinear interpolation
sample
i
= a sample at index i in a bounding cube for trilinear interpolation
p(x), q(x) = a probability distribution function
D(p||q) = relative entropy
I(X, Y) = mutual information of the random variables X and Y
H(X) = entropy of the random variable X
T

= a registration transformation with a parameter set

h() = a histogram
=
coordinates of a cluster center location
or
a parameter of a Gaussian distribution
x = a feature vector
d = the dimension of the feature vector x
= a state of nature, or class
P(
i
|x) =
the a posteriori probability of class
i
given a feature
measurement x
p(x|
i
) =
the class conditional probability of observing a feature
measurement x, given a class
i


x

P(
i
) = the a priori probability of observing the class
i

= a d x d covariance matrix
= standard deviation
, = estimated mean and standard deviation parameters
() = a Parzen window function
n = number of training samples
N = a neighborhood structure
S = set of a random field
r = order of a neighborhood structure
Z = a partition function in a Gibbs distribution
U(f) = an energy function in a Gibbs distribution
f
k,l
= an observation at a lattice point in a random field
,
1
,
2
= parameters of the energy function U(f)
T = temperature parameter of a Gibbs distribution


xi

LIST OF TABLES



I. Samples and sample weightings for trilinear interpolation...

31
II. Parameters of the MS studies used for assessing the registration software
developed..


45
III. Execution time and registration parameters for MS studies used.

46
IV. Quantification of MS disease burden from the sample results given in
figure 25


90



xii

LIST OF FIGURES



1. Sample slices from FLAIR MRI studies of patients with MS.
Hyperintense regions in the brain are indicative of plaques caused by MS..


4
2. The front-end to the patient database of Jewish Hospital.

9
3. The SGI Onyx2 visual supercomputer used as a computing platform in this
research..


10
4. Illustration of the registration problem in two dimensions. The squares in
the left and right figures represent the respective scanning area. Compared
with one another, the anatomy is located at a different position between
the two volumes, and has been rotated......




14
5. Scout scans for the same patient, at different time points. Note that the
lines that indicate the slice planes do not correspond in the two different
studies.



15
6. Sample slice comparison between two scans of the same patient, taken at
different points in time. Each column contains sample slices from one
study. The rows contain the same slice number from each study. As
apparent, the anatomy is not geometrically aligned between the two
scanning volumes...





16
7. Examples of sagittal scans (a), coronal scans(b), and axial scans (c)....

19
8. Formation a volume from an ordered set of images..

20
9. Illustration of eight samples in an isotropic volume. Each sample is
located at lattice points, with integer coordinates, and equal distances
between lattice locations which have a difference of 1.0 between a
coordinate..




22
10. Illustration of the parallel computation of the joint histogram necessary for
computation of the mutual information metric..


41

xiii


11. Sample registration results from each of the seven MS data sets. The first
column(a, d, g, j, m, p, and s) is a sample slice from the floating volume
used. The second column (b, e, h, k, n, q, and t) is the corresponding slice
from the re-sampled reference volume. The third column (c, f, i, l, o, r,
and u) is the checkerboard composite image of the two corresponding
slices from the floating and re-sampled reference volumes. The floating
volume and reference volumes used in each trial were from the same
patient








49
12. Samples of the (non-expert) manual segmentation of a patients brain from
the remaining tissue of the head. The slices obtained from the MRI study
((a), (b), and (c)) are manually segmented to obtain slices containing the
brain ((d), (e) and (f), respectively). The manually segmented images are
then made into binary masks ((g), (h), and (i), respectively)





56
13. Registration of two MRI studies of the same patient, taken at different
times. The first study ((a), (b), and (c)) was manually segmented to
extract the brain from the head. The second study ((d), (e), and (f)) was
registered to the first (the re-sampled slices are shown)




57
14. Segmentation of the second study by a binary mask calculated from the
registration parameters, and the binary mask used to segment the a priori
study. The raw input slices are shown ((a), (b), and (c)). The binary mask
was obtained by use of the registration parameters and the a priori mask
((d), (e) and (f), respectively). By application of the mask, the segmented
volume is obtained ((g), (h), and (i), respectively)..






58
15. Sample brain segmentation results of the a priori scan using a semi-
automatic technique improved upon in the CVIP Lab. The input slices
((a), (b), and (c)) are semi-automatically segmented to form a binary mask
of the brain ((d), (e), and (f), respectively). The mask is then applied to
form the segmented volume ((g), (h), and (i), respectively).





60
16. Brain segmentation of the second study. The second study ((a), (b), and
(c)) was registered to the a priori scan. The brain mask for the second
study ((d), (e), and (f), respectively) is then obtained using the a priori
brain mask, and the registration parameters. The binary brain mask is then
applied to generate the segmented volume ((g), (h), and (i), respectively)...





61
17. A sample slice from an MRI study of an MS patient, with FLAIR (a and
d), T1 (b and e), and T2 (c and f) weightings. The appearance of lesions is
not consistent between different weightings. MS lesions are easily
identified in the FLAIR images as hyperintense regions in the brain



66

xiv


18. Segmentation by thresholding. The input slice (a) is classified into three
classes, based on selection of class on the interval in which an intensity
falls. Corresponding labeled image slice is shown in (b), with the dark
gray indicating CSF, the light gray indicating normal brain material, and
the red indicating MS lesions





68
19. The histogram of the input slice shown in figure 18 (a). Intervals
containing a tissue class can be visually identified by a human. Peaking of
the histogram at low intensities represents CSF, and at high intensities, MS
lesions. Peaking of the histogram in the intermediate intensities indicates
normal brain tissue.





68
20. Non-linear anisotropic diffusion filtering of MRI slices for segmentation.
Input slices (a and c) are FLAIR weighted imagery. The corresponding
output slices (b and d, respectively) show much enhanced images, but with
the loss of some MS lesions...




72
21. Illustration of types of errors encountered with segmentation via image
enhancement...


73
22. Results of k-means clustering, using 5 clusters. The input slice is shown
in (a). The resulting clustered output image is shown in (b). By manual
re-labeling of clusters such that clusters of similar tissue belong to the
same cluster, the output slice in (c) is obtained...




76
23. Sample result of tissue classification in the brain using a simple Bayesian
classifier, with equal a priori probabilities, and Gaussian class conditional
probabilities



82
24. Neighborhood systems for an image modeled as a Markov random field.

86
25. Sample classification results obtained using a Bayesian classifier, with
class conditional probabilities modeled nonparametrically, and class a
priori probabilities obtained from a Markov random field model of the
image. Input slices are shown in (a) and (c), and the classified images are
shown in (b) and (d). In (b) and (d), normal brain material is indicated by
the color yellow, CSF by gray, and lesions as red.






89
26. Generation of the checkerboard image, by formation of a composite image
of square pixel regions from the re-sampled reference and floating
volumes.



93

xv


27. Sample slices from different registration studies, showing the a slice from
the floating volume ((a), (d), and (g)), corresponding slices from the re-
sampled, registered reference volume ((b), (e),and (h), respectively), and
the checkerboard slices ((c), (f), and (i), respectively)..




94
28. Volume rendering and arbitrary volume re-slicing for comparison of two
volumes that are registered via maximization of mutual information...


95
29. Frame interpolation ...

96
30. A sample of a web page developed for presentation of the results of this
study on the Internet... 99


1

I. INTRODUCTION



A. General Introduction

The purpose of the research documented in this thesis is to provide for
quantitative, computer vision based evaluation of the disease burden of a patient suffering
from multiple sclerosis (MS), a debilitating disease of the human nervous system. This
computer vision system shall allow for highly-automated quantification of the disease
burden of a patient, based upon measurement of brain and lesion volumes of a patient.
Additionally, it is desirable to explore facilitation of qualitative evaluation of the disease,
and to allow such quantitative and qualitative approaches over the course of an extended
period of study, with imaging studies taken periodically.
The imaging studies that will be evaluated are magnetic resonance imaging (MRI)
studies of the human brain. These studies shall be administered at a local hospital, and
patient data transferred to local (laboratory-based) computing facilities for analysis.
Analysis shall consist of alignment of studies to a reference study, segmentation of the
head, brain, and lesions, and quantitative computation of brain and lesion volumes.
Additionally, scientific visualization techniques shall be applied to allow for comparison
of studies taken at different time points in the course of the study.
It should be noted that in the context of the medical study that this research
addresses, patients are known to have MS, and will more likely than not be in advanced
stages of the disease.


2

B. Introduction to Multiple Sclerosis

Multiple sclerosis is a disease of the human central nervous system, affecting
approximately 250,000 to 350,000 people in the United States alone [1]. MS results in a
variety of clinically-observable deficiencies, such as speech difficulties, pain, impairment
of senses, loss of muscle control, and cognitive abnormalities [2].
In the brain, MS results in the inflammation and destruction of myelin, a fatty
covering insulating nerve cells [2]. This damage results in decreased ability of the
nervous system to control the body, leading to the clinically-observable symptoms of the
disease.
The causes of MS are not clearly accepted in the medical community.
Geographic, genetic, and environmental factors all seem to be present [1]. Many
researchers have proposed that MS is an autoimmune disease [1].
Though there are promising research developments, at this time, no cure is known
for this disease. Several treatment options are available, and allow for management of
the disease. Despite this, most patients progress in disability over the course of their life
[1]. Though not usually a fatal disease in and of itself, the resulting disabilities may
contribute to accidental mishaps [1].

C. Introduction to Magnetic Resonance Imaging of MS

Magnetic resonance (MR) imaging (MRI) is a medical imaging technology that
allows for non-invasive imaging of patient anatomy through the measurement of emitted

3

nuclear magnetic resonance (NMR) signals. An MRI system is constructed of at least
three basic subsystems: a main magnet to produce a strong, homogenous, static field,
denoted as the B
0
field; a subsystem for generation of a gradient magnetic field, for signal
localization; and a radio-frequency (RF) subsystem, for generation and transmission of a
rotating magnetic field, denoted as the B
1
field, and measurement of NMR signals [3].
An MRI system evokes NMR signals from tissue to be imaged. By controlling
the acquisition parameters of the scan, different image weightings may be obtained,
allowing for different and/or improved image contrast between different types of tissue.
Image contrast in MRI studies is fundamentally based on the measurement of spin-lattice
relaxation (T1) time , spin-spin relaxation (T2) time , and nuclear spin density (PD).
Different types of images include T1-, T2-, PD-weighted images, and fluid
attenuated inversion recovery (FLAIR) images. A study is said to be a T1-weighted
study when the dominant tissue characteristic generating image contrast is the T1 time of
a tissue [3]. A study is said to be T2-weighted when the dominant tissue characteristic
generating image contrast is the T2 time of a tissue [3]. Finally, a study is said to be PD-
weighted when the dominant tissue characteristic generating image contrast is the nuclear
spin density of a tissue [3].
Inversion-recovery sequences are used to utilize T1 contrast, while allowing for
differentiation of tissues with approximately equal T2 times or nuclear spin densities [3].
FLAIR studies are implemented with inversion-recovery sequences designed to suppress
the intensity of cerebro-spinal fluid (CSF).

4

When patients with MS are imaging using MRI modalities, lesions (also referred
to as plaques or deficits) can be contrasted against surrounding, normal brain tissue, by
choice of appropriate scan parameters, and depending on the state of the lesion [4].
Figure 1 below shows several slices of different FLAIR MR images of patients
with MS. MS lesions appear as hyper intense regions in the patient's brain.


(a)

(b)

(c)

(d)
FIGURE 1: Sample slices from FLAIR MRI studies of patients with MS. Hyperintense
regions in the brain are indicative of plaques caused by MS.

The use of MR imagery in the evaluation of MS involves the identification of
abnormal brain tissue (MS lesions), and normal, non-diseased brain tissue (gray and

5

white matter). MRI has been found to be a sensitive marker to changes in disease
progression, and evaluation of MRI studies of MS patients can be useful as an outcome
measurement in MS studies [4].
Despite these statements, evaluation of the disease cannot be completely based
upon MRI findings. MS lesion activity observed in MRI studies of the brain does not
always correspond to clinically-observed deficits [4]. As well, quantification of MRI
studies of MS have yet to be perfected, and most studies of MS do not generate enough
MRI data to evaluate the nuances in the application of MRI to study MS [4].

D. Introduction to Computer-Assisted Evaluation of MS

It is desirable to apply computer vision techniques to the study of MS using MR
imaging. Allowing a computer to automatically identify normal and abnormal brain
tissue would free an expert from the arduous task of manually examining each slice of a
study, while generally increasing the reproducibility of the identification by removing the
subjectivity of the human observer. The fields of segmentation and classification address
the task of this identification. Computer vision tools would also be useful for automatic,
retrospective alignment of patient studies, taken at different points in time, to allow for
qualitative comparison of different studies of a patient of the course of his or her
treatment. This type of analysis would aide the expert in deciding if the disease is
responding well to the present treatment, or if a change in the treatment is warranted.
The study of image/volume segmentation and classification addresses two tasks.
The first is to segment the brain (gray and white matter, lesions, and cerebro-spinal fluid

6

(CSF)) from the remaining tissues in the head (skull, skin, eyes, etc.). The second task is
to then segment the remaining imagery into CSF, normal brain material (gray and white
matter), and diseased brain tissue (here, MS lesions). Segmentation of the brain from the
remaining tissues image in the head allows for analysis of the brain only, and simplifies
this subsequent analysis. Segmentation of the remaining tissue into CSF, normal brain
tissue, and diseased tissue then allows for quantification of disease load.
There are several difficulties encountered with segmentation in the above context,
including noise, motion artifacts, limited spatial resolution, partial volume artifacts,
chemical shift artifacts, magnetic field inhomogeneity, and limited tissue contrast [3].
When a scan is taken of a patient, and later scans are taken, two factors must be
considered. First, scan parameters are likely to differ between scans, including voxel
dimensions, and perhaps scan weightings. Secondly, and most importantly, the patient
will not be positioned in the same location in the scanning volume. Thus, anatomy in two
different studies, taken with exactly the same scan parameters, will not be geometrically
aligned in the scanning volume, disallowing the possibility of comparing earlier and later
scans on a slice-by-slice basis. To allow for a comparison of multiple scans, a geometric
alignment of the anatomy imaged must be found; this function is known as volume
registration.
Difficulties encountered in volume registration include the same difficulties
encountered for segmentation.


7

E. Previous Work in MS Studies Using MRI and Image Processing Approaches

Multiple sclerosis is a common enough disease, and with enough intrigue to
investigation of cures and treatments, that it has received a great deal of attention by a
number of research centers. Likewise, the disease and MR imaging of its victims has
received a fair amount of attention from several individuals and groups interested in the
application of computer vision techniques for analysis of such MR studies.
Several studies at notable institutions have incorporated computer analysis of
MRI studies of patients with MS. An extensive study and application of computer-aided
analysis was conducted at the Surgical Planning Laboratory at Brigham and Womens
Hospital, of Boston, Massachusetts, U.S.A. [5, 6]. The system developed for the study
and quantification of MS utilizes an adaptive, statistical segmentation algorithm, known
as the expectation-maximization (EM) algorithm, for semi-automatic segmentation of
brain tissue. Three millimeter, continuous slices, weighted as PD and T2 scans are
acquired. Data is filtered, for noise smoothing. The brain is classified into four classes:
white matter, gray matter, CSF, and white matter lesions. Disease burden is evaluated by
computation of the volume of individual lesions and total lesion volume.
In [8], a brain tissue model is developed for segmentation of MRI studies of
patients with MS. The developed model is three-dimensional, voxel based, and provides
prior probabilities of white matter, gray matter, and CSF. In addition to providing prior
probabilities, the model-based approach is used to restrict the search for MS lesions to the
white matter of a patients brain. Also, as a part of [8], statistical and decision tree
classifiers are compared.

8

The research documented in [9] segments brain tissue using a stochastic
relaxation method, originally developed for applications in image enhancement [10]. The
implemented algorithm, incorporating the use of the iterated conditional modes (ICM)
algorithm [10], claims to identify MS lesions in the white matter of a patient, and also
claims to perform a partial volume analysis of the resulting segmentation.
In [11], lesion segmentation is implemented by application of fuzzy objects and
fuzzy connected sets ideas [12], and the approach is semi-automatic. An operator selects
sample points for gray and white matter, and CSF, which then are detected in their
entirety by detection as a so-called fuzzy connected set. The voids in the union of these
tissue classes are potential lesions, which are presented to an operator for acceptance or
rejection as MS lesions.
In [13], an algorithm that is claimed to be fully automatic for segmentation of MS
lesions from MRI studies is presented. The general idea is for a model-aided, intensity-
based brain segmentation to proceed, as classification of gray and white matter, and
CSF. Features not well explained by the model are considered outliers, and are labeled as
MS lesions.
This previous work comprises a rich set of ideas from which to build upon, to
create an engineered system for quantitative and qualitative evaluation of MS using MRI,
in the local research community. There is opportunity for improvement and further
innovation, however, through classification of gray matter lesions, further automation,
and a more comprehensive approach to the study of MS.

9

F. Introduction to Lab Facilities and Data Acquisition

MRI studies utilized in the development of this system were performed at Jewish
Hospital HealthCare Services, of Louisville, KY, U.S.A., under the direction of Dr.
Robert Falk. Scanning equipment used included 1.0 T and 1.5 T GE MRI machines, and
a 1.0 T Picker MRI machine.
MRI scan data was transferred from Jewish Hospital to computing facilities of the
Computer Vision and Image Processing (CVIP) Lab of the Electrical and Computer
Engineering Department, of the Speed Scientific School, at the University of Louisville.
This transfer involved the use of a front-end, PC-based tool to allow for manual retrieval
of data from the database stored at Jewish Hospital. This front-end is shown in figure 2.


FIGURE 2: The front-end to the patient database of Jewish Hospital.


10

This front-end application allows for approved studies to be downloaded over the
Internet, from Jewish Hospital, to a PC in the CVIP Lab. From this point, data is then
transferred to one of the supercomputers in the lab, for subsequent analysis.
Used for a research and development computing platform was a SGI Onyx2
visual supercomputer, with 40 300 MHz MIPS R12000 processors, 20 Gb of shared
RAM, and running Irix 6.5 as its operating system. It is important to note that while none
of the developed approaches require the use of this supercomputer, many benefit from the
available parallel processing capabilities of this machine. This machine is pictured below
in figure 3.


FIGURE 3: The SGI Onyx2 visual supercomputer used as a
computing platform in this research.

MRI studies are retrieved from Jewish Hospital in the form of a series of images
formatted per the Digital Imaging and Communications in Medicine (DICOM)

11

specification of the American College of Radiology (ACR) and National Electrical
Manufacturers Association (NEMA) [42]. Each image from a study is in a separate file.
The images are extracted from the DICOM files and formatted for use, including
necessary scaling. Additionally, necessary parameters from the DICOM header (such as
pixel sizes, slice thickness, etc.) are read and recorded for use.

G. Outline of the Components of the Researched Solution

The approaches taken in this research address two fronts of the study of multiple
sclerosis using MRI studies, image processing techniques, and visualization: quantitative
evaluation of disease burden via the use of segmentation and classification techniques,
and qualitative evaluation via the use of volume registration and visualization paradigms.
Segmentation and classification techniques from the fields of pattern recognition
and image processing are applied for segmentation of the brain from the head of a patient,
followed by classification of the extracted volume into CSF, normal brain material, and
diseased brain tissue. Emphasis is placed upon automation of these steps, using computer
vision tools, to allow for timely assessment of a large number of MRI studies, and for
objective, repeatable measurement of disease burden from MR images. From a
segmentation of normal brain material and diseased tissue, quantification of disease
burden may be evaluated by computation of the normal and diseased brain volumes of a
patient.
Qualitative evaluation of the disease burden of a patient is important to allow for
evaluation of disease pathology interactively by an expert, and to allow for judgment of

12

the accuracy of computer vision techniques. These problems are addressed in this study
by the application of volume registration and visualization techniques. Volume
registration is necessary to geometrically align scans taken at different points in time, to
compensate for the different locations in the scanning volume that the anatomy of interest
falls within. Visualization techniques are applied to allow for judgment of the quality of
volumetric alignment, validation of segmentation and classification approaches, and to
facilitate observation and discovery of the pathology MS, and response to treatment.

H. Summary

For the study of multiple sclerosis in brain MRI studies, it is desired to introduce
computer vision techniques to automate and improve quantitative measurements of
disease burden. Additionally, it is desired to introduce computer vision and visualization
techniques to facilitate qualitative evaluation of the disease, and response to treatment.


13

II. VOLUME REGISTRATION



A. Introduction to Volume Registration

Typically in an MRI study, a patient is placed in the scanner with little regard for
positioning of the anatomy of interest. The only constraints are that the anatomy of
interest falls within the scanning volume, and that the patient is generally placed in some
orientation such that gross anatomical features are placed in some direction (for example,
the patient's nose points upward in the scanner).
In the context of qualitative evaluation of MS studies however, this arrangement
leads to hindrances in evaluation, due to the fact that multiple scans must be compared
with one another. Due to the largely arbitrary positioning of the anatomy in the scanner,
in a slice-by-slice comparison between studies, quite different anatomy can by chance be
located on the same slice numbers in different studies. The goal of registration, therefore,
is to align the anatomy from one scan, to the anatomy from a second. When this function
is performed, the resulting volumes are said to be registered; without this function, the
volumes are said to be mis-registered.
Figure 4 below illustrates the problem of mis-registration in two dimensions.
Here, the box represents a scanning area. Comparing the positioning of the anatomy in
the left and right scanning areas, the volumes are out of alignment by three factors: two
translation quantities (a horizontal and vertical), and a rotation angle. Registration is the
function by which these quantities are discovered or calculated, thereby supplying the
information necessary to relate one volume to another, in the sense of alignment.

14



FIGURE 4: Illustration of the registration problem in two dimensions. The squares in
the left and right figures represent the respective scanning area. Compared with one
another, the anatomy is located at a different position between the two volumes, and has
been rotated.

Figure 5 below provides further evidence of this problem. Shown are two
different images, known as scout images, for the same patient, for MRI studies done at
different points in time. The scout images show the positions of the imaging planes in a
study, and are recorded by the technologist controlling the MRI machine at the time of
data acquisition. As can be observed from the two scout images shown, the positions of
these lines do not correspond with one another, indicating that a slice-by-slice
comparison of the studies will not allow for comparison between the same physical
locations in the patients brain.


15


(a)

(b)
FIGURE 5: Scout scans for the same patient, at different time points. Note that the lines
that indicate the slice planes do not correspond in the two different studies.

Finally, figure 6 below shows several slices from two MS studies. The sample
slices shown in the first column are from a single study. The sample slices shown in the

16

second column are the slices from a later study, having the same slice number as the
slices shown in the first column. As can be observed, the anatomy imaged is not
geometrically aligned.


(a)

(b)

(c)

(d)

(e)

(f)
FIGURE 6: Sample slice comparison between two scans of the same patient, taken at
different points in time. Each column contains sample slices from one study. The rows
contain the same slice number from each study. As apparent, the anatomy is not
geometrically aligned between the two scanning volumes.

17

Registration is a crucial problem to be addressed in many medical imaging tasks,
and can be useful for facilitating a comparison between two or more studies of a patient,
merging two or more imaging modalities to facilitate diagnosis, and even to aid in
segmentation.
It is desirable to be able to perform registration using computer vision approaches,
rather than imposing limitations in the scanning procedure, or affixing artificial fiducial
markers on the patients head. For best accuracy, artificial markers would likely be
affixed to the skull, and therefore would be inconvenient and potentially painful for the
patient. Additionally, this procedure would also introduce a risk of infection.
Furthermore, using computer vision techniques, it is also desirable to be able to apply
registration retroactively, allowing for current data sets to be aligned with data sets taken
previously in a patients history, or perhaps with an imaging modality that prevents the
use of artificial markers.
In the study of MS using MRI, for comparison of scans taken at different points of
time in a clinical study, of the same patient, a registration technique is necessary. Such a
tool would allow for alignment of patient anatomy in the different scans. When this
alignment is accomplished, qualitative comparison of scans becomes easier to an expert
viewer, as image slices will now contain the same anatomy, and quantitative comparison
between studies is enabled in the same manner.
Registration can also be used to assist in segmentation. For example, if a model
of patient anatomy is known, then a study can be registered to that model, allowing for
segmentation of certain classes of problems to be made trivial, as the segmentation of the

18

data set is then known a priori from the model. In this research, this approach is used for
segmentation of a patients brain from his head, and is discussed in detail in later portions
of this thesis.
In this context, a well-known and useful volume registration technique, known as
registration by maximization of mutual information, was investigated [14, 15]. This
technique has generally been found to perform well, and is useful in clinical settings [16].
This technique was studied, implemented, and tested using MS patient studies.
Additionally, the performance of this method was enhanced by application of parallel
programming techniques.
Below, this technique, implementation, and parallelization will be discussed.
First, an imaging model will be described, by which data from an imaging study is
modeled in three-dimensional space. Secondly, the types of transformations between two
volumes that are useful in this context will be addressed. Thirdly, volume registration
using computer vision, and specifically, the criterion of mutual information, shall be of
concern. Then, the application of this criterion to the problem of volume registration will
be expounded upon, followed by the application of parallel programming techniques to
improve the performance of the software implementation of this registration technique.
Lastly, results obtained using this method are shown, for MS studies using MRI.


19

B. Imaging Model

Of preliminary concern is the construction of a volume from an imaging study,
and modeling the relationship of this volume to three-dimensional space.
Data obtained from an imaging study such as an MRI-based modality, or
computed tomography (CT), consists of a set of ordered slices, representing consecutive
slices in parallel, along a direction normal to each slice. MRI studies are typically taken
along one of three customary sets of planes: parasagittal, coronal, or axial planes.
Parasagittal planes are planes parallel to the sagittal plane, which itself divides the brain
into two symmetrical parts; coronal planes are parallel to the long axis of the body, and
normal to parasagittal planes; axial planes are normal to parasagittal and coronal planes
[17]. Given a choice of imaging planes, slices are then ordered in a particular direction,
such as patient left to right, and so forth. Examples of slices from each of the three
imaging directions are shown below in figure 7.


(a)

(b)

(c)
FIGURE 7: Examples of sagittal scans (a), coronal scans (b), and axial scans (c).


20

Given a set of ordered image slices, a volume is formed by placing slices
consecutively in the volume, ascending in a direction normal to each image plane by a
fixed amount (determined by slice thickness and slice gap). Each slice is placed
squarely upon the slice lower in the volume, such that the faces of the three-
dimensional volume formed have right angles between edges. Figure 8 below illustrates
this process.



FIGURE 8: Formation a volume from an ordered set of images


21

There are two coordinate systems that will be used to formulate the mechanics of
the operation of registration. The first is an image coordinate system, to represent a
volume in a 3D isotropic coordinate system; this coordinate system is specific to a
particular volume. The second coordinate system that will be considered is a world
coordinate system, of which there is only one. This coordinate system places each
volume in 3D space proper, accounting for origin and voxel sizes (a voxel is a sample
from the volume; a pixel in an image slice).
The image coordinate system provides a simple way of addressing samples within
a volume. The volume is formed as a set of lattice points, such that each lattice point has
integer coordinates. Furthermore, this lattice is isotropic, meaning that the distance
between consecutive lattice locations differs by a value of 1.0 in a single coordinate.
Figure 9 below illustrates a portion of such an isotropic volume. At non-integer
coordinates, the volume may be approximated using a volume interpolation technique.
It should be noted that [20] provides documentation of many of the
transformations and representations that will be used below to mathematically formulate
an imaging model, and the types of operations permitted with the model.
Coordinates in both the image and world coordinate systems, as well as all
transformations will be represented as column vectors, in homogeneous coordinates. A
coordinate in the image coordinate system will be given by the vector l, and coordinates
in the world coordinate system will be represented by the vector w, both shown below in
eq. 1.


22


( )
( )
T
z y x
T
z y x
w w w w
l l l l
1
1
=
=

(1)



FIGURE 9: Illustration of eight samples in an isotropic volume. Each sample is located
at lattice points, with integer coordinates, and equal distances between lattice locations
which have a difference of 1.0 between a coordinate.

The world coordinate system places each volume into 3D space, accounting
properly for origins, voxel sizes, and a gantry angle. Conversion from image coordinates
to world coordinates must consider these factors.
The convention used for setting an origin for a volume is to calculate the center
location of the volume. Thus, if a volume has a dimension d
i
for the dimension i, then the
center coordinate is given by c
i
= (d
i
1) / 2. Therefore, for volume dimensions d
x
, d
y
,
and d
z
, the center is computed as given in eq. 2.

23


T
z y x d d d d
c |
.
|

\
|
=

= 1
2
1
2
1
2
1
2
1

(2)


Incorporation of the volume center as an origin for locating the volume in 3D
space is given by multiplication of the image coordinate vector l by the matrix C, given in
eq. 3.


(
(
(
(

=
1 0 0 0
1 0 0
0 1 0
0 0 1
z
y
x
c
c
c
C
(3)


Voxel sizes are the spatial sampling periods of the imaged volume. There are
three such sizes, v
x
, v
y
, and v
z
, representing the size of a voxel in each of the coordinate
dimensions. Incorporation of the voxel sizes in proper scaling of the image coordinates
to world coordinates is given by multiplication of the image coordinate vector l by the
matrix V, given below in eq. 4.



24

(
(
(
(

=
1 0 0 0
0 0 0
0 0 0
0 0 0
z
y
x
v
v
v
V
(4)


Finally, an additional term, a gantry angle , is taken into consideration. This
term measures the angle of the gantry as it is positioned in the scanner. This factor is
easily incorporated in conversion of an image coordinate vector l to world coordinates by
the matrix , given below in eq. 5. Note that this matrix is simply a rotation matrix, a
class of transformations to be discussed further below.


(
(
(
(

=
1 0 0 0
0 1 sin 0
0 0 cos 0
0 0 0 1

(5)


Therefore, with each of these matrices considered, the conversion from image to
world coordinates can be represented as the matrix product of the , V, and C matrices.
The resulting coordinate transformation is referenced as A
i,w
, and is given below in eq. 6.



25

(
(
(
(

= =
1 0 0 0
sin sin 0
cos 0 cos 0
0 0
,


y y z z y
y y y
x x x
w i
c v c v vz v
c v v
c v v
C V A
(6)


It will be of interest to be able to convert from world coordinates into image
coordinates. This can be simply done by inverting the A
i,w
matrix, to obtain A
i,w
-1
, as
given in eq. 7 below, in closed form.


(
(
(
(
(
(
(

1 0 0 0
1
cos
sin
0
0
cos
1
0
0 0
1
1
,
z
z z
y
y
x
x
w i
c
v v
c
v
c
v
A

(7)


Thus, the relationships given in eq. 8 allow for conversion between image and
world coordinates.


w A l
l A w
w i
w i
=
=
1
,
,

(8)


26


Each of the above parameters (d
x
, d
y
, d
z
, v
x
, v
y
, v
z
, ) are known a priori
parameters obtained from the DICOM header of a study.
Two volumes will be used in the registration, a reference volume R, and a floating
volume F. The purpose of registration is to find a transformation that aligns these two
volumes.

C. Transformations

For alignment, rotation and translation will be considered. Alignment
transformations that consist of only translation and rotation are known as rigid-body
transformations.
Only the translation and rotation transformations are considered here, as these
effects are observed in the data to be by far the predominant effects necessary for
registration of two or more studies of the same patient. Other transformations are judged
to be either negligible in effect, or not applicable. Included in these transformations are
scaling (or magnification) between two studies, as the voxel sizes (the metric dimensions
of a voxel) of each study are known from the scanner, and are known to be accurate
enough for use, given that the scanner is in good repair.
Translation quantifies a three-dimensional offset along coordinate axes between
two volumes in world coordinates. In homogeneous coordinates, with t
x
, t
y
, and t
z

representing translation along the x, y, and z coordinate axes, respectively, the translation

27

can be formulated by a matrix T, that multiplies a coordinate w to obtain a translated
coordinate vector. T is given below in eq. 9.


(
(
(
(

=
1 0 0 0
1 0 0
0 1 0
0 0 1
z
y
x
t
t
t
T
(9)


A rotation is quantified by three angles,
x
,
y
,
z
, that are axis-angle
parameterizations of rotation. These parameters specify a rotation around the unit
direction vectors of the world coordinate system. Corresponding to each parameter is a
rotation matrix, denoted as R
x
, R
y
, and R
z
, respectively. A complete rotation matrix,
taking into account each rotation angle, is given by the matrix product of R
x
, R
y
, and R
z
,
and forms the matrix R, given in eq. 13 below. R
x
, R
y
, and R
z
are given in eqs. 10
through 12, respectively.


(
(
(
(

=
1 0 0 0
0 cos sin 0
0 sin cos 0
0 0 0 1
x x
x x
x R



(10)



28

(
(
(
(

=
1 0 0 0
0 cos 0 sin
0 0 1 0
0 sin 0 cos
y y
y y
y R



(11)


(
(
(
(

=
1 0 0 0
0 1 0 0
0 0 cos sin
0 0 sin cos
z z
z z
z R



(12)


Rz Ry Rx R = (13)


Utilization of the matrices T and R constitutes a rigid-body transformation. The
complete transformation can be written as the product of these two matrices, and shall be
denoted as A. To calculate the rigid body transformation of a coordinate w, the
coordinate vector w is multiplied by the transformation matrix A, as shown in eq. 14,
where w
1
is a coordinate, and w
2
is the rigid body transformation of that coordinate.


1 2
w A w = (14)



29

Next, a formulation of the above rigid body transformation shall be given. Below,
w
F
, l
F
, w
R
, and l
R
will represent world and image coordinates in the floating volume, and
world and image coordinates in the reference volume, respectively. A
i,w,F
and A
i,w,R
will
represent the transformations from image to world coordinates for the floating and
reference volumes, respectively. The matrix A represents the transformation between
two volumes, as a function of the registration parameters. A
FR
will represent the
transformation between the floating and reference volume, in the image coordinate
system of each volume.
The below formation is a simple derivation based on eqs. 6, 7, 8, and 14.
Essentially, by taking the transformation between world coordinates, given by eq. 14, and
the relationships between image and world coordinates given by eqs. 6 through 8, eq. 15
below can be written.


( )
F FR R
F F w i R w i R
F F w i R R w i
F R
w T l
w T A T l
w T A l T
w A w
=
=
=
=

, ,
1
, ,
, , , ,

(15)


Therefore, using the results given in eq. 15, computation of the matrix T
FR
allows
only coordinates in the image to be dealt with explicitly.


30

D. Volume Interpolation

With the application of T
FR
to convert between image coordinates in the floating
volume to the reference volume, it is highly unlikely that the coordinate l
R
will fall on a
sampled point, and therefore, volume interpolation will be necessary to approximate the
value of the volume at the coordinate l
R
.
Here, trilinear interpolation is used, which is an extension of linear interpolation
in one dimension, and bilinear interpolation in two dimensions, to the three-dimensional
structure of a volume. An approximation of the value of the volume at the coordinate l
R

is given by a linear weighting of the surrounding eight voxels, that form the smallest
bounding cube around l
R
in the lattice.
Trilinear interpolation uses the quantities given in Table I below. As a reference,
the sample closest to the origin in the image coordinate system in the surrounding cube of
eight voxels is selected as a reference. The image coordinates of this voxel are denoted
by (lower_left_N1, lower_left_N2, lower_left_N3). The value of the volume at a
coordinate sample
i
is denoted by V(sample
i
). The interpolated value in the volume V,
denoted by interpolated_value, is given by the sum of the product of weight
i
and
V(sample
i
), for i from 0 to 7, indexing the eight surrounding voxels.


31

TABLE I
SAMPLES AND SAMPLE WEIGHTINGS FOR TRILINEAR INTERPOLATION.

3 _ _
2 _ _
1 _ _
N left lower z delta
N left lower y delta
N left lower x delta
Z
Y
X
=
=
=

Z Y X
Z Y X
Z Y X
Z Y X
Z Y X
Z Y X
Z Y X
Z Y X
delta delta delta weight
delta delta delta weight
delta delta delta weight
delta delta delta weight
delta delta delta weight
delta delta delta weigth
delta delta delta weight
delta delta delta weight
=
=
=
=
=
=
=
=
7
6
5
4
3
2
1
0
) 1 (
) 1 (
) 1 ( ) 1 (
) 1 (
) 1 ( ) 1 (
) 1 ( ) 1 (
) 1 ( ) 1 ( ) 1 (

) 1 3 _ _ , 1 2 _ _ , 1 1 _ _ (
) 1 3 _ _ , 1 2 _ _ , 1 _ _ (
) 1 3 _ _ , 2 _ _ , 1 1 _ _ (
) 1 3 _ _ , 2 _ _ , 1 _ _ (
) 3 _ _ , 1 2 _ _ , 1 1 _ _ (
) 3 _ _ , 1 2 _ _ , 1 _ _ (
) 3 _ _ , 2 _ _ , 1 1 _ _ (
) 3 _ _ , 2 _ _ , 1 _ _ (
7
6
5
4
3
2
1
0
+ + + =
+ + =
+ + =
+ =
+ + =
+ =
+ =
=
N left lower N left lower N left lower sample
N left lower N left lower N left lower sample
N left lower N left lower N left lower sample
N left lower N left lower N left lower sample
N left lower N left lower N left lower sample
N left lower N left lower N left lower sample
N left lower N left lower N left lower sample
N left lower N left lower N left lower sample

interpolated_value =

=

7
0
) (
i
i i
sample V weight


E. Overview of Registration Approaches

Computer-assisted registration has received a large amount of attention in
medical imaging, owing to its importance, and the strong desire for an automated, non-
invasive approach. Consequently, there are a number of different paradigms that have

32

been introduced for 3D volume registration, including landmark-based methods,
segmentation-based methods, surface-based methods, and volumetric methods [18, 19].
In this work, registration is pursued using a technique based on volumetric
registration. The choice of this approach allows for the features used for registration to
be the samples comprising the volumes to be registered directly, with no pre-processing
or feature extraction applied. This technique, known as volume registration by
maximization of mutual information, relies on the evaluation of a metric function to
quantify the quality of alignment, of the floating and reference volumes, given a
registration parameter vector. The metric function used is the mutual information
function of the floating and reference volumes.

F. Registration Metric/Criteria

Relative entropy, also known as the Kullbak Leibler distance, between two
probability mass functions p(x) and q(x), is defined as the quantity D(p||q), given in eq.
16 below [21].

=
X
x q
x p
x p q p D
) (
) (
log ) ( ) || (
2

(16)



33

For two discrete random variables X and Y, with marginal probability mass
functions p
X
(x) and p
Y
(y), and a joint distribution p(x, y), the mutual information
function I(x,y) is the relative entropy between the joint distribution p(x, y), and the
distribution p
X
(x)p
Y
(y), the joint distribution when X and Y are independent random
variables. Thus, the mutual information of X and Y is given in eq. 17 below [17].

= =
Y X Y X
Y X
y p x p
y x p
y x p y p x p y x p D Y X I
,
2
) ( ) (
) , (
log ) , ( )) ( ) ( || ) , ( ( ) , (
(17)


If X and Y are independent random variables, then p(x, y) is given by the product
of the marginals p
X
(x) and p
Y
(y), and therefore, the quantity that is the argument of the
logarithm function in eq. 17 is one. As the logarithm of one is zero, the mutual
information I(X, Y), when X and Y are statistically independent, is zero.
There is a close relationship between entropy, a measure of information content,
and the mutual information quantity. The entropy H(X) of a discrete random variable X
is defined in eq. 18. The joint entropy H(X,Y) of the discrete random variables X and Y
is defined below in eq. 19. The conditional entropy H(Y|X) of the discrete random
variables X and Y is defined below in eq. 20 [21].



34

=
X
x p x p X H ) ( log ) ( ) (
2


(18)

=
Y X
y x p y x p Y X H
,
2
) , ( log ) , ( ) , (

(19)

=
Y X
x y p y x p X Y H
,
2
) | ( log ) , ( ) | (
(20)


Entropy is a common measurement of information content. Information content
is increased as entropy is increased, and decreased as entropy is decreased. The less
concentrated a probability density or mass function is, the more information content that
is encoded in the random variable. For example, consider a continuous random variable
X. If X is distributed as a uniform random variable, the information content of X is
greatest, because over the range of X, the probability of X taking on a value x
1
is equal in
all cases to X taking on a value of x
2
. For X distributed as a Gaussian random variable,
with a mean and variance
2
, X encodes less information, as the values of X around
are more likely than values far from , with the spread or concentration of probability
around quantified by the variance
2
.
By simple algebraic manipulation, and using the definition of conditional
probabilities, the mutual information function I(X,Y) can be related in a number of ways
to the entropy quantities given in eqs. 18 through 20. These relationships are given
below in eqs. 21 and 22.



35

) | ( ) ( ) | ( ) ( ) , ( X Y H Y H Y X H X H Y X I = =

(21)
) , ( ) ( ) ( ) , ( Y X H Y H X H Y X I + = (22)


For volumetric registration, there will be two volumes, a floating volume, and a
reference volume. The floating volume will be transformed by a transformation T

, a
transformation with registration parameters . Using the intensities of the volumes as the
features to use for registration, the random variable R shall denote the intensities
observed in the reference volume, the random variable F shall denote the intensities
observed in the floating volume, and the random variable T

F shall denote the intensities

observed in the transformed floating volume, given the registration parameters .
The mutual information function, I(X, Y), will be used as a measure of alignment,
by consideration of I(R, T

F). Therefore, the registration problem is then to maximize

I(R, T

F), by manipulation of the registration parameters .

The metric I(R, T

F), from the definition of mutual information in eq. 17, and the
relationships of mutual information to entropy in eqs. 21 and 22, can be written as below,
in eqs. 23 through 25, respectively.


) ( ) (
) , (
log ) , ( ) , (
2
,
f T p r p
f T r p
f T r p F T R I
f T R F T R

=



(23)
) | ( ) ( ) , ( F T R H R H F T R I

=

(24)

36

) , ( ) ( ) ( ) , ( F T R H F T H R H F T R I

+ = (25)


In the process of registration, only the overlapping portions of the reference and
transformed floating volumes will be considered in computation of the mutual
information metric, as will be discussed below. Therefore, the quantities H(R) and
H(T

F) change little, over the range of registration parameters considered.

Considering the relationship between the mutual information metric and entropy
given in eq. 24, then, it can be observed that maximization of I(R, T

F) is equivalent to
minimization of H(R|T

F). Thus, by minimization of H(R|T

F), given T

F, the
information content of R is to be minimized. Qualitatively, if T

F is known, then the

information measure H(R|T

F) is low, as minimization of I(R, T

F) has established a
relationship between the observations of R, and the observations of T

F, regardless of the
mathematical nature of the relationship. This property allows the mutual information
metric to be useful for multimodal registration, where other similarity metrics perform
poorly [22].
Considering the relationship between the mutual information metric and entropy
given in eq. 25, it can be observed that maximization of I(R, T

F) is equivalent to
minimization of H(R, T

F). Thus, by minimization of H(R, T

F), the information content

of (R, T

F) is minimized. In terms of the joint probability mass function p(r, T

f), this
corresponds to building concentrations of probability, as observed in a comparison
information content of a random variable distributed as a Gaussian to a random variable
distributed uniformly. This process allows for registration by favoring registration

37

parameters that establish a relationship between the reference and floating volumes, with
this relationship being the spatial alignment of anatomy, implicitly.

G. Computation of the Mutual Information Metric

Computation of the mutual information metric is based upon direct evaluation of
eq. 23. Furthermore, this computation is improved in terms of execution speed on the
Onyx2 supercomputer by dividing the task of computation of I(R, T

F) across several
processors. Computation of the metric, and parallelization of this computation is
discussed below.
Beginning with computation of the metric, from eq. 23, three quantities are
necessary: p(r, T

f), p
R
(r), and p
TF
(T

f). The marginals p

R
(r) and p
TF
(T

f) may be
obtained directly from the joint probability function p(r, T

f). The joint probability mass

function p(r, T

f) will be approximated by the normalized joint histogram h(r, T

f). Here,
normalization refers to scaling of the histogram, such that the sum of approximated
probabilities equals 1.0. The marginals are then approximated from h(r, T

f) by
summation over the rows of h(r, T

f), and then the columns.

Computation of h(r, T

f) involves a complete iteration over each sample in the

floating volume. For each sample, the transformation T

is applied, to arrive at a
coordinate set in the image coordinate system of the reference volume. If the
transformed coordinate is outside the measured reference volume, then the remaining
operations are not executed, and the process starts again with the next sample in the
floating volume. Otherwise, a sample in the reference volume at the transformed

38

coordinates is approximated using trilinear interpolation, and discretized. The two
samples, one from the floating volume, and one from the reference volume, are then
binned in the joint histogram.
In the studies considered here, each sample obtained from the MRI is an 8-bit
sample, allowing for 256 discrete levels. In the computation of the joint histogram
h(r,T

f), each discrete level is utilized in the joint and marginal histograms. Therefore,
there are 256 x 256 = 65,536 bins in the joint histogram, and 256 each in the marginal
histograms.
Computation of the joint histogram involves the processing of each sample in the
floating volume, application of a transformation to the coordinate of the sample in the
floating volume to obtain a coordinate in the reference volume, interpolation in the
reference volume, and binning in the joint histogram. For a typical 256 x 256 x 20 MRI
volume, there are thus 256 x 256 x 20 = 1,310,720 samples to process.
Following computation of the joint histogram, normalization and computation of
the marginal histograms must be performed. This involves one pass over the joint
histogram, therefore processing 256 x 256 = 65,536 elements from the joint histogram.
This processing consists of normalization, and summation to compute the marginal
histograms.
Following this operation, the mutual information metric itself may be computed.
This processing involves computation of the sum given eq. 23, and involves one pass
over the joint histogram, therefore processing 256 x 256 = 65,536 elements that compose
the sum given in eq. 23.

39

Therefore, computation of the joint histogram is by far the most computationally
costly component in computation of the mutual information metric. Therefore,
performance may be best increased by decreasing the execution time of the computation
of the joint histogram. Computation of the joint histogram is an amenable problem for
parallel execution, as computation of a part of the joint histogram does not depend on the
computational results of any other part of the joint histogram, allowing individual bins, or
entire regions of the joint histogram to be computed independently, and then merged to
form the total joint histogram.
The architecture of the SGI Onyx2 supercomputer allows for tasks running on
different processors to access memory anywhere within the 20 Gb of shared memory in
the machine. Reads from an arbitrary location in memory, then, are unfettered. Writes to
memory, however, should be carefully planned, so as to avoid potential problems with
performance decreases, due to cache refreshes. Therefore, a desired parallel solution
should permit reading of common input data from only one location (to avoid
unnecessary duplication and other overhead), but also keep outputs of the computation
local to a process.
For implementation of parallel execution, POSIX pthreads are used. The
implementation on the SGI Onyx2 allows for full utilization of the shared memory
capabilities of the machine. Furthermore, the use of POSIX pthreads allows for the
developed application to be easily ported to other architectures and operating systems,
such as a personal computer (PC) running a variant of the Linux operating system.

40

A solution to this problem, fitting within the above constraints, is a division of the
computation of the joint histogram into two tasks. The first task computes a number of
joint histograms over sub-volumes. The second task merges these sub-joint histograms to
form the total joint histogram. Below, figure 10 illustrates the architecture of the
solution.

41


F
I
G
U
R
E

1
0
:

I
l
l
u
s
t
r
a
t
i
o
n

o
f

t
h
e

p
a
r
a
l
l
e
l

c
o
m
p
u
t
a
t
i
o
n

o
f

t
h
e

j
o
i
n
t

h
i
s
t
o
g
r
a
m

n
e
c
e
s
s
a
r
y

f
o
r

c
o
m
p
u
t
a
t
i
o
n

o
f

t
h
e

m
u
t
u
a
l

i
n
f
o
r
m
a
t
i
o
n

m
e
t
r
i
c
.


42


Given a division of the floating volume into a set of sub-volumes, each process or
thread in task 1 computes a joint histogram over a sub-volume. Subsequent tasks cannot
proceed until all of the task 1 threads are finished in their execution, therefore, load
balancing between threads is accomplished by parameterizing the number of slices from
the floating volume to use per thread, and then by dynamic computation of the number of
threads that are necessary to execute. Each sub-joint histogram is computed as the joint
histogram is computed, as discussed above. The outputs of task 1 threads are a sub-joint
histogram corresponding to each sub-volume.
Given a set of sub-joint histograms, threads in task 2 compute the total joint
histogram over a region of the joint histogram. Each thread in task 2 has an assigned
region over which to compute the joint histogram. Given an element in the total joint
histogram to compute, the quantity is computed by summation over all sub-joint
histograms of the corresponding element. Load balancing between threads is
accomplished by dividing the total joint histogram into regions of equal numbers of
elements, and assigning a thread to compute a single region. The output of task 2 threads
is the total joint histogram.
Therefore, given a set of registration parameters, the mutual information metric
I(R,T

F) may be efficiently computed. As will be shown, this parallelization allows for

registration to be performed in an amount of time that is reasonable for the needs of the
proposed use as part of a study of the effectiveness of a multiple sclerosis treatment.


43

H. Search Algorithm

The mutual information metric provides a quantitative measure of spatial
alignment between two volumes, given a choice of registration parameters. To obtain the
best alignment, it is necessary to maximize the metric. Maximization of the metric,
which is parameterized in terms of the registration parameters, is numerically
accomplished with the use of a search or maximization algorithm.
In the original formulation of registration by maximization of mutual information
in [14], Powells multidimensional optimization method, with Brent line minimizations
was used for maximization of the mutual information metric [23]. Subsequently, [24]
compares different classical optimization methods maximizing the mutual information
metric. One such method included in the study was the use of the classic Nelder and
Mead or simplex algorithm for maximization.
This method solely uses the objective function directly for optimization, and
therefore does not require the expensive computation of derivatives. This method is a
geometry-based method, using the geometric operations of contraction, expansion, and
reflection to manipulate a simplex to a maximum of the objective function. This method
was found to perform well in [24], and is utilized here for maximization of the mutual
information metric. The formulation implemented is derived from the presentation of the
algorithm in [23] and [25].

44

I. Implementation

The implemented software for volume registration allows for volume registration
of two volumes with patient axes aligned a priori, over six registration parameters, which
include three translation quantities, and three rotation quantities, for 3D rigid-body
registration. The developed software tool utilizes parallel programming techniques to
obtain reasonable execution times. Output from the program includes the registration
parameters, and optionally, a re-sampled reference volume that is re-sampled along the
imaging planes of the floating volume.
The software was developed and tested on the SGI Onyx2 supercomputer, and
utilizes the parallel processing capabilities of this machine. The software, however, may
be ported to other architectures and operating systems with a trivial amount of effort, as
only standard, non-proprietary programming languages and libraries are used.

J. Results: MS studies

To assess the performance and accuracy of the implementation, the registration of
sample MS studies conducted at Jewish Hospital was performed. Data from five patients
(A, B, C, D, and E), and 7 studies were used. Table II below characterized the data sets
used, including specifications (imaging plane, weighting, dimensions, and voxel sizes) of
the reference and floating MRI studies.


45

TABLE II
PARAMETERS OF THE MS STUDIES USED FOR ASSESSING THE REGISTRATION
SOFTWARE DEVELOPED.

Reference Study
study patient
date
img.
plane
weighting dimensions voxel sizes (mm)
1 A 8/22/2001 axial FLAIR 256 x 256 x 21 0.937503 x 0.937500 x 5.0
2 B 7/27/2001 axial FLAIR 256 x 256 x 20 0.9375 x 0.9375 x 5.0
3 C 11/7/2000 axial FLAIR 256 x 256 x 20 0.9375 x 0.9375 x 5.0
4 A 8/22/2001 coronal T1 256 x 256 x 25 0.78125 x 0.78125 x 5.0
5 C 11/7/2000 sagittal T1 256 x 256 x 12 0.9375 x 0.9375 x 5.0
6 D 10/13/2001 axial FLAIR 256 x 256 x 21 0.937494 x 0.9375 x 5.0
7 E 3/27/1999 axial FLAIR 256 x 256 x 21 0.93749 x 0.9375 x 5.0
Floating Study

date
img.
plane
weighting dimensions voxel sizes (mm)
1 A 9/12/2000 axial FLAIR 256 x 256 x 20 0.9375 x 0.9375 x 5.0
2 B 1/29/2001 axial FLAIR 256 x 256 x 21 0.9375 x 0.9375 x 5.0
3 C 4/8/2000 axial FLAIR 256 x 256 x 20 0.9375 x 0.9375 x 5.0
4 A 9/12/2000 coronal T1 256 x 256 x 23 0.859375 x 0.859375 x 5.0
5 C 4/8/2000 sagittal T1 256 x 256 x 12 0.9375 x 0.9375 x 5.0
6 D 6/9/2000 axial FLAIR 256 x 256 x 20 0.9375 x 0.9375 x 5.0
7 E 11/3/2001 axial FLAIR 256 x 256 x 20 0.937496 x 0.9375 x 5.0


Table III below shows the registration parameters obtained, as well as the
execution time of the registration application. The registration parameters shown are the
six parameters comprising the parameter vector, including the three translation quantities
(t
x
, t
y
, and t
z
), and the three rotation angles (
x
,
y
, and
z
). The execution time shown
includes the entire execution time of the program, including initialization, the search for
the optimal registration parameters, and generation of a re-sampled output volume.

46

TABLE III
EXECUTION TIME AND REGISTRATION PARAMETERS FOR MS STUDIES USED.

registration parameters
study patient translations (x,y,z)
(mm)
rotations (x,y,z)
(degrees)
execution time (sec)
1 A
-0.447622
0.591327
-0.219799
4.197726
1.836846
-5.276162
47
2 B
0.170340
-0.009501
4.054677
5.106087
-4.828026
1.184937
59
3 C
-2.446951
-1.918487
3.016138
-0.168511
0.021976
-1.697560
58
4 A
-0.970308
1.244239
0.351349
2.725014
-5.633704
0.460878
57
5 C
2.759514
9.669111
-2.554870
2.449247
1.219536
3.367644
60
6 D
2.759514
9.669111
-2.554870
2.449247
1.219536
3.367644
60
7 E
-4.095278
2.683078
3.758586
-0.873155
5.486783
6.106783
62

The results obtained were submitted to Dr. Robert Falk of Jewish Hospital for
evaluation of the accuracy of the registration. Based on his judgment as a radiologist, the
registration approach was found to work well, with the exception of one study. Judgment
of accuracy was based on the application Dr. Falks training as a radiologist, and
experience in medical imaging to qualitatively evaluate how closely the slices from the
re-sampled reference volume correspond to the anatomy imaged in the slices of the
floating volume. Study 2 was judged to be objectionable to a degree on the basis of this
criterion.
It is expected that decreased slice thickness will assist greatly in the reduction of
registration error, due to better resolution of the study. Complete head studies with a

47

smaller slice thickness of MS patients were not available for use (the studies here all have
a slice thickness of 5.0 mm).
Validation of the registration results is a difficult problem. Direct evaluation of
the error in the registration parameters computed is not possible, as the actual registration
parameters are unknown. One possible way to quantify error is to have an expert
manually select corresponding points in the reference and floating volumes, and then to
calculate the registration between the selected, corresponding points. These registration
parameters would then be used as the actual registration parameters that may be
compared to the computed registration parameters for quantification of registration error.
This approach would introduce error in the manually selected points, however, nor was a
data set of corresponding points available for use here.
Figure 11 below shows sample slices from the registration studies. Each row of
the figure corresponds to a single study. The first column of the figure shows a sample
slice from the floating volume used for each study. The second column of the figure
shows the corresponding slice from the re-sampled reference volume. Finally, the third
column of the figure shows the checkerboard composite image formed by fusion of the
corresponding floating and re-sampled reference volume slices.


48


(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

49


(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)
FIGURE 11: Sample registration results from each of the seven MS data sets. The first
column(a, d, g, j, m, p, and s) is a sample slice from the floating volume used. The
second column (b, e, h, k, n, q, and t) is the corresponding slice from the re-sampled
reference volume. The third column (c, f, i, l, o, r, and u) is the checkerboard composite
image of the two corresponding slices from the floating and re-sampled reference
volumes. The floating volume and reference volumes used in each trial were from the
same patient.



50

K. Summary

The implemented registration method works well, on the basis of qualitative
evaluation of the results by a radiologist, and the efficiency of the software application in
terms of execution time. Additionally, the use of parallel programming techniques allows
for the method to be applied in a feasible period of time to be used in the context of the
proposed application in MS studies. The developed software tool is easy to use, and
easily modifiable for future enhancements.
In terms of performance, of the seven studies evaluated, one was found to be
objectionable to an expert radiologist.


51

III. BRAIN SEGMENTATION FROM THE HEAD



A. Introduction and Necessity

Brain segmentation from the head refers to the process of extracting brain tissue
from the remaining tissues found in the head. Extracting the brain, including normal and
diseased tissue, as well as CSF, from the remaining tissues comprising the head, allows
for a simplified classifier to be applied for subsequent quantitative analysis of MS
lesions.
A Bayesian classifier is utilized for classification of normal and abnormal brain
tissue, and CSF. To this end, it is desirable to minimize the number of tissues that must
be modeled. The number of tissues that must be modeled, at minimum, is three: normal
brain tissue, abnormal brain tissue (MS lesions), and CSF. By removing other tissues,
such as eyes, skin, and fat, which are also imaged in an MRI study of the head, the
subsequent statistical classification can be simplified to the minimum number of tissues
that must be modeled. Removal of these extraneous tissue classes is accomplished here
with the use of a registration-based technique, to register a scan to a scan with a known
head segmentation.
Over several studies of an MS patient, it is not expected that the volume of the
brain will change appreciably. Changes in brain volume over time will be assumed to be
only due to brain atrophy, a reasonable assumption owing to the pathological nature of
MS [1,2,4]. As well, it is expected that if the obtained brain segmentation is slightly

52

larger than the actual brain, this is acceptable, as CSF surrounds the brain, and a model
for CSF must be considered in any case.
Brain segmentation from the head, also known as brain peeling, has been
addressed by a number of research centers [26, 27, 28, 29]. These methods each claim at
least a degree of automation, and others, outright full automation. Reviewing these
studies, however, indicates that there is a large amount of data dependency of the method,
owing to choice of scan parameters. As the data sets here are simply provided, and were
acquired with scan parameters that benefited the radiologist, implementation of a
technique based on these studies is not pursued.
As it is assumed that a patients brain volume does not change dramatically, and
is limited to perhaps only brain atrophy, an approach to the segmentation of the brain
from the head is explored that is based upon registration, and utilizes the registration tool
developed during the course of this research.

B. Brain Segmentation Utilizing Registration

The underlying concept of registration-aided brain segmentation is that for a
patient, if an a priori segmentation is known, then registration of an arbitrary study
allows for the brain segmentation problem to be solved. This solution is trivial, as the
segmentation for the arbitrary volume is known from alignment of the study to the study
for which the segmentation is known. The known segmentation may then be applied to
the arbitrary study, achieving brain segmentation.

53

For implementation of this approach, a rigid registration software tool, such as
developed for this study, must be available. Also, it is necessary to be able to generate a
brain segmentation from a patient, to serve as the a priori segmentation. This
segmentation may generally be manual, semi-automatic, or fully automatic. Here,
manual and semi-automatic methods [30] are explored for generation of the a priori
segmentation.
While undesirable in principle due to the need for human involvement, this
approach is acceptable for low numbers of patients, as only an initial brain segmentation
is necessary. As well, other factors related to human involvement, such as accuracy and
repeatability, are mitigated, as segmentation of the brain from the head is a gross
anatomical-recognition task. It is also acceptable if the segmentation includes additional
CSF surrounding the brain, and small errors in removing brain tissue should not be
appreciable with respect to the total volume of the brain, and given the spatial sampling
rates of the studies investigated.
A related approach would be to replace the a priori segmentation with a non-
patient specific head model, incorporating a known brain segmentation with the model.
Then, using non-rigid registration, an arbitrary patient study would be registered to the
model to provide for brain segmentation of the patient study. In this situation, the model
replaces the a priori patient segmentation of the above approach.
For implementation of an approach based upon a head model, a general head
model must be known, and available for use. Such a model would likely be generated
from an actual patient study, or perhaps multiple studies, and manually segmented by an

54

expert (such as a radiologist, or other professional knowledgeable with regard to head
anatomy). As well, it may be useful to introduce some degree of generality to this model,
such as outlining the brain segmentation to a contour, eliminating patient-specific
anatomy, such as structural formations (i.e. folds) in the brain. Unfortunately, in the
present study, a head model is not available for use, and this variation will not be further
explored.
Computationally, this approach is very efficient. Registration of subsequent
patient studies to a standard is necessary, for reasons beyond brain segmentation.
Therefore, alignment of an arbitrary study to an a priori study incurs no additional
computational cost, as the study for which an a priori segmentation is known can simply
be the standard study to which all patient studies are aligned. Application of the known
segmentation to the arbitrary study is trivial, and involves only the masking of a volume,
which can be viewed as a multiplication of each voxel in a volume, by either zero to
remove the voxel, or one to include the voxel in the segmentation. Therefore, the
marginal cost of this approach for head segmentation is minimal.
For generation of the a priori segmentation, manual and semi-automatic
approaches are considered. From the available MS studies, a patient was selected for
which several studies are available. A manual segmentation of the brain was performed
on one of the studies. This involved removing all non-brain tissue and background from
the study, slice by slice, using pre-existing, image-editing software. A non-expert manual
segmentation, performed by the author, is utilized in the study presented here.

55

A second approach explored for brain segmentation involves the use of a
technique for semi-automatic segmentation of the brain from the head. This technique
involves the implementation and extension of the LEGIONs segmentation algorithm [31]
in the CVIP Lab by Mr. Chuck Sites [30]. This technique implements unsupervised
segmentation, with labels applied to similar regions of a volume. Following this
segmentation, labels not of interest are manually discarded, and like tissues with different
labels are manually re-labeled to have a common label. This approach may be viewed as
an aide for expert manual segmentation, by providing an initial solution.
Prior to application of the segmentation tool, image enhancement and noise
filtering is implemented with the use of a class of filters known as non-linear anisotropic
diffusion filters [32]. This useful filter class smooths image content of similar
appearance, minimizing the effects of noise, while preserving edges in the image. The
filter is applied slice-by-slice, and utilizes a parallel implementation for use on the SGI
Onyx2 supercomputer.

C. Results: Manual a priori Segmentation

Below are samples of the results obtained, when an initial a priori segmentation
of a patient is obtained manually. Figure 12 shows sample slices from the study used for
generation of the a priori mask by manual segmentation, the segmented brain, and the
respective binary mask. Figure 13 shows the registration of the a priori study to a
subsequent study of the same patient. Finally, figure 14 shows the results of application
of a binary mask applied to the second study. The binary mask was obtained using the

56

binary mask of the a priori scan, and the registration parameters discovered in the
registration process.


(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)
FIGURE 12: Samples of the (non-expert) manual segmentation of a patients brain
from the remaining tissue of the head. The slices obtained from the MRI study ((a), (b),
and (c)) are manually segmented to obtain slices containing the brain ((d), (e) and (f),
respectively). The manually segmented images are then made into binary masks ((g),
(h), and (i), respectively).


57


(a)

(b)

(c)

(d)

(e)

(f)
FIGURE 13: Registration of two MRI studies of the same patient, taken at different
times. The first study ((a), (b), and (c)) was manually segmented to extract the brain
from the head. The second study ((d), (e), and (f)) was registered to the first (the re-
sampled slices are shown).


58



(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)
FIGURE 14: Segmentation of the second study by a binary mask calculated from the
registration parameters, and the binary mask used to segment the a priori study. The
raw input slices are shown ((a), (b), and (c)). The binary mask was obtained by use of
the registration parameters and the a priori mask ((d), (e) and (f), respectively). By
application of the mask, the segmented volume is obtained ((g), (h), and (i), respectively).


The accuracy of the resulting segmentation depends on the accuracy of the
registration, and the accuracy of the a priori segmentation. The registration results were
found to be acceptable to an expert (as evaluated in section II). The a priori
segmentation is assumed to be accurate on the basis of a manual segmentation. Despite

59

this argument, further work is necessary to numerically quantify the accuracy of the
segmentation.

D. Results: Semi-Automatic a priori Segmentation


Below are results obtained, when an initial a priori segmentation of a patient is
obtained by utilization of the segmentation tool, followed by manual re-labeling and
extraction of labels of interest. Figure 15 below shows sample results from the semi-
automatic segmentation method. The method is applied, and manual re-labeling is used
to identify the brain from the labels obtained in the segmentation. This re-labeling results
in a binary mask which can be used for extracting the brain from the head; the binary
mask obtained becomes the a priori brain mask. Figure 16 below shows sample results
from the registration of a second study to the a priori study, and application of the brain
mask obtained from the registration parameters and the a priori brain mask.

60



(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)
FIGURE 15: Sample brain segmentation results of the a priori scan using a semi-
automatic technique improved upon in the CVIP Lab. The input slices ((a), (b), and (c))
are semi-automatically segmented to form a binary mask of the brain ((d), (e), and (f),
respectively). The mask is then applied to form the segmented volume ((g), (h), and (i),
respectively).


61



(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)
FIGURE 16: Brain segmentation of the second study. The second study ((a), (b), and
(c)) was registered to the a priori scan. The brain mask for the second study ((d), (e),
and (f), respectively) is then obtained using the a priori brain mask, and the registration
parameters. The binary brain mask is then applied to generate the segmented volume
((g), (h), and (i), respectively).

As can be observed from the results of the semi-automatic method, there are
errors that prevent this approach from further use. For example, an MS lesion in the slice

62

in figure 12 (c) is removed from the segmentation of the brain. Additionally, isolated
islands are present, as shown by the segmentation of the slice in figure 12 (a). Some
brain material is lost in the segmentation, as shown by the segmentation of the slice in
figure 12 (b).

E. Summary

Brain segmentation, the process of extracting the brain from the head, is a
necessary pre-processing step to classification of brain tissue in an MRI study of an MS
patient. To this end, segmentation via registration of a study to a study with a known
brain segmentation was explored, with useful results generated. The implemented
approach incurs a minimal amount of computational overhead, given that registration of
subsequent patient studies will be performed regardless of use in brain segmentation.
The manual procedure for generation of an a priori brain mask works well, but is
too demanding of human intervention to be useful in a clinical study of numerous
patients. The semi-automatic procedure, at present, does not generate accurate enough
brain segmentations to be used in a clinical study. Despite this, the segmentation may be
useful as an initial solution for manual segmentation.


63

IV. TISSUE SEGMENTATION



A. Introduction and Necessity

Following segmentation of the brain from the remaining tissues imaged in the
head, tissue classification addresses the task of classifying tissues within this volume of
interest. States of nature, or classes, considered here include normal brain tissue (white
and gray matter), diseased or abnormal brain tissue (MS lesions), and CSF. From the
results of brain segmentation above, the operation of classification assigns a single class
to each voxel of this volume.
Resulting from this assignment of class, quantitative measures of MS disease
burden may be computed by computation of the total volume of MS lesions, and the total
volume of normal brain tissue. An increase in the volume of MS lesions is assumed to be
indicative of worsening of the disease. Decreases in the volume of normal brain material,
owing to brain atrophy and conversion of normal brain tissue to diseased tissue, are also
assumed to be indicative of worsening of the disease.
Several classification/segmentation schemes were experimented with over the
course of study of this problem, beginning with the very simple, and advancing to more
complicated, but necessary techniques to achieve good results.
A statistical classification approach, using the Bayes decision rule, is ultimately
utilized in the approach described here. Class conditional probabilities are estimated
nonparametrically, using Parzen windowing. A priori class probabilities are modeled

64

using a Markov random field model. The results obtained demonstrate the effectiveness
of this approach.

B. Feature Selection

Generally obtained in the MS studies available for use, are T1, T2, and FLAIR
weighted MRI studies. The weighted studies are acquired simultaneously, and therefore,
no inter-study registration is necessary. Each weighting contrasts tissues with respect to
other tissues in the head differently. For example, with a T1 or FLAIR weighting, CSF is
dark relative to surrounding brain tissue, while in a T2 weighted study, CSF is bright
relative to surrounding brain tissue.
From a tissue classification perspective, different weightings provide for
additional raw features that may be useful in achieving improved classification. The
basis of this statement is that the use of additional features may help to provide
discriminatory power in feature space, such that there is reduced error in the resulting
classification by separation of feature measurements of tissues in the feature space. For
example, considering two tissue classes, and only one feature, there may be significant
overlap in feature space between the class conditional probability distributions of the
feature for the different classes. The inclusion of a second useful feature may allow for
improvement in the probability of error of the classifier, by allowing for improved
separation of tissues in feature space. With too many features, however, training
becomes difficult, the computation time of the classifier generally increases, and
performance of the classifier may actually decrease [33].

65

Hyper-intense regions in the FLAIR studies are the desired regions to be
classified as MS lesions. In consideration of the additional weightings available (the T1
and T2 studies), it is observed that lesion appearance across the FLAIR, T1, and T2
studies is not consistent, as shown in figure 17. Comparing the appearance of the lesions
in the FLAIR image to T1 image, observing abnormal tissue in the T1 image is difficult
at best, and as a result, the T1 images are not useful for discrimination of lesions against
brain material. Comparing the appearance of the lesions in the FLAIR image to the T2
image, it is observed that the T2 image shows limited contrast of lesions against brain
material for some lesions. Furthermore, the diseased tissue imaged in the FLAIR image
is not always contrasted as abnormal tissue in the T2 image, and at other times, the
diseased material found in the T2 image appears quite differently depending on the
lesion.
Therefore, as the FLAIR image gives the best indications of disease, and because
the T1 and T2 images are either not useful (T1), or not consistent with the FLAIR
imagery (T2), the FLAIR imagery alone will be used for classification. The features that
will be used in classification will be the intensity observed in the FLAIR imagery.

66



(a)
TR = 12631 ms, TE = 96 ms

(b)
TR = 450, TE = 14 ms

(c)
TR = 7058 ms, TE = 100 ms

(d)
TR = 6000 ms, TE = 99 ms

(e)
TR = 450 ms, TE = 14 ms

(f)
TR = 3577 ms, TE = 114 ms
FIGURE 17: A sample slice from an MRI study of an MS patient, with FLAIR (a and d),
T1 (b and e), and T2 (c and f) weightings. The appearance of lesions is not consistent
between different weightings. MS lesions are easily identified in the FLAIR images as
hyperintense regions in the brain.

It should be noted that differences in lesion appearance across different study
weightings may be attributed to different stages of the disease on a per-lesion basis. This
information, however, is not incorporated in the classification scheme developed in this
research.
Gross observation of the FLAIR imagery, following segmentation of the brain
from the head, shows that there are three dominant intensity levels, corresponding with

67

the tissue types desired to segment. CSF appears as the least intense tissue class. MS
lesions appear as hyperintense, or bright regions. Between these two general intensity
levels, lies the intensity of normal brain material.

C. Thresholding

A simple, elementary method for segmentation of lesions, brain tissue, and CSF is
via thresholding of FLAIR intensities. This segmentation scheme relies upon the
selection of a range of intensity levels for each tissue class. These intensity ranges are
exclusive to a single class, and span the dynamic range of the image. Subsequently, a
feature is classified by selecting the class in which the value of the feature falls within the
range of feature values of the class.
Figure 18 below shows a sample FLAIR slice from an MRI study of an MS
patient, and a classified image resulting from segmentation by thresholding. Figure 19
below shows the histogram of the input slice. From the histogram, intensities having a
value from 0 to 109 are classified as CSF, from 110 to 219 as normal brain material, and
from 220 to 255, as MS lesions. The choice of these threshold levels is highly subjective.
For example, the threshold between CSF and normal brain material could be chosen to be
106, 109, 110, 120, or any other intensity around these levels. Selection of the threshold
generally depends on visual identification of a peak in the histogram corresponding to a
tissue class, and selection of a range of intensities around the peak for inclusion in the
range belonging to the tissue class. As well, visual inspection of the imagery reveals the

68

expected number of tissue classes present, and the selection of the thresholds is
influenced by this observation.


(a)

(b)
FIGURE 18: Segmentation by thresholding. The input slice (a) is classified into three
classes, based on selection of class on the interval in which an intensity falls.
Corresponding labeled image slice is shown in (b), with the dark gray indicating CSF,
the light gray indicating normal brain material, and the red indicating MS lesions.


FIGURE 19: The histogram of the input slice shown in figure 18 (a). Intervals
containing a tissue class can be visually identified by a human. Peaking of the histogram
at low intensities represents CSF, and at high intensities, MS lesions. Peaking of the
histogram in the intermediate intensities indicates normal brain tissue.

Although this technique is very intuitive, it is demanding on an expert to select
intensity ranges for each class, and for each slice. This process is time consuming,
subjective in nature, and has low repeatability.

69


D. Segmentation by Image Enhancement

A second image-processing approach to classification of the brain is segmentation
by image enhancement. In image processing terminology, an operation for image
enhancement improves the quality of the image in a particular manner, either
subjectively, or objectively. In the context of segmentation, an MRI slice of the brain
may be viewed as an image upon which enhancement may be performed. This model
assumes that a tissue class ideally has a single intensity, and that noise and scanning
artifacts corrupt this level to produce the distribution of intensities observed for a tissue
class. Thus, by application of image enhancement techniques for reducing noise and
smoothing the image, the enhanced image approximates the ideal image, at least to a
greater extent than the original image.
A class of filters known as nonlinear anisotropic diffusion filters can be useful for
image enhancement [32], as discussed for pre-filtering MRI data before the application of
the head labeling technique previously. As aforementioned, this class of filters performs
smoothing while preserving edges. The smoothing procedure is modeled as a local
diffusion process. For 2D isotropic image filtering, the fundamental equation for an
iteration of the filter is given below in eq. 26, where I(x,y,t) is the intensity of the image
at coordinates (x,y), at a time t, and c(i) is known as a diffusion function [32].



70

| |
) , 1 , ( ) , , (
) , , ( ) , 1 , (
) , , 1 ( ) , , (
) , , ( ) , , 1 (
) ( ) ( ) ( ) ( ) , , ( ) , , (
t y x I t y x I s
t y x I t y x I n
t y x I t y x I w
t y x I t y x I e
s c s n c n w c w e c e t t y x I t t y x I
=
+ =
=
+ =
+ + +

(26)


Eq. 26 incorporates numerical integration to obtain a final enhanced image I. The
time t orders the sequence of progressively enhanced images. The quantity t controls
the time step size, and affects the accuracy of the integration, and hence, the stability of
the filter. A value of 0.2 for the time step was simply selected, and in no cases, resulted
in observed instability of the filter when used with the experiments presented here. An
exponential function is used here for c(i), and is given below in eq. 27, where is a
constant known as the diffusion constant [32].


|
|
.
|

\
|
|
.
|

\
|
=
2
| |
exp ) (

i
i c
(27)


The diffusion constant controls the effect of the filter. The value of 8.0 was
used in the studies here for image enhancement. This value was chosen experimentally,
to give the best results visually for a study. It was observed that a constant value of this
parameter produced consistent results across different studies.

71

The filter operates iteratively, and operation is by selection of a number of
iterations to perform. Increasing the number of iterations results in improved image
enhancement, but results in longer computation times. It was observed experimentally
that 2,000 iterations provides an acceptable tradeoff between these conflicting factors,
and this parameter was used in the experiments presented here.
The filter is applied in an iterative fashion, across each pixel of the input image.
Computation at a time t, for a given pixel, depends only on the present value of the
image. Therefore, the filtering process may be easily parallelized for improved execution
time. Owing to the simple structure of the filter, this is accomplished on the SGI Onyx2
supercomputer using the MIPS compiler to automatically parallelize the execution of the
filter.
Figure 20 below gives two input FLAIR slices from two different MS patients.
The above filter was applied, prior to manual segmentation of the brain from the head.
The results of application of this filter to these input images are also given in figure 20.

72



(a)

(b)

(c)

(d)
FIGURE 20: Non-linear anisotropic diffusion filtering of MRI slices for segmentation.
Input slices (a and c) are FLAIR weighted imagery. The corresponding output slices (b
and d, respectively) show much enhanced images, but with the loss of some MS lesions.

Unfortunately, while promising, the results introduce several difficulties. First of
all, lesions that do not have a strong edge on all sides are smoothed into brain material,
which leads to large classification errors when subsequent labeling is applied. Secondly,
large numbers of iterations of the filter are required before regions of similar tissue have
nearly the same intensity. Even at this point, some type of thresholding is necessary, and
incurs similar difficulties as discussed above. Lastly, the resulting images do not entirely
contain the idealized three similar intensities. This is best illustrated by the input slice in
figure 20 (a), and the output slice in figure 20 (b), where intermediate intensity levels are

73

obtained, corresponding to blurred regions and scanning artifacts in the input slice.
Figure 21 below illustrates some of the types of errors encountered.



FIGURE 21: Illustration of types of errors encountered with segmentation via image
enhancement.


E. Segmentation by Unsupervised Clustering

Clustering methods are algorithms that operate on an input data set, grouping data
into clusters based on similarity of the data in like clusters. As such, clustering
algorithms are unsupervised classifiers, assigning states of nature without regard to any a
priori labeled training samples. Clustering algorithms are also useful for data
exploration, allowing a user to discover patterns or similarities in a data set [33].

74

Explored in this research are clustering techniques for unsupervised classification.
A well-known clustering algorithm, k-means clustering, was implemented, and tested on
sample slices of FLAIR weighted MRI slices from MS patients.
The k-means algorithm accepts as input the number of clusters to organize data
within, initial location of cluster centers, and a data set to cluster.
The number of clusters that the algorithm fits the data to, denoted by c, is
specified to the algorithm, and represents a parameter the user desires to experiment with,
or as a parameter expressing the expected or desired number of classes to discern from
the data. With the k-means algorithm proper, there are no conditions upon which c is
decreased or increased. Additionally, there are no conditions upon which data is
excluded or included in consideration of fit to classes; all data provided as input to the
algorithm is classified. A given piece of data, or feature measurement, is assigned
exclusively to one class (as compared to fuzzy k-means clustering, where a degree of
membership is assigned to each data item, for each class).
Initial cluster centers are provided here as random locations in feature space.
There is a cluster center
i
for each cluster, such that i = 1c. Each cluster center is a d-
component column vector, where d is the number of input features in the data set. For
the case of a single feature taken to be the intensity of the FLAIR study, d = 1.
The data set provided to the k-means algorithm consists of N items, with each
item a d-component column vector, denoted as x
i
, i = 1N.

75

The k-means algorithm is an iterative algorithm, assigning a class at each iteration
to each data element x
i
. Iterations of the algorithm cease when there is no change in the
classification solution during an iteration.
Each iteration consists of classifying the data set by comparison of the data set to
the current cluster centers. A data item is assigned to the same class as a cluster center
i

if the Euclidean distance between the data item and
i
is the least distance between the
data item and all cluster centers
i
, i = 1...c. Following class assignment, cluster centers
are updated, by computation of the centroid of the data set classified as the same class.
The k-means algorithm was implemented, and tested with a sample MRI slice of a
patient with MS. It was generally found that the k-means clustering approach found
structure in the data, however, it was not structure that is of interest in the classification
problem of normal brain tissue, diseased brain tissue, and CSF. It was initially expected
that for a slice containing MS lesions, that three clusters may be appropriate, in
consideration of the desired classification into three tissue classes. It was observed,
however, that best results were obtained when a larger number of clusters were permitted,
followed by manual re-labeling of clusters of like tissue classes.
Figure 22 below illustrates results of application of the k-means clustering
algorithm, using five classes. The input image in (a) is a FLAIR image. The k-means
algorithm is executed, using five clusters, to obtain the output, classified slice in (b). (c)
shows the results after manual re-labeling of class, such that tissues of the same type,
split into two or more clusters, are re-labeled to a single class.

76



(a)

(b)

(c)
FIGURE 22: Results of k-means clustering, using 5 clusters. The input slice is shown in
(a). The resulting clustered output image is shown in (b). By manual re-labeling of
clusters such that clusters of similar tissue belong to the same cluster, the output slice in
(c) is obtained.

Due to the observation that the number of clusters used is not necessarily related
to the number of classes desired in the study, and thus, the need for manual intervention,
classification via the use of a clustering algorithm such as k-means is not found to be an
acceptable means of segmentation of lesions from brain tissue. It should also be noted
that it is very likely that even though each patient will be known to have MS, individual
slices from the brain, taken individually, may not show evidence of MS. Therefore, the
number of clusters used would likely need to be adjusted based on if the slice contains
MS lesions, which requires further involvement by an expert.
It is observed, however, that a clustering algorithm, such as k-means, in
combination with the nonlinear anisotropic diffusion filter, may prove to be useful for
generation of initial training data for more complex classification paradigms.


77

F. Bayesian Classification

The above approaches, while generating tantalizing results, all fail with regard to
accuracy, ease of application, and logical relation to the classification problem. A
statistical classification approach addressees these concerns, providing an optimal
solution to the classification problem when tissue models are known or well
approximated. Additionally, statistical classification is well founded in probability
theory, and has enjoyed success in a number of applications, including classification of
remote sensing data, scene classification, and classification of medical imagery.
Statistical classification is fundamentally based upon the application of Bayes rule
for quantifying a decision procedure. Bayes rule, given in eq. 28 below, specifies how to
compute a posterior probability of a class, given feature measurements [33].


) (
) ( ) | (
) | (
x p
P x p
x P
j j
j

=
(28)


In eq. 28,
j
denotes a class, or state of nature. The vector x denotes a feature
vector, obtained from measurements of quantities of interest. For image analysis, the
feature vector x is composed for each pixel, and consists of the features of the pixel. For
the approach here, the feature x therefore shall consist of one component, the intensity of
the pixel from the FLAIR image. The quantity p(x|
j
) is known as the class conditional

78

probability density function for class
j
. This distribution quantifies the probability of
measurement of a feature, given a state of nature. The quantity P(
j
) is known as the a
priori probability, and quantifies the probability of observing a state of nature, regardless
of any feature measurement. The quantity p(x) is known as the evidence, and serves only
as a scale factor, such that the quantity in eq. 28 is indeed a true probability, with values
between zero and one. The quantity P(
j
|x) is known as the posterior probability, and
quantifies the probability of observing class
j
, given that the feature x was measured.
The quantity p(x) in eq. 28 can normally be ignored for implementing decision
theory, as it is only a constant for establishing a probability between zero and one, and
constant for all classes. Therefore, the quantity given below in eq. 29, known as the
maximum a posteriori (MAP) estimate of eq. 28, is used.


) ( ) | ( ) | (
j j j
P x p x P = (29)


Bayes decision rule, based upon the application of eq. 28 or 29, gives a
framework for how to make a sound decision. For c possible classes, Bayes decision rule
mandates the selection of class
j
for a feature x, if the a posteriori quantities in eq. 28 or
29 are maximum for class
j
, compared to
i
, i = 1c, i j. This rule is stated below in
eq. 30 [33].


79


Decide
j
if P(
j
| x) > P(
i
| x) for i = 1c, i j (30)


Bayes decision rule is optimal in the sense of minimization of the probability of
error [33].
Despite the optimality of the Bayesian classifier, the decision rule is optimal only
if the probability models for the class conditional distributions, and the a priori
probabilities are known. In the context of classification of brain tissue, the probability
models are not known, and therefore, must be approximated. The performance of the
Bayesian classifier is directly related to how well these distributions can be modeled.
In the development below, two Bayesian-based approaches are considered. The
first assumes Gaussian models for the class conditional probability distributions, and
equal a priori probabilities. The results obtained from this approach demonstrate the
need for additional complexity to be introduced in the modeling of the class conditional
and a posteriori probability models. Subsequently, the class conditional probability
models are improved using Parzen windowing, a nonparametric density estimation
technique. Additionally, a priori probability models are improved by approximation of
the imaged volume using a Markov random field model.


80

G. Bayesian Classification: Gaussian Conditionals, Equal a priori Probabilities

As an initial approach, class conditional probabilities are modeled using a
Gaussian distribution, and in the absence of any other information, equal a posteriori
probabilities are assumed [34].
The general form of a Gaussian probability distribution is given below in eq. 31.
In the context of the present problem, a single feature is used, being the intensity from the
FLAIR imagery. The form of a univariate distribution is given below in eq. 32.
In eq. 31, d is the dimension of the distribution, represents the d x d covariance
matrix, x is a d-component column vector, and is a d-component mean vector. In eq.
32, is the standard deviation, x is a scalar, and is the mean value of the distribution.


|
.
|

\
|

=

) ( ) (
2
1
exp
) 2 (
1
) | (
1
2
1
2

x x x p
t
d
j


(31)
|
|
.
|

\
|
|
.
|

\
|
=
2
2
1
exp
2
1
) | (

x
x p
j

(32)


As no information is known otherwise, the a posteriori probabilities for each class
are assumed to be equal. Therefore, for three classes, the probability P(
i
) is equal to 1/3
for each class. These probabilities are equal for each class, and therefore, do not affect
the result of the computation of eq. 28 or 29, and therefore, for the case considered here,

81

the posterior quantity upon which a decision is made is given completely by the result of
the computation of eq. 32.
From manually selected training samples, the mean and the standard deviation
are estimated using the maximum likelihood estimators (MLEs) for a univariate Gaussian
probability distribution function, given below in eqs. 33 and 34, respectively [35]. The
MLE estimates of the mean and standard deviation are and , respectively. The
quantity n represents the number of training samples available, and x
k
is the k
th
training
sample.

=
=
n
k
k
x
n
1
'
1


(33)
( )

=
=
n
k
k
x
n
1
2 '
'
2
) (
1

(34)


Below in figure 23, a sample result of the application of this scheme is presented.
In terms of performance, this classifier performs poorly. Upon close evaluation,
however, it is evident that the classifier works decently when the tissue is completely
unambiguous, and not in close proximity to boundaries with other tissues. For example,
the classification of normal brain material works well when not in close proximity to MS
lesions, and vice-versa. Based on the sample slice presented below in figure 23, as well

82

as other results obtained, it was observed that the classifier finds the lesions. Too much
of the surrounding tissue, however, is erroneously classified as diseased tissue.


(a)

(b)
FIGURE 23: Sample result of tissue classification in the brain using a simple Bayesian
classifier, with equal a priori probabilities, and Gaussian class conditional probabilities.

H. Bayesian Classification: Nonparametric Conditionals, a priori Probabilities Modeled
by Markov Random Fields

The above classification results leave much to be desired. Upon inspection of the
MAP estimator that is the basis of the Bayesian classifier, it is apparent that broadly, the
probability models for the class conditional and a priori distributions must be improved.
It is expected that improvement in these models will result in subsequent improvement in
the performance of the classifier.
In the above classifier, the class conditional probability model was assumed to
have a known parametric form. In this approach, the form of the distribution is known,
leaving only the parameters of the particular distribution to estimate. In the above
classifier, the parametric form assumed was that of a Gaussian. The parameters of the

83

Gaussian distribution, the mean and the standard deviation, were estimated using
maximum likelihood estimators.
A more general approach involves the use of nonparametric techniques for
density estimation [33]. Such approaches allow for direct estimation of the probability
distribution, eliminating the necessity of choosing a parametric form of the class
conditional distribution. Assumption of a parametric form is acceptable, if it is known
that the true distribution is Gaussian, or if a Gaussian is a good approximation to the true
class conditional distribution.
Use of a nonparametric technique, however, removes the need for the assumption
and estimation of the parameters of the assumed parametric model. Instead, the
probability distribution is estimated directly from the training data.
The nonparametric technique known as Parzen windowing is used here to
improve the model of the class conditional probabilities [33]. For each tissue class,
training samples are used to form an estimated probability distribution. The estimated
probability distribution is given by evaluation of eq. 35 below, where n is the number of
training samples, and (x) is a Parzen window function, taken to be a Gaussian kernel,
with a mean of zero, and a variance of one, here.


( )

=
=
n
i
i j
x x
n
x p
1
1
) | (
(35)



84

Estimation using the Parzen window estimate of eq. 35 can be made
computationally efficient by the generation of look-up tables for p(x|
j
). The use of
look-up tables avoids the computation of a sum, of a large number of training samples (n
of them), at each evaluation for some x and
j
.
The a priori probabilities P(
j
) were assumed in the simple Bayesian classifier,
above, to be equal for each tissue class. This assumption, however, is not well justified.
For example, in the input slice in figure 21 (a), it is very apparent that the probability of
normal brain material exceeds that of abnormal brain material. In the improvement of the
simple Bayesian classifier, then, it is also necessary to consider a more appropriate model
for the a priori probabilities.
It is reasonable to consider an a priori probability on a local scale, and consider
the spatial nature of the feature measurements obtained from MRI studies of the brain. A
feature measurement has a spatial location within the brain. If this feature measurement
is known to have a state of nature of
i
, then it is reasonable to assume that the feature
measurements surrounding this known voxel have an increased probability of also having
a state of nature of
i
. This assumption is reasonable, as there is order to anatomy
imaged, with abnormal brain material forming in regions, for example. The opposite of
this observation would be to take a slice, such as shown in figure 21 (a), and randomly
swap voxels until there is no region homogeneity in the image.
Markov random field (MRF) models [36, 37] provide such a modeling of
neighborhood influences that can be utilized to form an improved probability model for
the a priori probability P(
j
).

85

A Markov random field models an image or volume as a random field, a
structured collection of random variables, F = {F
1
, , F
m
}, and defined on the set S.
Here, each random variable is a lattice point in the sampled volume. A Markov random
field is a special case of a random field. There are two conditions imposed to allow a
random field to be labeled as a Markov random field [36]:
1. F f f p > , 0 ) ( , known as the positivity condition
2. ) | ( ) | (
} {
i
N i i S i
f f p f f p =

, known as the Markovianity condition

The positivity condition is implicitly satisfied with the application of the Markov
random field model of a priori probabilities. The Markovianity condition states that the
probability of an observation f
i
, given the other random variables in the field, is equal to
the probability of the observation f
i
, given a neighborhood around the sampled location,
denoted as N
i
. This statement fits well with the above reasoning, with respect to
consideration of local areas around a voxel to assist in formulating the a priori class
probability P(
i
).
Neighborhood structures are regions surrounding a sample, and chosen for
incorporation into the a priori probability model. For a regular sampling lattice (such as
the images here), a neighborhood is characterized by its order r, with the order defining
the nature of the neighborhood. The order r is related to the neighboring samples that are
included in the probability model by considering a circle of radius r . Figure 24 below
illustrates neighborhood structures for an image.


86


FIGURE 24: Neighborhood systems for an image modeled as a Markov random field.

Neighborhood models for a volume are constructed in the same methodology, in
three dimensions.
The Markovianity condition alone is not amenable to calculation for use in a
classifier. An equivalence between Markov random fields and Gibbsian random fields,
however, provides for a means of computation. A set of random variables F = {F
1
, ,
F
m
} is said to be a Gibbs random field (GRF) if the probability of its configurations is a
Gibbs distribution. A Gibbs distribution is of the form given in eq. 36. The constant T,
given in eq. 36, is known as the temperature constant. The quantity Z, appearing in eq.
36 and given in eq. 37, is known as a partition function, and is essentially a normalizing
constant such that eq. 36 is a true probability. The function U(f), appearing in eq. 36 and
given in eq. 38, is known as an energy function. In eq. 38, U(f) is given by the sum of the
function V
c
(f), over all C. Each V
c
(f) is known as a clique potential, and the value of this
function is related in some manner to the configuration of a clique. A clique is a

87

grouping of samples in a neighborhood system, such that the grouping includes voxels
that are neighbors of one another in the same system.


|
.
|

\
|
= ) (
1
exp
1
) ( f U
T Z
f p

(36)

|
.
|

\
|
=
F
f U
T
Z ) (
1
exp

(37)

=
C
c
f V f U ) ( ) (
(38)


A result known as the Hammersley-Clifford theorem states that if and only if a
random field F on S is a Markov random field with respect to a neighborhood system N,
then F is a Gibbs random field on S with respect to a neighborhood system N [36]. This
result allows the conditional probability given as the Markovianity condition of an MRF
to be converted to the non-conditional probability of a Gibbs distribution, given in eq. 36.
This allows for computation of the a priori class probabilities modeled as an MRF, and
therefore, incorporation of this model into a practical classifier.
In the classifier constructed here, a simple model for the energy function U(f) is
utilized [38]. This model consists of a linear combination of products of elements in the
cliques. For experimentation, a first order neighborhood system for image segmentation
is considered. The resulting model for U(f) is given below in eq. 39.


88


( ) ( ) ( )
1 , 1 , 2 , 1 , 1 1 , ,
) ( ) (
+ +
+ + + + = =
l k l k l k l k l k l k
f f f f f f U f U (39)


The quantity f
i,j
in eq. 39 refers to the sample at indices (i,j) in the labeled image.
The parameters ,
1
, and
2
are parameters that allow for adjusting relative weights or
contributions of neighborhood interactions. In the experiments here, is taken to be 1.0,
and
1
and
2
are both taken to be 0.75. These weightings allow the current classification
at index (i, j) to take an importance somewhat greater than the neighborhood
classifications, and gives equal weighting to the neighbors of the feature measured at (i,
j). Also, it should be noted that the temperature parameter T is taken to be 1.0.
The implemented algorithm accepts a slice of data, and training points. An initial
classification solution is obtained by assumption of equal a priori class probabilities.
Subsequent iterations implement modeling of a priori class probabilities using the MRF
model discussed above. Each iteration classifies the entire image, using the previous
classification as a basis to model neighborhood interactions. The iterative process
terminates when no changes are made in the classification. Class conditional
probabilities are modeled throughout from the training points initially supplied.
A sample of the results obtained is given below in figure 25. The classifier
successfully detects the lesions, and in general, is observed to perform well. Additional
comments on the behavior of the classifier are given below. In terms of computational

89

performance, each segmentation executes within seconds on the SGI Onyx2
supercomputer, as a non-threaded application.


(a)

(b)

(c)

(d)
FIGURE 25: Sample classification results obtained using a Bayesian classifier, with
class conditional probabilities modeled nonparametrically, and class a priori
probabilities obtained from a Markov random field model of the image. Input slices are
shown in (a) and (c), and the classified images are shown in (b) and (d). In (b) and (d),
normal brain material is indicated by the color yellow, CSF by gray, and lesions as red.

I. Quantification

With the classification results obtained, quantification of disease burden as judged
from MRI may be obtained by computing the total volume of lesions, and the total
volume of normal brain material. This can be simply accomplished by having the

90

computer count the number of labels assigned for each class, and then multiplying this
number by the volume of each voxel, which is known a priori from the scanner. Results
of this process are given in Table IV, below, for the output slices given in figure 25.

TABLE IV
QUANTIFICATION OF MS DISEASE BURDEN FROM THE SAMPLE RESULTS
GIVEN IN FIGURE 25.

Input
Slice
Number
of
normal
brain
tissue
voxels
Number
of
abnormal
brain
tissue
voxels
Volume
of normal
brain
tissue
(mm
3
)
Volume
of
abnormal
brain
tissue
(mm
3
)
Percentage
of normal
brain tissue
volume to
total brain
volume
Percentage
of
abnormal
brain tissue
volume to
total brain
volume
Figure.
23 (a)
12222 1,463 53709.960 6429.199 89.3 % 10.7 %
Figure
23 (c)
13920 1,053 61171.875 4627.441 93.0 % 7.0 %

J. Accuracy

The results obtained subjectively show good performance. Lesions are detected,
and highlighted. The classifier erroneously detects lesions around the outer perimeter of
the brain, mistaking scanning artifacts with MS lesions.
Results were shown to Dr. Robert Falk, a radiologist at Jewish Hospital. Dr.
Falks comments on the results obtained were positive.

91

K. Summary

Pattern recognition approaches were explored for classification of the tissues
present in the brain. Several approaches were attempted, including image processing
approaches using thresholding and image enhancement, unsupervised classification via
the use of the k-means algorithm, and statistical classification utilizing Bayes rule, with
Gaussian probability models for class conditional probability distributions.
Improvement of the statistical classifier by nonparametric modeling of the class
conditional probability distributions using Parzen windowing, and modeling of the a
priori class probabilities via the use of a Markov random field image model, provided the
best results obtained. Quantification of MS may then be implemented by computing the
total volume of normal and diseased brain material, from the resulting states of nature
assigned by the classifier and the voxel sizes used in the acquisition of the study.


92

V. VISUALIZATION



A. Introduction

Several scientific visualization techniques are applied here for the study of the
results of the registration and segmentation processes implemented. For the evaluation of
the volume registration results, a simple data fusion technique is applied, which overlaps
regions of the floating and re-sampled reference volumes. As well, a visualization
application is presented which allows for volume rendering, and re-slicing volumes along
arbitrary planes, allowing for more thorough evaluation of registration accuracy. For the
evaluation of segmentation results, a simple morphing or frame interpolation technique
for transitioning from the raw input image to the classified, colored output image is
explored. Finally, a web-based approach for presentation of results is considered.
Scientific visualization is of interest in this problem, due to the complexity of the
data sets involved, and the desire to facilitate discovery with regard to the pathology of
MS. Utilization of visualization procedures allows the raw data and processed results to
be manipulated on a higher level than otherwise available.

B. Visualization for the Study of Volume Registration

Following the process of volume registration using the software tool developed
for this research, one of the outputs available is a re-sampled version of the reference
volume, sampled at the locations corresponding to the sampling locations in the floating

93

volume. With the re-sampled reference volume, the reference and floating volumes may
then be compared on a slice-by-slice basis, as corresponding anatomy is registered in 3D
space.
As an aid to judging the quality of registration, a simple data fusion technique was
implemented for visualization. This technique takes regions from the re-sampled
reference volume and the floating volume to form a patchwork, composite image that
contains components of the imaged anatomy in both the reference and floating volumes.
This composite image, referred to as a checkerboard image, facilitates judgment of the
accuracy of the volume registration by allowing contours (such as the skull, or features
within the brain) to be followed between volumes, directly within one image. The
regions used here are 32 x 32 square pixel regions. Figure 26 below illustrates the
generation of the checkerboard images from the floating and re-sampled reference
volumes. Figure 27 below shows sample results obtained, from several registration
studies.


FIGURE 26: Generation of the checkerboard image, by formation of a composite image
of square pixel regions from the re-sampled reference and floating volumes.

94



(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)
FIGURE 27: Sample slices from different registration studies, showing the a slice from
the floating volume ((a), (d), and (g)), corresponding slices from the re-sampled,
registered reference volume ((b), (e),and (h), respectively), and the checkerboard slices
((c), (f), and (i), respectively).

An additional approach to the comparison of registration results was made with
the utilization of volume rendering, and arbitrary volume re-slicing. Figure 28, below,
shows an example of this application. The graphics components of this application were

95

developed using the Visualization Toolkit [43, 44], and implemented on the SGI Onyx2
supercomputer.


FIGURE 28: Volume rendering and arbitrary volume re-slicing for comparison of two
volumes that are registered via maximization of mutual information.

96

C. Visualization for the Study of Volume Segmentation

A very simple technique is considered for comparison of one volume to another,
on a slice-by-slice basis. This technique takes two inputs, an initial image, and a final
image. Between these two images, intermediate images are generated that represent an
interpolation between the initial and final images, as illustrated below in figure 29. This
process allows for a simple fade or morphing between two volumes, and can be used for
direct comparison of two raw scan volumes, or comparison of a raw scan volume to a
classified and colored volume. The initial, final, and intermediate volumes may be
combined in sequential order, and at a specified frame rate to form a movie that
progresses from the initial image/volume to the final image/volume.


FIGURE 29: Frame interpolation.


97

The interpolation scheme implemented in this research consists of interpolation
on a sample to sample basis. Thus, a sample from the initial image is interpolated until
the sample in the corresponding location in the final image is reached, independent of the
other samples in the image/volume.
Simple linear interpolation from a sample x
0
to x
1
is given in eq. 40, below. The
interpolated value, x(t), is a function of time t, a parameter for ordering the interpolated
images/volumes. The time parameter will range from 0, which generates the initial value
x
0
, to 1, which generates the final value x
1
.


0 0 1
) ( ) ( x t x x t x + = (40)


The above interpolation scheme may be generalized, by a time-dependent kernel
function k(t), other than t. Eq. 41 below generalizes the interpolation given in eq. 40.


0 0 1
) ( ) ( ) ( x t k x x t x + = (41)



Different kernel functions, including exponential, and quadratic functions were
experimented with. Kernels, such as an exponential function, with less abrupt transitions

98

from zero to one were found to produce the most visually appealing results, in
comparison to a linear interpolation scheme such as eq. 40.
For input and/or output images that are color, interpolation for each sample is
performed on the color components of the samples, using a red, green, blue (RGB) color
parameterization.

D. Web-Based Presentation of Results

Finally, web-based presentation of results was implemented, for presentation of
the results on the Internet. Therefore, the results may be viewed and evaluated from
virtually anyplace in the world (anyplace with Internet access available, at least). Figure
30 below shows an example of one of the web pages that was created for this presentation
format.


99


FIGURE 30: A sample of a web page developed for presentation of the results of this
study on the Internet.

E. Summary

Several simple visualization techniques were implemented and utilized for the
judgment of volume registration accuracy, and for facilitation of comparison of one

100

volume to another. These techniques are simple, but useful, and form groundwork for
more involved visualization to be applied for the study of MS.

101

VI. CONCLUSIONS AND RECOMMENDATIONS



The components of the preliminary system for the study of MS using MRI include
registration for volumetric alignment of patient studies over the course of time, brain
segmentation from the head facilitated by the use of registration, segmentation and
classification of brain tissue to the three tissue classes: CSF, normal brain material, and
diseased brain material, and the use of scientific visualization for the comparison of
patient scans and validation of the registration and segmentation processing.
While additional follow-up work is necessary, the developed approaches provide
a solid foundation on which to build future development in the study of MS using MRI,
as well as to future studies in medical imaging.
Based on the research and development implemented in this thesis, the following
recommendations for future work are made:
For better volumetric registration, enhancement of the mutual
information approach to incorporate additional features into the registration
for better spatial alignment [39], improvement in the metric itself [40], and
perhaps incorporation of non-rigid body registration[41]. The results
obtained for registration here are useful, however, there is room for
improvement.
For improved brain segmentation from the head, exploration of
techniques to directly segment the brain, or further development of the
registration approach explored here, to incorporate a high resolution, expert-

102

segmented model of the head, to provide the a priori segmentation. The
results obtained here show that the registration approach can be useful,
however, manual segmentation of the brain for an a priori segmented model
is far from ideal for use in a large scale study.
For improvement in the segmentation/classification of the extracted
brain material into CSF, normal brain tissue, and diseased brain tissue, a
means of correcting for inter-intensity variation in the MRI. The wide
variation in intensity levels encountered from the data available to this
research negates any possibility of application of a statistical classification
technique based upon training data from a slice.
For automation in the segmentation/classification of the extracted
brain material, generation of training data automatically. It is hypothesized
that by use of a filter (such as the non-linear anisotropic diffusion filter
applied in other areas here), an unsupervised classification algorithm (such as
the k-means clustering algorithm studied here), and a rule set, that generation
of training data may be possible. Inter- and intra-slice intensity correction
would assist in this effort, as well.
From an intensity-corrected study, and automated generation of
training data, 3D segmentation of lesions from the brain is possible, allowing
for more complex, 3D visualization tools to be applied for qualitative
discovery.
For multiple scans of a patient, taken at different time points, it may

103

be possible to use previous scans, prior to a given time, to improve in the
future segmentation and classification of brain material into CSF, normal
brain tissue, and diseased brain tissue. This would allow for, if for nothing
else, providing an initial solution for an iterative approach, and possibly
assisting in generating training data.
Improved quantification of error in the registration and segmentation
processing. At the present, the judgment of quality of the registration and
segmentation procedures largely depends upon qualitative evaluation by an
expert; it would be useful to quantify these errors numerically.


104

REFERENCES

[1] J. H. Noseworthy, C. Lucchinetti, M. Rodriguez, and B. G. Weinshenker,
Multiple sclerosis, The New England Journal of Medicine, vol. 343, no. 13, pp.
938-952, Sept. 28, 2000.

[2] National Institutes of Neurological Disorders and Stroke, Multiple Sclerosis:
Hope Through Research, NIH Publication No. 96-75, July 1, 2001.

[3] Z. P. Liang and P. C. Lauterbur, Principles of Magnetic Resonance Imaging: A
Signal Processing Perspective. New York: IEEE Press, 2000.

[4] D. H. Miller, P. S. Albert, F. Barkhof, G. Francis, J. A. Frank, S. Hodgkinson, F.
D. Liblin, D. W. Paty, S. C. Reingold, and J. Simon, Guidelines for the Use of
Magnetic Resonance Techniques in Monitoring the Treatment of Multiple
Sclerosis, Annals of Neurology, vol. 39, no. 1, Jan. 1996.

[5] C. R. G. Guttmann, R. Kikinis, M. C. Anderson, M. Jakab, S. K. Warfield, R. J.
Killiany, H. L. Weiner, F. A. Jolesz, Quantitative Follow-up of Patients with
Multiple Sclerosis: Reproducibility, JMRI, vol. 9, no. 4., pp. 509-518, June
1999.

[6] R. Kikinis, C. R. G. Guttmann, D. Metcalf, W. M. Wells, G. J. Ettinger, H. L.
Weiner, and F. A. Jolesz, Quantitative Follow-up of Patients with Multiple
Sclerosis: Technical Aspects, JMRI, vol. 9, no. 4, pp. 519-530, June 1999.

[7] W. M. Wells, W. E. L. Grimson, R. Kikinis, and F. A. Jolesz, Adaptive
Segmentation of MRI Data, IEEE Transactions on Medical Imaging, vol. 15, no.
4, pp. 429-442, August 1996.

[8] M. Kamber, R. Shinghal, D. L. Collins, G. S. Francis, and A. C. Evans, Model-
Based 3-D Segmentation of Multiple Sclerosis Lesions in Magnetic Resonance
Brain Images, IEEE Transactions on Medical Imaging, vol. 14., no. 3, pp. 442-
453, Sept. 1995.


105

[9] B. Johnston, M. S. Atkins, B. Mackiewich, and M. Anderson, Segmentation of
Multiple Sclerosis Lesions in Intensity Corrected Multispectral MRI, IEEE
Transactions on Medical Imaging, vol. 15, no. 2, pp. 154-169, April 1996.

[10] J. Besag, On the Statistical Analysis of Dirty Pictures, Journal of the Royal
Statistical Society B, vol. 48, no. 3, pp. 259-30, 1986.

[11] J. K. Udupa, L. Wei, S. Samarasekera, Y. Miki, M. A. van Buchem, and R. I.
Grossman, Multiple Sclerosis Lesion Quantifiction Using Fuzzy-Connectedness
Principles, IEEE Transactions on Medical Imaging, vol. 16, no. 5, pp. 598-609,
Oct. 1997.

[12] J. K. Udupa and S. Samarasekera, Fuzzy Connectedness and Object Definition:
Theory, Algorithms, and Applications in Image Segmentation, Graphical Models
and Image Processing, vol. 58, no. 3, pp. 246-261, May 19996.

[13] K. V. Leemput, F. Maes, D. Vandermeulen, A. Colchester, and P. Suetens,
Automated Segmentation of Multiple Sclerosis Lesions by Model Outlier
Detection, IEEE Transactions on Medical Imaging, vol. 20, no. 8, pp. 677-688,
August 2001.

[14] F. Maes, A. Collignon, D. Vandermeulen, G. Marchal, and P. Suetens,
Multimodality Image Registration by Maximization of Mutual Information,
IEEE Transactions on Medical Imaging, vol. 16, no. 2, pp. 187-198, April 1997.

[15] W. M. Wells, P. Viola, H. Atsumi, S. Nakajima, and R. Kikinis, Multi-modal
Volume Registration by Maximization of Mutual Information, Medical Image
Analysis, vol. 1, no. 1, pp. 35-51, 1996.

[16] C. R. Meyer, J. L. Boes, B. Kim, P. H. Bland, K. R. Zasadny, P. V. Kison, K.
Koral, K. A. Frey, and R. L. Wahl, Demonstration of Accuracy and Clinical
Versatility of Mutual Information for Automatic Multimodality Image Fusion
Using Affine and Thin-Plate Spline Warped Geometric Deformations, Medical
Image Analysis, vol. 1., no. 3, pp. 195-206, 1996/7.


106

[17] J. Nolte, The Human Brain: An Introduction to its Functional Anatomy. St.
Louis: Mosby Year Book, 3
rd
edition, 1993.

[18] J. B. A. Maintz and M. A. Viergever, A Survey of Medical Image Registration,
Medical Image Analysis, vol. 2., no. 1, pp. 1-36, 1998.

[19] M. A. Audette, F. P. Ferrie, and T. M. Peters, An Algorithmic Overview of
Surface Registration Techniques for Medical Imaging, Medical Image Analysis,
vol. 4, pp. 201-217, 2000.

[20] G. Wolberg, Digital Image Warping. Los Alamitos: IEEE Computer Society
Press, 1990.

[21] T. M. Cover and J. A. Thomas, Elements of Information Theory. New York:
John Wiley & Sons, 1991.

[22] G. P. Penney, J. Weese, J. A. Little, P. Desmedt, D. L. G. Hill, and D. J. Hawkes,
A Comparison of Similarity Measures for Use in 2D-3D Medical Image
Registration, IEEE Transactions on Medical Imaging, vol. 17, no. 4, pp. 586-
595, August 1998.

[23] W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery, Numerical
Recipes in C: The Art of Scientific Computing. Cambridge University Press,
Second Edition, 1992.

[24] F. Maes, D. Vandermeulen, and P. Suetens, Comparative Evaluation of
Multiresolution Optimization Strategies for Multimodality Image Registration by
Maximization of Mutual Information, Medical Image Analysis, vol. 3, no. 4, pp.
373-386, 1999.

[25] S. L. S. Jacoby, J. S. Kowalik, and J. T. Pizzo, Iterative Methods for Nonlinear
Optimization Problems. Englewood Cliffs: Prentice-Hall, 1972.


107

[26] M. Joliet and B. M. Mazoyer, Three-Dimensional Segmentation and
Interpolation of Magnetic Resonance Brain Images, IEEE Transactions on
Medical Imaging, vol. 12, no. 2, pp. 269-277, June 1993.

[27] M. E. Brummer, R. M. Mersereau, R. L. Eisner, and R. R. J. Lewine, Automatic
Detection of Brain Contours in MRI Data Sets, IEEE Transactions on Medical
Imaging, vol. 12, no. 2, pp. 153-166, June 1993.

[28] T. Kapur, W. E. L. Grimson, W. M. Wells, and R. Kikinis, Segmentation of
Brain Tissue from Magnetic Resonance Images, Medical Image Analysis, vol. 1,
no. 2, pp. 109-127, 1996.

[29] M. S. Atkins and B. T. Mackiewich, Fully Automatic Segmentation of the Brain
in MRI, IEEE Transactions on Medical Imaging, vol. 17, no. 1, pp. 98-107, Feb.
1998.

[30] C. Sites, A. Farag, and T. Moriarty, Visualization Algorithms for Medical
Imaging with Telemedicine and High Performance Computing Applications,
Technical Report, CVIP Laboratory, December 2000.

[31] N. Shareef, D. L. Wang, and R. Yagel, Segmentation of Medical Images Using
LEGION, IEEE Transactions on Medical Imaging, vol. 18, no. 1, pp. 74-91, Jan.
1999.

[32] G. Gerig, O. Kbler, R. Kikinis, and F. A. Jolesz, Nonlinear Anisotropic
Filtering of MRI Data, IEEE Transactions on Medical Imaging, vol. 11, no. 2,
pp. 221-232, June 1992.

[33] R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification. New York: John
Wiley & Sons, 2
nd
edition, 2001.

[34] H. E. Cline, W. E. Lorensen, R. Kikinis, and F. Jolesz, Three-Dimensional
Segmentation of MR Images of the Head Using Probability and Connectivity,
Journal of Computer Assisted Tomography, vol. 14, no. 6, pp. 1037-1045,
November/December 1990.


108

[35] R. V. Hogg and E. A. Tanis, Probability and Statistical Inference. Upper Saddle
River: Prentice-Hall, 6
th
edition, 2001.

[36] S. Z. Li, Markov Random Field Modeling in Image Analysis. Tokyo: Springer-
Verlag, 2001

[37] K. Held, E. R. Kops, B. J. Krause, W. M. Wells, R. Kikinis, and H. W. Mller-
Grtner, Markov Random Field Segmentation of Brain MR Images, IEEE
Transactions on Medical Imaging, vol. 16, no. 6, pp. 878-886, Dec. 1997.

[38] C. W. Therrien, Decision, Estimation, and Classification. New York: John Wiley
& Sons, 1989.

[39] J. P. W. Pluim, J. B. A. Maintz, and M. A. Viergever, Image Registration by
Maximization of Combined Mutual Information and Gradient Information, IEEE
Transactions on Medical Imaging, vol. 19, no. 8, pp. 809-814, August 2000.

[40] C. Studholme, D. L. G. Hill, and D. J. Hawkes, An Overlap Invariant Entropy
Measure of 3D Medical Image Alignment, Pattern Recognition, vol. 32, pp. 71-
86, 1999.

[41] B. Likar and F. Pernus, A Hierarchical Approach to Elastic Registration Based
on Mutual Information, Image and Vision Computing, vol. 19, pp. 33-44, 2001.

[42] American College of Radiology, National Electrical Manufacturers Association,
"Digital Imaging and Communications in Medicine (DICOM): Version 3.0",
Draft Standard, ACR-NEMA Committee, Working Group VI, Washington, DC,
1993.

[43] W. J. Schroeder, L. S. Avila, K. M. Martin, W. A. Hoffman, and C. C. Law, The
VTK Users Guide. Kitware, 2001.

[44] W. Schroeder, K. Martin, and B. Lorensen, The Visualization Toolkit: An Object-
Oriented Approach to 3D Graphics. Upper Saddle River: Prentice-Hall, 2
nd

Edition, 1998.

109

VITA


Jeremy Michael Nett, son of Michael and Kathy Nett, was born on July 9, 1978 in
Louisville, Kentucky. He graduated as class valedictorian from Tates Creek High School
in Lexington, Kentucky, in 1996. In the fall of 1996, he entered the Speed Scientific
School at the University of Louisville. In spring of 2000, he graduated summa cum laude
from the University of Louisville with a Bachelor of Science degree in Electrical
Engineering. During his undergraduate studies, he completed three co-operative
internships with Thomson Consumer Electronics, in Indianapolis, IN. In the summer of
2000, he began his studies towards a Masters of Engineering degree at the University of
Louisville's Computer Vision and Image Processing Lab, in the Electrical and Computer
Engineering Department.