+Model BIO-2709;

No. of Pages 11

ARTICLE IN PRESS

BioSystems xxx (2007) xxxxxx

Automated classication of atherosclerotic plaque from magnetic resonance images using predictive models
Russell W. Anderson a, , Christopher Stomberg b , Charles W. Hahm a , Venkatesh Mani c , Daniel D. Samber c , Vitalii V. Itskovich c, , Laura Valera-Guallar b , John T. Fallon d , Pavel B. Nedanov a , Joel Huizenga a , Zahi A. Fayad c
ISCHEM Corporation, La Jolla, CA, USA b Bates White, LLC, Del Mar, CA, USA c Imaging Science Laboratories, Department of Radiology, The Zena and Michael A Wiener Cardiovascular Institute, The Marie-Jos e and Henry R, Kravis Cardiovascular Health Center, Mount Sinai School of Medicine, New York, NY, USA e d Department of Pathology, Mount Sinai School of Medicine, USA Received 30 May 2006; received in revised form 4 October 2006; accepted 8 November 2006
a

Abstract The information contained within multicontrast magnetic resonance images (MRI) promises to improve tissue classication accuracy, once appropriately analyzed. Predictive models capture relationships empirically, from known outcomes thereby combining pattern classication with experience. In this study, we examine the applicability of predictive modeling for atherosclerotic plaque component classication of multicontrast ex vivo MR images using stained, histopathological sections as ground truth. Ten multicontrast images from seven human coronary artery specimens were obtained on a 9.4 T imaging system using multicontrastweighted fast spin-echo (T1-, proton density-, and T2-weighted) imaging with 39- m isotropic voxel size. Following initial data transformations, predictive modeling focused on automating the identication of specimens plaque, lipid, and media. The outputs of these three models were used to calculate statistics such as total plaque burden and the ratio of hard plaque (brous tissue) to lipid. Both logistic regression and an articial neural network model (Relevant Input Processor NetworkRIPNet) were used for predictive modeling. When compared against segmentation resulting from cluster analysis, the RIPNet models performed between 25 and 30% better in absolute terms. This translates to a 50% higher true positive rate over given levels of false positives. This work indicates that it is feasible to build an automated system of plaque detection using MRI and data mining. 2006 Elsevier Ireland Ltd. All rights reserved.
Keywords: Magnetic resonance imaging; MRI; Coronary disease; Atherosclerosis; Vulnerable plaque

This research was funded, in part, by an NIH Phase I SBIR Grant awarded to Dr. Anderson and the ISCHEM Corporation. Partial support was also provided by: NIH/NHLBI R01 HL71021, NIH/NHLBI R01 HL78667 (ZAF). Corresponding author at: ISCHEM Corporation, 354 Gravilla Street, La Jolla, CA 92037, USA. Tel.: +1 858 459 0011; fax: +1 858 459 8122. E-mail address: russell wayne anderson@hotmail.com (R.W. Anderson). In memory of our friend and colleague, Dr. Vitalii Itskovich. Dr. Vitalii Itskovich worked with Drs. Zahi A. Fayad and Valentin Fuster for many years in the area of atherosclerosis research conducted at the Mount Sinai School of Medicine. Vitaliis graduate work in the area of aortic wave velocity measurements provided groundwork for his noted accomplishments in the eld of MR pulse sequencing. He was instrumental in the development of sequences useful for atherosclerotic plaque imaging including REX, IR-TFL-DIFF, and GRASP. His expertise in the eld of MR image analysis led to the development of algorithms based on cluster analysis for of automating plaque segmentation. His contributions extended to diverse areas of research such as the characterization of aortic root atherosclerosis, detection of lesion regression and in disease progression in Marfans syndrome. In September of 2003, Vitalii was diagnosed with a rare and particularly virulent form of gastric cancer. Vitalii vigorously researched his cancer and became very involved with his treatment plans outliving all expectations before succumbing at age 34 on a cold yet sunny New York City January day. 0303-2647/$ see front matter 2006 Elsevier Ireland Ltd. All rights reserved. doi:10.1016/j.biosystems.2006.11.005

Please cite this article in press as: Anderson, R.W. et al., Automated classication of atherosclerotic plaque from magnetic resonance images using predictive models, BioSystems (2007), doi:10.1016/j.biosystems.2006.11.005

+Model BIO-2709;
2

No. of Pages 11

ARTICLE IN PRESS
R.W. Anderson et al. / BioSystems xxx (2007) xxxxxx

1. Introduction Atherosclerosis is the most common cause of ischemic heart disease, which can lead to unstable angina, myocardial infarction (heart attack), stroke, and sudden death. Studies indicate that plaque composition rather than the degree of stenosis or blood vessel narrowing is the key factor for predicting vulnerability to rupture or thrombosis (Fayad and Fuster, 2001; Fuster et al., 1999). Several invasive methods have been used to identify plaque, including intravenous ultrasound (IVUS), angioscopy, intravascular MR, and thermography (Fayad and Fuster, 2001; MacNeill et al., 2003). These invasive tests are not appropriate for screening or serial examinations. Noninvasive imaging methods such as magnetic resonance imaging (MRI), ultrasound, or computed tomography (CT) may be used to obtain a much more accurate assessment of plaque burden, which facilitates the characterization of disease progression (Fayad et al., 2002, 2004). MRI is emerging as one of the dominant non-invasive methods for identifying and studying atherosclerotic plaque (Fayad and Fuster, 2001) (Corti et al., 2001; Fayad, 2003; Fayad et al., 2002, 2004; Yuan et al., 2001a). A key advantage of MRI is that structures can be imaged using several contrast weightings. For example, T1-, T2-, and proton density-weighted images of the same anatomical tissue can be quite different, depending on the chemical components and structure of the tissue; hence, features not easily distinguished in one contrast weighting can show strong contrast in others (Cai et al., 2002; Shinnar et al., 1999; Yuan et al., 2001b). Currently, atherosclerotic plaque is characterized by visual inspection by expert readers (Fayad and Fuster, 2001; Yuan et al., 2001b). This is a time-consuming and error-prone process, subject to several subjective biases. Classication accuracy is subject to inter- and intraobserver variability. Some automated tools have been developed to assist in the labeling process, including algorithms for aligning images (Kerwin et al., 2001; Suri and Laxminarayan, 2003), or outlining gross structure boundaries, such as the lumen or vessel wall boundaries (Chao et al., 2003; Mansard et al., 2004; Thomas et al., 2001). Several groups have developed local texture segmentation algorithms for clustering sub-tissues, the most common method being a variant of K-means clustering (Xu et al., 2002; Yang et al., 2003; Yuan et al., 1999; Itskovich et al., 2004; Wolf et al., 2005). However, such unsupervised methods have high false-positives when used as classiers and require a human operator to vali-

date, adjust, and override the results. Clarke et al. (2003) extended this approach by developing a tissue-specic map, incorporating information from eight MR contrast weightings, and using a template matching approach to identify plaque components in endarterectomy specimens. This approach is similar to cluster analysis, with the important distinction that clusters are matched to idealized exemplars. In most cases, regression or predictive models have higher discrimination performance and are more robust (generalize to novel conditions) than template matching, as they operate on a less restricted solution space (Anderson et al., 2003). Multiple contrast MR images offer a unique opportunity to improve feature classication accuracy as the number of image attributes is multiplied by the number of modalities. (The number of predictive variables that can be derived from combinations of these attributes is literally innite.) However, human expert examination cannot take full advantage of this potential because of the complexity of the information that must be integrated. Unsupervised learning techniques such as segmentation algorithms can incorporate and summarize highly complex information, but do not take full advantage of what the expert knows from other sources of information such as anatomical cues or other diagnostic measurements. Supervised learning techniques, such as predictive models, capture relationships empirically, from known outcomes (in this case, post-operative histology) and thus combine statistical pattern classication with experience. Once a sample set of images has been labeled by an expert, a statistical model can be t (using regression or optimization techniques) to exploit the collected evidence (Bishop, 1995). In this study, we report the use of predictive modeling for plaque component classication from ex vivo MR images.
2. Methods This section describes the methods used to prepare tissue samples and acquire MR images. Imaging was performed on a Bruker 9.4T MR system (Bruker Instruments, Billerica, MA). Seven human coronary artery specimens 1.28 cm long (1 from Pathobiological Determinants of Atherosclerosis in Young (PDAY) data library (Zieske et al., 2000) and 6 from the Institutions Autopsy Service) were imaged using a 10 mm birdcage coil. The tissue was then processed using customary histological procedures designed to minimize tissue disruption from decomposition and temperature variation. A thermocouple-based temperature-controlling element was used to maintain the samples at a physiologic temperature of 37 C. A paraformaldehyde-xed, longitudinally cut and attened specimen (25 months post-mortem) was obtained in an

Please cite this article in press as: Anderson, R.W. et al., Automated classication of atherosclerotic plaque from magnetic resonance images using predictive models, BioSystems (2007), doi:10.1016/j.biosystems.2006.11.005

+Model BIO-2709;

No. of Pages 11

ARTICLE IN PRESS
R.W. Anderson et al. / BioSystems xxx (2007) xxxxxx 3

Fig. 1. Color composite method for MR images. T1-weighted (T1W), PD-weighted (PDW) and T2-weighted (T2W) axial plane images were assigned red (R), green (G), and blue (B) colors respectively and were combined to obtain the composite image of the coronary artery with color distributions corresponding to different tissue types.

airtight container. The specimen was then refrigerated and rewarmed before imaging, while the coronary artery specimens from the autopsy service were imaged fresh. Each specimen was placed in an 8-mm-diameter polyethylene tube that was lled with Fomblin solution (peruoropolyether; Ausimont USA Inc., Morristown, NJ) used to exclude air from the imaged volume. Fomblin does not interact chemically with the imaged tissue, nor does it have a residual MR signal that could interfere with that originating from the imaged tissue. Specimens were imaged using a 3D fast spin-echo (FSE) sequence with an isotropic voxel of 39 microns (eld of view (FOV) = 1.0 cm, echo-train length = 4, and acquisition matrix = 2563 ). Of the seven specimens, three of the seven cross-sections were made past arterial bifurcations, yielding a total of 10 arterial crosssections (Fig. 5, rst column). Color composite MR images: Multi-contrast weighted MRI was used to characterize each specimen, using PDW, T1W, and T2W images with repetition time/echo time (TR/TE) = 2000/9, 500/9, and 2000/25 ms, respectively. These images were subsequently mapped green, red, and blue color channels resulting in one color composite image (Fig. 1) (Itskovich et al., 2004). The acquisition time was 8 h for the PDW and T2W imaging, and 2 h for the T1W imaging. The position and size of the 3D volume were not changed from one MR acquisition to another. The false color images for the complete dataset are shown in the second column in Fig. 5. Histopathology: Each sample specimen underwent decalcication, parafn embedding, and serial sectioning. Fivemicron sections were stained with a Combined Massons

Elastic (CME) stain. Two digitally imported photomicrographs of histopathology slices from each sample were randomly chosen for analysis. These images enabled identication of the main tissue components (loose brous, media, dense brous, bro-cellular cap, thrombus, lipid/necrotic core) (Garvey, 1984). Corresponding areas on the sample MRI images were manually traced and labeled according to their identication from the histopathology images. Subsets of histopathological sections and their corresponding labeled multicontrast-weighted MR images were used for model training and validation, and were considered the gold standard against which images segmented by our algorithms were compared (Fig. 2). 2.1. Image processing and feature enhancement The overall objective of image processing is to create transformations that reveal salient image characteristics and facilitate subsequent analysis. Types of image processing operations employed (while not mutually exclusive) can be conveniently classied according to on their mathematical objective: (1) noise reduction, (2) dimension reduction, (3) texture or feature detection and (4) derived variables (customized variables, typically designed using expert domain knowledge). Noise reduction: The composite multi-contrast images were ltered in order to reduce noise. The image was rst median-ltered to remove noise with impulse characteristics. Commonly employed in image processing, a median lter replaces the intensity value of a pixel with the statistical median

Fig. 2. Labeled color composite MR image with corresponding histopathological section. An MR image of an atherosclerotic blood vessel (left) is labeled based on information from corresponding histology (right). This information will later be used as a basis for constructing a statistical model.

Please cite this article in press as: Anderson, R.W. et al., Automated classication of atherosclerotic plaque from magnetic resonance images using predictive models, BioSystems (2007), doi:10.1016/j.biosystems.2006.11.005

+Model BIO-2709;
4

No. of Pages 11

ARTICLE IN PRESS
R.W. Anderson et al. / BioSystems xxx (2007) xxxxxx

Fig. 3. Examples of K-means cluster analyzed blood vessel images obtained by MR. The K-Means clustering algorithm transformed and reduced the original MR data into more statistically signicant variables, which facilitate subsequent image processing.

of intensities in its neighborhood. Median lters are noted for their ability remove impulse type noise without blurring the edges residing within the image. The median ltered image was then smoothed with an adaptive Wiener lter that adjusts to statistics in the surrounding square pixel neighborhood with N pixels on a side. The mean and variance were estimated from the intensities a, where is the neighborhood local to the ltered pixel: = (1/N 2 ) a(n1 , n2 )
n1 ,n2

2 = (1/N 2 )

a2 (n1 , n2 ) 2
n1 ,n2

These estimates were then used to assign the parameter b to a Wiener lter, where 2 is an estimate of the overall noise variance taken as the expected value of the local variances across the entire image. b(n1 , n2 ) = + ( 2 + 2 ) (a(n1 + n2 ) 2 ) 2

A two-dimensional convolution was performed on each image intensity plane with the coefcients b above. Dimension reduction: Dimension reduction refers to a family of methods employed to reduce noise and dimensionality of raw image data. Common methods include principal component analysis (PCA), averaging, independent component analysis (ICA), and cluster analysis. Itskovich et al. (2004) demonstrated K-means clustering algorithms can create classes

with strong discrimination performance when applied to multimodal MRI data. In developing the predictive models, the K-means cluster membership was treated as one of many potential variables to be included in the predictive models. The K-means clustering algorithm was employed to form k data classes, the members of which reside in (feature-space) locations that were least distant from the estimated centroid of each class (Lloyd, 1982). Complete algorithmic details of the version of K-means clustering used in this study are available at the website for the SOM toolbox, maintained by the University of Helsinki (http://www.cis.hut./projects/somtoolbox/). An example image, classied by the resulting clusters is shown in Fig. 3. By inspection, one can see that some clusters had high correlation to particular tissue types; however, interpretation of the cluster classes still requires human validation and renement. K-means is the dominant methodology used to segment plaque in MR images. Although, the method is typically used to aid human operators, it is possible to automate the methodology using template matching techniques (Clarke et al., 2003) and evaluate its classication performance. As a predictive variable, the cluster membership functions distinguished muscle, and hard plaque (brous tissue), and lipid with a KolmogorovSmirnov (KS) statistic of 57, 57, and 58, respectively (Table 1, Fig. 4ac). However, as will be demonstrated, predictive models (which include the cluster membership as an input variable) far outperform such unsupervised methods. Texture measures: Tissues and plaque components are often distinguished by their texture. However, as there are no canon-

Table 1 Model performance: performance of the RIPNet models is based on the maximum Kolmogorov-Smirnov statistic (Max-KS) and the Gini coefcient measurements of the ROC curve in Fig. 5 Model Max-KS Ripnet (%) Plaque Muscle Lipid Average 76.3 83.0 91.0 83.4 Logistic (%) 73.0 76.1 78.0 75.7 K-means (%) 57.4 57.3 58.0 57.6 Gini Ripnet (%) 83.8 89.7 94.5 89.3 Logistic (%) 81.0 80.8 81.9 81.2 K-means (%) 67.1 56.6 55.3 59.7

The ROC curve is computed using the holdout sample from the modeling data set.

Please cite this article in press as: Anderson, R.W. et al., Automated classication of atherosclerotic plaque from magnetic resonance images using predictive models, BioSystems (2007), doi:10.1016/j.biosystems.2006.11.005

+Model BIO-2709;

No. of Pages 11

ARTICLE IN PRESS
R.W. Anderson et al. / BioSystems xxx (2007) xxxxxx 5

Fig. 4. Performance comparison of three classication methods on multicontrast MR Images. Classication performance for K-means clustering under ideal conditions, i.r. it assumes an automated system would rank-order clusters as well as a human expert, Logistic Regression, and Neural Network Classiers, ROC curves. (a) Plaque detection, (b) muscle detection, (c) lipid detection.

ical or mathematical denitions of texture, image readers typically describe textures using subjective terminology. For example, muscle and collagenous tissues are often striated (banded), while necrotic cores appear mottled, or variegated. This aspect of the data was quantied using two classes of texture measures, statistics on the local intensity variations and spatial frequency. Standard statistics, such as mean, median, and standard deviation, were used for the different pixel measures, applied to neighborhoods of various sizes. Additionally, the two dimensional discrete cosine transform (DCT) (Rao and Yip, 1990) was used to estimate the spatial spectral energy for both x and y orientations. For an N M pixel area, the following equation was used to estimate the DCT coefcients using the image intensity values for a sliding window A Amn . In our experiment, square areas were used (N = M), for ranges of N from 3 to 7.
M1 N1

Bpq = p q
m=0 n01

Amn cos

(2n + 1)q (2m + 1)p cos , 2M 2N

0pM1 0qN 1 1/ M, p = 0 p = 2/M, 1 p M 1 1/ N, q = 0 = 2/N, 1 q N 1 ,

Derived variables: Derived variables are combinations of more basic entities. Examples of variables falling under this category include products, ratios, and coordinate system transforms. The statistical modeling tools generate additional derived variables such as cross products of the existing variables. Predictive modeling: The predictive modeling in this study focused on automating the identication and discrimination of the two major plaque components present in the data available (lipid and hard plaque). An additional model was developed to detect the media (or arterial muscle) both to distinguish this tissue type from plaque and to establish the boundaries of the blood vessel walls and calculating their cross-sectional area. Combined, the outputs of these three models were used to calculate statistics such as total plaque burden and the ratio of hard plaque to lipid, which gives the model results clinical value. Although desirable, additional models for identifying other major plaque components, such as thrombus and calcication, were not possible due to data limitations. Unfortunately, all of the histological sections in the dataset had been decalcied prior to staining, preventing accurate image labeling. Furthermore, calcication can assume several morphological forms, so any effort to develop a robust model to detect calcium would necessarily require several examples of calcied tissues, although a recent study demonstrated strong discrimination performance can be achieved using multimodal MRI (Wolf et al., 2005). Generalized thrombus detection presents similar data challenges. The character, texture, and composition of thrombus change over time. Young thrombus is clear

Please cite this article in press as: Anderson, R.W. et al., Automated classication of atherosclerotic plaque from magnetic resonance images using predictive models, BioSystems (2007), doi:10.1016/j.biosystems.2006.11.005

+Model BIO-2709;
6

No. of Pages 11

ARTICLE IN PRESS
R.W. Anderson et al. / BioSystems xxx (2007) xxxxxx

and homogeneous. As the clot ages, platelets are replaced with brous tissue, etc. Hence, the development of a robust model for thrombus will require examples from several patients at various stages. Efforts are underway to collect additional data in vivo and ex vivo to characterize these key plaque components. The predictive models were trained on a pixel-by-pixel basis to eliminate the complexities associated with morphology. The models for each type of tissue were estimated separately. For the three models each pixel-record in the training dataset contains hundreds of local characteristics derived from the image processing steps described above along with one discretionary variable that identies whether the pixel is classied as the tissue being modeled (dependent variable), based on the ground-truth classication for that pixel. The predictive models, once estimated, generate predictions of feature on a pixel-by-pixel basis. The outcome for each model consists of the predicted probability that a pixel belongs to a type of tissue. This approach is theoretically robust to changes in resolution. For example, decreases in resolution represent a form of spatial averaging that may decrease specicity of, but may not totally invalidate predictions from the model. Devicedependence remains an issue because many of the predictor variables use pixel-based measures of neighborhood that are not robust to changes in resolution. An articial neural network (ANN) modeling approach known as the Relevant Input Processor Network (RIPNet) was used for predictive modeling (Perez-Amaral et al., 2003; PerezAmaral et al., 2005). The use of RIPNet involves the following four steps: (1) data segmentation/sampling, (2) anomalous data detection, (3) variable pre-selection and transform generation, (3) RIPNet model estimation and validation, and (4) candidate model testing. Data segmentation/sampling: The scanning, data processing, and hand labeling processes resulted in 10 images (199,000 pixel records) that could be used for training predictive models. All images contained signicant examples of plaque. Seven of the 10 images (108,000 pixels) were pooled in the training sample, while three were excluded entirely from the estimation and validation process (holdout sample). The target (dependent) variables were indicator variables equal one if the pixel belonged to the target class, and zero otherwise. There were three target variables, one for each feature class. The RIPNet procedure requires data to be split into training, validation, and testing data sets. The training data are used to t the models. The validation data are used to evaluate and pick candidate models. The testing data are used essentially once, after training and validation have produced a series of candidate models, to pick the model or models that stand out from the others in terms of predictive performance. The three images in the holdout sample were not used in any of these stages, but instead used a nal test of real-world performance. Anomalous data detection and the anomaly variable: Because nonlinear models are highly sensitive to outliers, it is important to eliminate outlier data if any exist. The Anomaly detection tool identies records as outliers if they are distant (as measured by L1-norm) from their k-nearest neighbors.

This tool also performs one other task, which is to generate the anomaly variable. The anomaly variable translates two distance measures for each observation (one for each of the target populations dened by the dependent variable) into a likelihood ratio statistic. This likelihood statistic embeds information about the dependent variable in a non-linear transform of input variables that can be used for predictive purposes. The anomaly variable used in each model was composed of up to ve continuous input variables. Additional variable transform generation and preselection: The nal task before beginning the model tting process was an additional phase of variable transform generation and preliminary elimination of non-predictive variables. The transforms generated at this stage include, for example, cross-products of the inputs, and group transforms of the continuous inputs (using a clustering technique to encode continuous into discrete variables). All variables were tested for relevance in predicting the target variable using an outof-sample pseudo R-squared statistic. This statistic measures the approximate percentage of variation in the target variable explained by the independent variable. Variables that have low or negative univariate or bivariate pseudo R-squared statistics were permanently dropped from the potential candidate variable pool. Model estimation and variable selection: This is the point at which RIPNet estimation commenced. The RIPNet algorithm, in addition to its internal model-selection capabilities, allows ranges of model complexity to be evaluated. Initial modeling is based on a small subset of predictor variables, and uses few additional transformations. The resulting model, which is similar to a standard logit model, is used as an initial benchmark. The full range of complexity was explored including, for example, allowing large or small numbers of candidate variables, high or low levels of network complexity, and high or low levels of variable redundancy in the model. These selections each create a range within which RIPNet optimizes a network for best prediction performance on the validation data. The result is a set of best candidate models, each one corresponding to a particular setting for the complexity of the network. Candidate model testing: Given the process described above for generating models, a technique was required to pick one model from among several candidates. Data Mining Reality Check (DMRC) (White, 1999, 2000) was used to perform this evaluation. This technique utilizes bootstrap distributions to generate valid p-values for the hypothesis that the tested model has the same out-of-sample forecast performance as a designated benchmark. A small p-value indicates signicantly better performance. In this study a parsimonious model was chosen as the initial benchmark. When a demonstrably better model was identied by DMRC from among a class of candidates, that model became the benchmark against which further models were compared. This process was repeated until no additional candidate models yielded demonstrably better forecasting than the existing benchmark model. The benchmark model at the end of this process became the nal selected model.

Please cite this article in press as: Anderson, R.W. et al., Automated classication of atherosclerotic plaque from magnetic resonance images using predictive models, BioSystems (2007), doi:10.1016/j.biosystems.2006.11.005

+Model BIO-2709;

No. of Pages 11

ARTICLE IN PRESS
R.W. Anderson et al. / BioSystems xxx (2007) xxxxxx 7

Image re-construction: Finally, after estimating the model, we compiled the results of the hard plaque, lipid and media models using a graphical method that allowed us to quickly visualize the results. For each pixel, we compared the predicted probability from the three models, and assigned the tissue category that had the highest probability. Once a category was assigned, we re-constructed the image by sorting the pixels and graphing them using a different color for each tissue type, as shown in Fig. 7.

and false positive classication rates for every possible classication threshold, where for a given threshold: False Positive Rate Total number of (Predicted = 1 and Y = 0) = Total number of (Y = 0) True Positive Rate Total number of (Predicted = 1 and Y = 1) = Total number of (Y = 1) For example, if the threshold is very high, then we will have very few false positives, but as a result, we will also have a small number of true positives. On the other hand, if we set a very low threshold, we will have a large number of true positives, but we will expect a larger number of false positives. In order to compute the ROC

3. Results Table 1 summarizes the performance of the RIPNet models based on two different statistical measures: the maximum Kolmogorov-Smirnov statistic and the Gini coefcient measure aspects of the ROC curve computed on the testing sample. An ROC curve is a graphical representation of the trade off between the false negative

Fig. 5. Full dataset of tissue samples used. Each frame in the gure corresponds to a tissue slice that was prepared and analyzed according to the methods described in previous sections. From left to right, the images in each section of the gure are: Histological sections (used to establish ground truth on training samples), MR false color images (red: PD, green: T1, blue: T2), expert labeling (used for NN training), NN model classication (yellow: hard plaque, red: media, blue: Lipid-rich plaque). Note: the slides on the right-hand side show two separate tissue samples that were xed on the same slide. They were treated as independent samples in this study. Our experts were unable to fully label images 7 and 9, due to poor MRI image quality, so they were excluded from the training sets. The model outputs for 7a and 9a are quite encouraging in this respect, as the models we able to nd reasonable tissue boundaries on Figs. 7a and 9a, exceeding the performance of human experts unaided by histology. However, both human experts and the models failed to nd boundaries for 7b and 9b.

Please cite this article in press as: Anderson, R.W. et al., Automated classication of atherosclerotic plaque from magnetic resonance images using predictive models, BioSystems (2007), doi:10.1016/j.biosystems.2006.11.005

+Model BIO-2709;
8

No. of Pages 11

ARTICLE IN PRESS
R.W. Anderson et al. / BioSystems xxx (2007) xxxxxx

Fig. 6. Model performance: labeled ground-truth is shown in the left panel and the modeling predictions are shown in the right panel for the same images. Muscle appears red, lipids appear blue, and plaque appears yellow in labeled images and in model results. Some regions of this image contain saturation artifacts. Note: due to lack of data, no model was developed to developed to identify thrombus (prominent in Fig. 6b). Interestingly, pixels containing thrombus did not register highly in any of the three tissue-type models (plaque, lipid, or media). This indicates that the characteristic features of thrombus are quite distinct; indicating that, given sufcient data, discrimination of thrombus from these tissue types should not be technically difcult.

curves shown in Figs. 4ac, we have taken threshold values between 0 and 1 in increments of 0.01. The ROC curves from which these results are derived are shown in Fig. 4. Models based on the K-means and standard logistic regression are used as a basis for comparison. The ROC curve for K-means clustering was created by ranking the ve clusters performance in classifying the three tissue types. As the best-performing cluster segmentation obtained had ve clusters; hence, there are only four inection points in this performance curve. The RIPNet models universally perform between 25 and 30% better in absolute terms than K-means. This translates to a 50% higher true positive rate over a given level of false positives. The combined model results are shown in last image in Fig. 5 (larger version in Fig. 6). The labeled groundtruth is shown in the left panel and the modeling predictions are shown in the right panel for the same images. Muscle appears pink in the labeled images, and red in the model results. Lipids appear white in the labeled images, and blue in the model results, while plaque appears yellow in both images. Each pixel is assigned to a category depending on which model generates the highest probability for that pixel. Pixels with below 30% probability for all of the model predictions are coded as black. Fig. 6a provides a clear example of the capabilities of the predictive models on these images. There is a high degree of correspondence between the pixels labeled plaque in the original image, and those labeled plaque by the models. Likewise, muscle is well identied. Fig. 6b demonstrates of the ability of the models to detect not only the hard plaques, but also the lipid areas in the image. The muscle areas are not uniformly identied, but this occurs in precisely those areas where the original

image suffered saturation artifacts and where the muscle wall is thin. Interestingly there are some false-positives coming from the muscle model along areas of the brous cap enclosing the lipid core shown in this image. Over 400 variables were included in this analysis, and thousands of network nodes were created from these variables. Out of all of these nodes, 140 were selected for each model. Table 2 illustrates the top several nodes selected for each model. An examination of these nodes illustrates the benets of using an automated technique over other approaches. For example, the plaque model contains mainly T1 and PD variables, and for almost all of the included variables a 7-pixel neighborhood measure was selected over all of the others available. These types of selections would have been almost impossible to reproduce manually without enormous effort. Likewise, many of the combinations of variables are not obvious to the human eye. For example, the top node for the lipid model is a linear combination of the max of the four-pixel neighborhood for T2, the discrete cosine transform of PD, and the anomaly variablecombinations for which

Fig. 7. Tissue segmentation algorithm. Plaque burden estimates were obtained by comparing the ratio of pixels classied as plaque vs. the number of pixels within the wall. In these examples, plaque burdens are estimated to be 28% and 62%, respectively.

Please cite this article in press as: Anderson, R.W. et al., Automated classication of atherosclerotic plaque from magnetic resonance images using predictive models, BioSystems (2007), doi:10.1016/j.biosystems.2006.11.005

+Model BIO-2709;

No. of Pages 11

ARTICLE IN PRESS
R.W. Anderson et al. / BioSystems xxx (2007) xxxxxx 9

Table 2 Top relevant inputs Plaque Model Rank 1 2 3 4 5 6 Type Node Node Node Node Node Node Description K-means classication T1 mean of 7 pixel square area PD median of 7 pixel square area T1 minimum of 3 pixel area T1 minimum of 7 pix PD discrete cosine transform of 7 pixel square

more challenging ones for our experts to label in the rst place, so it makes for a challenging test. 3.1. Leave-one-out cross-validation A legitimate concern regarding our earlier results (Anderson et al., 2003) was that the data used for model validation came from hold-out sample pixels from the same subject; in other words, the models have not been cross-validated on a fully independent sample. As already mentioned, only 10 specimens had been imaged under the same conditions. Of these, only three images had signicant lipid deposits and one with signicant thrombus. However, seven images contained plaque deposits, which was sufcient to conduct a leave-one-out cross-validation (Lachenbruch and Mickey, 1968). Methods: Seven standard logistic regression models for plaque identication/prediction. In this approach one image out of the total seven images containing plaque was retained as validation data for testing the model and the remaining six images were used as training data. The cross-validation process was then repeated seven times with each of the seven images used exactly once as the validation data For simplicity, K-means cluster membership was not used as an input variable. Results: An estimate of the model performance was then calculated by averaging the seven KolmogorovSmirnov statistics. The results are as follows:
KS Training Validation 0.70 0.62 STD 0.020 0.089

Lipid Model Rank 1 Type Hidden Unit Description T2 max of 4 pixel square area PD DCT 5 pixel square area Anomaly T1 mean of 3 pixel square area PD max of 3 pixel square area Luminance(CMYK transform) PD discrete cosine transform of 5 pixel square area Anomaly variable T2 range of 3 pixel square area PD variance of 4 pixel square area PD Inter-Quartile Range of 5 pixel area PD discrete cosine transform of 6 pixel square are Anomaly

Hidden Unit

Hidden Unit

Muscle Model Rank 1 2 3 4 Type Node Node Node Hidden Unit Description Anomaly Anomaly T2 variance 4 pixel square l area Anomaly PD skewness 3 pixel square area T13 pixel square area T25 pixel square area Chromiance (YCbCr Transform) PD discrete cosine transform of 7 pixel square area Anomaly variable

Considering the small sample size, the degradation of performance is modest. Furthermore, the validation performance still exceeds that of the best K-means performance. 4. Discussion Automated plaque characterization (validated by histology, a consensus of experts, or long-term clinical outcomes) would establish a more accurate, consistent, and objective measure for both longitudinal and cohort studies. Any misclassication biases introduced by automation would be systematic; hence, comparisons of plaque burden in serial investigations on the same patient will yield a reliable measure of relative changes. Without histology, classication of plaque from MR images remains somewhat subjective, but reasonably validated from long-term clinical outcomes (rupture, biopsy, stroke, and autopsy). One major advantage of

no clear explanation exists today. Certainly none of the usual heuristic methods would have uncovered these. The models do a reasonably good job separating muscle from plaque (Fig. 7). The performance on this image for the muscle model appears only slightly worse than for images in the estimation sample; the muscle model tracks the general outline of the artery wall, but as in other images there are signicant areas of drop-outs. The plaque model, on the other hand, appears to identify the labeled plaque areas well. This image was one of the

Please cite this article in press as: Anderson, R.W. et al., Automated classication of atherosclerotic plaque from magnetic resonance images using predictive models, BioSystems (2007), doi:10.1016/j.biosystems.2006.11.005

+Model BIO-2709;
10

No. of Pages 11

ARTICLE IN PRESS
R.W. Anderson et al. / BioSystems xxx (2007) xxxxxx ysis. U.S. Provisional Patent led August 21, 2003, 60/497,375, U.S. Patent led August 21, 2004 10/923,124 (ISCHEM Corp). Bishop, C., 1995. Neural Networks for Pattern Recognition. Oxford University Press, New York. Cai, J.M., Hatsukami, T.S., Ferguson, M.S., Small, R., Polissar, N.L., Yuan, C., 2002. Classication of human carotid atherosclerotic lesions with in vivo multicontrast magnetic resonance imaging. Circulation 106, 13681373. Chao, H., Kerwin, W.S., Hatsukami, T.S., Hwang, J.N., Yuan, C., 2003. Detecting objects in image sequences using rule-based control in an active contour model. IEEE Trans. Biomed. Eng. 50, 705710. Clarke, S.E., Hammond, R.R., Mitchell, J.R., Rutt, B.K., 2003. Quantitative assessment of carotid plaque composition using multicontrast MRI and registered histology. Magn Reson Med. 50, 11991208. Corti, R., Fayad, Z.A., Fuster, V., Worthley, S.G., Helft, G., Chesebro, J., Mercuri, M., Badimon, J.J., 2001. Effects of lipid-lowering by simvastatin on human atherosclerotic lesions: a longitudinal study by high-resolution, noninvasive magnetic resonance imaging. Circulation 104, 249252. Fayad, Z.A., 2003. MR imaging for the noninvasive assessment of atherothrombotic plaques. Magn. Reson. Imag. Clin. N Am. 11, 101113. Fayad, Z.A., Fuster, V., 2001. Clinical imaging of the high-risk or vulnerable atherosclerotic plaque. Circ. Res. 89, 305316. Fayad, Z.A., Fuster, V., Nikolaou, K., Becker, C., 2002. Computed tomography and magnetic resonance imaging for noninvasive coronary angiography and plaque imaging: current and potential future concepts. Circulation 106, 20262034. Fayad, Z.A., Sirol, M., Nikolaou, K., Choudhury, R.P., Fuster, V., 2004. Magnetic resonance imaging and computed tomography in assessment of atherosclerotic plaque. Curr. Atheroscler. Rep. 6, 232242. Fuster, V., Fayad, Z.A., Badimon, J.J., 1999. Acute coronary syndromes: biology. Lancet 353 (Suppl. 2), SII5SII9. Garvey, W., 1984. Modied elastic tissue-Masson trichrome stain. Stain Technol. 59, 213216. Itskovich, V.V., Samber, D.D., Mani, V., Aguinaldo, J.G., Fallon, J.T., Tang, C.Y., Fuster, V., Fayad, Z.A., 2004. Quantication of human atherosclerotic plaques using spatially enhanced cluster analysis of multicontrast-weighted magnetic resonance images. Magn. Reson. Med. 52, 515523. Kerwin, W., Han, C., Chu, B., Xu, D., Luo, Y., Hwang, J.N., Hatsukami, T., Yuan, C., 2001. A quantitative vascular analysis system for evaluation of atheroscleriotic lesion by MRI. In: Medical Image Computing and Computer-assisted Intervention, 4th International Conference, Utrecht, Netherlands, pp. 786794. Lachenbruch, P.A., Mickey, R.M., 1968. Estimation of error rates in discriminant analysis. Technometrics 10, 111. Lloyd, S.P., 1982. Least squares quantization in PCM. IEEE Trans Inform Theory IT-2, 129137. MacNeill, B.D., Lowe, H.C., Takano, M., Fuster, V., Jang, I.K., 2003. Intravascular modalities for detection of vulnerable plaque: current status. Arterioscler. Thromb. Vasc. Biol. 23, 13331342. Mansard, C.D., Canet Soulas, E.P., Anwander, A., Chaabane, L., Neyran, B., Serfaty, J.M., Magnin, I.E., Douek, P.C., Orkisz, M., 2004. Quantication of multicontrast vascular MR images with NLSnake, an active contour model: in vitro validation and in vivo evaluation. Magn. Reson. Med. 51, 370379. Perez-Amaral, T., Gallo, G., White, H., 2003. A exible tool for model building: the relevant transformation of the inputs network approach (RETINA). Oxford Bull. Econ. Stat. 65, 821838.

having an automatic imaging analysis would be to signicantly reduce the number of patients and followup period to demonstrate the effectiveness of newly developed cardiovascular drugs. Furthermore, better knowledge of the composition of atherosclerotic lesions will allow for a more precise risk-stratication of the patients as well as to facilitate the selection of the most appropriate therapeutic approaches based in the intrinsic characteristics of the patient. We consider this effort on ex vivo data demonstrates that, under ideal conditions, plaque characterization can be automated with high delity. From a pattern recognition perspective, ex vivo data present several advantages over in vivo data. First, the resolution of the 9.4 T research MRI system (39 m3 ) is at least an order of magnitude greater than standard 1.5 T clinical systems (2 mm). Second, several image processing problems associated with in vivo imaging (intramodal image registration, anatomical context, and motion artifacts) are eliminated, allowing us to concentrate purely on the pattern recognition problem. Third, the ex vivo images can be validated directly from accompanying histological examinations, reducing the subjectivity of classication. Hence, these results will give an upper bound on what can be expected from an in vivo system. Another limitation of the data in this study was the lack of proper examples for all of the clinically-relevant pathologies, most notably thrombus and calcication. For example, there were only two examples of thrombus (both in Fig. 6b). These considerations make it clear that although the models work quite well within the connes of this limited dataset, further work will be required to complete this effort. Recent work by Wolf et al. (2005) indicates that automated detection of calcied plaques using multicontrast MRI may not be problematic. This work has served to prove that it is conceptually possible to build an automated system of plaque detection using MRI and data mining. Current efforts to collect sufcient multimodal in vivo images of arterial plaques are underway to investigate the feasibility of noninvasive detection and classication of plaque using standard, clinical (1.5 and 3 T) machines. Acknowledgement The authors wish to thank Halbert White, Ph.D. for his assistance with model estimation and statistical testing. References
Anderson, R.W., Brotherton, T.W., Huizenga, J., 2003. Automated Methods and Systems for Vascular Plaque Detection and Anal-

Please cite this article in press as: Anderson, R.W. et al., Automated classication of atherosclerotic plaque from magnetic resonance images using predictive models, BioSystems (2007), doi:10.1016/j.biosystems.2006.11.005

+Model BIO-2709;

No. of Pages 11

ARTICLE IN PRESS
R.W. Anderson et al. / BioSystems xxx (2007) xxxxxx 11 Xu, D., Hwang, Yuan, C., 2002. Segmentation of multi-channel image with Markov random eld based active contour model. J. VSLI Signal Proc. 31, 4555. Yang, F., Holzapfel, G., Schulze-Bauer, C., Stollberger, R., Thedens, D., Bolinger, L., Stolpen, A., Sonka, M., 2003. Segmentation of wall and plaque in in vitro vascular MR images. Int. J. Cardiovasc. Imag. 19, 419428. Yuan, C., Lin, E., Millard, J., Hwang, J.N., 1999. Closed contour edge detection of blood vessel lumen and outer wall boundaries in blackblood MR images. Magn. Reson. Imag. 17, 257266. Yuan, C., Mitsumori, L.M., Beach, K.W., Maravilla, K.R., 2001a. Carotid Atherosclerotic Plaque: noninvasive MR characterization and identication of vulnerable lesions. Radiology 221, 285 299. Yuan, C., Mitsumori, L.M., Ferguson, M.S., Polissar, N.L., Echelard, D., Ortiz, G., Small, R., Davies, J.W., Kerwin, W.S., Hatsukami, T.S., 2001b. In vivo accuracy of multispectral magnetic resonance imaging for identifying lipid-rich necrotic cores and intraplaque hemorrhage in advanced human carotid plaques. Circulation 104, 20512056. Zieske, A.W., Malcom, G.T., Strong, J.P., 2000. Pathobiological determinants of atherosclerosis in youth (PDAY) cardiovascular specimen and data library. J. LA State. Med. Soc. 152, 296 301.

Perez-Amaral, T., Gallo, G., White, H., 2005. A comparison of complementary automatic modeling methods: RETINA and PcGets. Econometric Theory 21, 262277. Rao, K.R., Yip, P., 1990. Discrete Cosine Transform: Algorithms, Advantages, Applications. Academic Press, Boston. Shinnar, M., Fallon, J.T., Wehrli, S., Levin, M., Dalmacy, D., Fayad, Z.A., Badimon, J.J., Harrington, M., Harrington, E., Fuster, V., 1999. The diagnostic accuracy of ex vivo magnetic resonance imaging for human atherosclerotic plaque characterization. Arterioscler. Thromb. Vasc. Biol. 19, 27562761. Suri, J.S., Laxminarayan, S., 2003. Angiography and Plaque Imaging: Advanced Segmentation Techniques. CRC Press, Florida. Thomas, J.B., Rutt, B.K., Ladak, H.M., Steinman, D.A., 2001. Effect of black blood MR image quality on vessel wall segmentation. Magn. Reson. Med. 46, 299304. White, H, 1999. System and method for testing prediction model. U.S. Patent #5,893,069, Issued April 6, 1999. White, H., 2000. A reality check for data snooping. Econometrica 68, 10971126. Wolf, R.L., Wehrli, S.L., Popescu, A.M., Woo, J.H., Song, H.K., Wright, A.C., Mohler III, E., Harding, J.D., Zager, E.L., Fairman, R.M., Golden, M.A., Velazquez, O.C., Carpenter, J.P., Wehrli, F.W., 2005. Mineral volume and morphology in carotid plaque specimens using high-resolution MRI and CT. Arteriosclerosis, Thrombus, Vasc. Biol. 25, 17291735.

Please cite this article in press as: Anderson, R.W. et al., Automated classication of atherosclerotic plaque from magnetic resonance images using predictive models, BioSystems (2007), doi:10.1016/j.biosystems.2006.11.005