Article

American Journal of Evaluation

2014, Vol. 35(4) 467-484
ª The Author(s) 2014
Testing a Model of Participant Reprints and permission:
sagepub.com/journalsPermissions.nav
Retention in Longitudinal DOI: 10.1177/1098214014523822
aje.sagepub.com
Substance Abuse Research

Devin Gilmore1 and Gabriel P. Kuperminc1

Abstract
Longitudinal substance abuse research has often been compromised by high rates of attrition, thought
to be the result of the lifestyle that often accompanies addiction. Several studies have used strategies
including collection of locator information at the baseline assessment, verification of the information,
and interim contacts prior to completing the follow-up to minimize attrition, however it is unclear
whether these strategies are equally effective for participants struggling with varying levels of housing
stability, support for sobriety, and substance abuse severity. The current study extends research
supporting the effectiveness of follow-up strategies with a focus on locator form completion and
continual verification contacts. Results indicated that each additional piece of locator form information
and verification contact significantly and independently increased the odds for completing a follow-up
interview, and that these effects were not moderated by participant characteristics. Practical and
theoretical implications for longitudinal substance abuse research are discussed.

Keywords
attrition, substance abuse, longitudinal, retention, dropout

Drug and alcohol addiction have long been recognized by the health care system as chronic, long-term
conditions with multiple periods of relapse and recovery (Compton, Glantz, & Delaney, 2003). Accord-
ingly, federal agencies (Institute of Medicine, 1998; National Institute on Drug Abuse, 2004) have called
for studies evaluating the clinical and cost effectiveness of community-based drug treatment programs.
Evidence that patients presenting to emergency departments (EDs) have high rates of alcohol and illicit
drug abuse and dependence (Cherpitel, 1998; Vitale & Van de Mheen, 2006) has led to increased focus
on providing drug abuse treatment services in the ED (Blow et al., 2011). Brief interventions (BIs) have
been suggested as an efficient way to deliver such treatment in the ED, but research on their effectiveness
has been limited by methodological problems, including low follow-up rates (Cunningham et al., 2009).
These types of evaluations usually require longitudinal, repeated measures designs with pretest
and follow-up assessments to track change over time (Mowbray & Luke, 1996). The validity of

1
Georgia State University, Atlanta, GA, USA

Corresponding Author:
Devin Gilmore, Georgia State University, 140 Decatur St., 11th floor, Atlanta, GA 30303, USA.
Email: devingilmorecp@gmail.com
468 American Journal of Evaluation 35(4)

conclusions drawn from these studies is threatened by participant loss or attrition (Claus, Kindleber-
ger, & Dugan, 2002). If attrition is random, the only threat that results is a loss of statistical power
(Howard, Krause, & Orlinsky, 1986). However, because the degree and direction of bias resulting
from high attrition rates often remain unknown, attrition presents threats to both the internal and
external validity of longitudinal studies (Brown, 1990; Hansen, Collins, Malotte, Johnson, & Field-
ing, 1985; Shadish, Cook, & Campbell, 2002). For example, Scott (2004) examined the number of
attempts required to reach study participants as a predictor of variability in participant characteristics
and found that participants who took fewer attempts to reach differed substantially on outcomes such
as drug use and illegal activity. This suggests that the results of the study would have been biased if
the harder to reach group was not interviewed.

The Follow-Up Process

Tracking strategies to minimize participant attrition typically include phone calls (Meyers, Webb,
Frantz, & Randall, 2003), letters, Internet searches (Zand et al., 2006), and street outreach (Scott,
2004). Few studies have addressed causes of attrition specifically enough to draw firm conclusions
about the effectiveness of these strategies (Hansen, Tobler, & Graham, 1990). Some findings indi-
cate that telephone calls are more effective than letters; in-person contact (Nemes, Wish, Wraight, &
Messina, 2002); and searches through agencies such as social security, credit, and criminal justice
(Cottler, Compton, Ben-Abdallah, Horne, & Claverie, 1996). The effectiveness of these strategies
may also differ as a function of participant characteristics. For example, a study of adolescents found
that contact with friends and family members was the most effective strategy overall and that letters
and driver’s record searches were more effective for reaching homeless as compared to housed par-
ticipants (Hobden, Forney, Durham, & Toro, 2011). Homeless participants might receive letters at
temporary housing facilities, or through a friend or family member with a stable residence.
To maximize follow-up rates in studies with hard-to-reach populations, researchers often attempt
to gather extensive contact, or locator, information during the initial interview. This information
typically includes the participant’s own phone number and address, and information that can be used
to locate friends, family members, or others who might be able to locate the participant (Coen,
Patrick, & Shern, 1996).
Wright, Allen, and Devine (1995) used locator forms to track 670 homeless individuals with sub-
stance use problems and stressed the importance of verifying the information quickly after it has
been collected. They confirmed the accuracy of the information by calling the phone numbers given
within 24 hr after the initial interview. Cottler, Compton, Ben-Abdallah, Horne, and Claverie (1996)
advised in a study of substance abusers that contact information be collected from participants at the
beginning of the study, before they become impatient or tired. In a 4.5-year longitudinal study of
homeless adolescents, Hobden, Forney, Durham, and Toro (2011) collected contact information
from four friends or family members of the participant at intake and verified this information at each
point of contact with participants. Using this method, they found that numbers for friends and family
members (collateral contacts) were the most important predictor of follow-up completion and
advised that as much contact information as possible be collected at intake.
Summarizing the follow-up methodology used in several studies of patients enrolled in outpati-
ent, inpatient, and residential substance abuse treatment programs, Scott (2004) stressed the impor-
tance of continual efforts to track participants. Letters and telephone contacts were used at regular
intervals to keep in touch with participants. These procedures yielded follow-up rates of 70% during
a 21-month postintervention follow-up window, among participants who completed a 2-week court-
mandated Driving under the Influence (DUI) program (Kleschinsky, Bosworth, Nelson, Walsh, &
Shaffer, 2009). The sample for that study (detailed in Shaffer et al., 2007) was 88% White and
82% male, with 32% of participants earning less than US$20,000 per year.
Gilmore and Kuperminc 469

Most tracking procedures for hard-to-find individuals have emphasized collecting as much loca-
tor form information as possible at intake, including contact details from friends and family who
may help to locate the participant later in the study (Cottler et al., 1996; Dennis et al., 2002). Efforts
to maintain contact with participants throughout the study are also important elements of an effective
follow-up process (Scott, 2004). Of course, efforts to maintain contact must be balanced with sen-
sitivity to participants’ privacy needs, including awareness of when participants are avoiding contact
with researchers or wish to withdraw from the study.
Maintaining contact is particularly important in light of Hobden et al.’s (2011) findings that only
2% of homeless and 9% of housed participants responded to letters and that 80% of homeless parti-
cipants changed their contact information at some point during the study. At the same time, despite
lower response rates to letters among homeless participants, follow-up staff reported that letters
were more useful for contacting homeless participants than for housed participants. These findings
indicate that the effectiveness of specific tracking procedures may vary as a function of participant
characteristics.

Participant Characteristics and Circumstances

Research has examined how different participant characteristics and circumstances affect the ability
of researchers to locate individuals for follow-up. Attrition analyses often focus on understanding
and correcting biases resulting from dropout or item nonresponse related to participant circum-
stances (e.g., housing status and substance abuse severity) and demographic characteristics (e.g.,
race and age). Drug-using participants may be especially difficult to reach due to circumstances such
as housing instability and criminal activity (Bale, Arnoldussen, & Quittner, 1984), disconnection
from friends and family (Ziek, Beardsley, Deren, & Tortu, 1996), unemployment (Cottler et al.,
1996), and substance reuse after treatment (Nemes et al., 2002; Walton, Ramanathan, & Reischl,
1998).
Studies have often yielded contradictory results regarding how participant characteristics affect
attrition (Hobden et al., 2011). Bale, Arnoldussen, and Quittner (1984) found that employed parti-
cipants were more difficult to reach, while Cottler et al. (1996) found that unemployed participants
were more difficult to reach. Nemes et al. (2002) and Walton, Ramanathan, and Reischl (1998)
found contradictory results for the effects of age, employment, and marital status, but both studies
found that substance reuse after treatment was associated with increased contact difficulty.
For people struggling with drug abuse, the underlying mechanisms behind follow-up difficulty
may not be fully explainable in terms of measurable participant characteristics and observable cir-
cumstances. Drug-using participants undergoing treatment may cycle back and forth between a
‘‘clean’’ network and a substance-using network of individuals (Scott, 2004), and the clean network
may be more helpful and reliable for follow-up contact than the using network.
Research is needed to more fully explore how tracking strategies might vary in effectiveness for
participants differing in levels of substance abuse, access to housing, and support for sobriety. Such
information can be used to tailor tracking strategies to different groups of participants, as well as to
better understand and explain participant attrition.

Follow-Up Strategies for Transient, Drug-Using Participants

Individuals who are heavy drug users are often highly mobile and disorganized. At the same time,
pressure to conceal drug use and associated illegal activity from authorities may result in transient
living arrangements that make them more difficult to locate. These circumstances are thought to
contribute to high drop-out rates in longitudinal studies (Ziek et al., 1996). To address these chal-
lenges, Scott (2004) developed the Engagement, Verification, Maintenance, and Confirmation pro-
tocol, and has used it to manage participant tracking and data collection for several outcome studies
470 American Journal of Evaluation 35(4)

of residential inpatient, intensive outpatient, and methadone maintenance programs. The protocol
uses an extensive interview to gather contact information and periodic contacts throughout the
course of the study to continually update and refine this information. Each contact is referred to
as a ‘‘milestone’’ and any failure to reach a participant for a given milestone triggers a set of inten-
sive tracking procedures, such as street outreach and database searches, which continue until the
milestone is completed.

The Present Study

In summary, attrition has long presented a problem in longitudinal research; however, efforts to
identify mechanisms of attrition are relatively recent. In this article, we propose a model of partici-
pant attrition that accounts for the interactions between follow-up process milestones (locator form
completion and verification calls) and key participant characteristics and circumstances thought to
contribute to differential attrition in populations with severe substance abuse issues. We test this
model using data from an evaluation of a substance abuse treatment program conducted in two urban
EDs. We analyzed participant and follow-up process data using ordinary least squares (OLS) and
logistic regression to test whether success at early stages of the follow-up process was associated
with longer term follow-up, and if so, whether the process is equally effective for participants with
different characteristics or circumstances. Finally, we discuss implications of these findings for
longitudinal studies of hard-to-reach populations.

Hypotheses
We examined two main hypotheses. First, we hypothesized that completing each milestone of the
follow-up process is independently associated with an increased chance of completing subsequent
milestones. Second, we hypothesized that participant circumstances (substance abuse severity, hous-
ing stability, and support for sobriety) would moderate these associations. Specifically, we expected
that the effect of each milestone on completion of the next milestone would be attenuated for parti-
cipants with high substance abuse severity, low housing stability, and low support for sobriety. Pre-
vious research has linked these circumstances to attrition in longitudinal substance abuse research.
In particular, substance abuse severity contributes to housing instability and decreased connection
with people who support sobriety (Ziek et al., 1996). In turn, some evidence suggests that housing
instability and social connections play a role in the effectiveness of some forms of participant track-
ing (Hobden et al., 2011; Scott, 2004).

Method
In this section, we describe the study sample, the intervention context, and the follow-up procedures
used to collect repeated measures data on participant outcomes. We also describe the collection and
operationalization of key measures, including follow-up process milestones and sample circum-
stances/characteristics. We also discuss the amount and type of data missing from intake files,
including statistical methods to account for the missing data.

Sample
The sample consisted of ED patients (N ¼ 555) at two large medical centers in the Southeastern
United States who were enrolled in the evaluation of a substance abuse intervention project during
an 18-month period from October 1, 2009, through March 29, 2011. To enter the study, patients pre-
sented to the ED with a range of physical and mental complaints, such as chest or abdominal pain,
dizziness, shortness of breath, hallucinations, and exposure to STDs. Of these 555 individuals, we
omitted 33 participants because they had no locator information. We omitted an additional 15
Gilmore and Kuperminc 471

participants because an off-site follow-up team completed these interviews and we could not verify
details of the efforts we used locating them for follow-up. Additionally, 25 participants withdrew
from the study and 7 participants died before their follow-up interview. These omissions resulted
in a final sample of n ¼ 475. Exclusion of these individuals largely did not affect the sample com-
position: Being dropped was unrelated to gender, years of education, age, or income, but African
Americans were less likely than others to be dropped from analysis, w2(1) ¼ 8.51, p  .01.

Intervention Context
The present study was conducted in the context of an intervention that consisted of screening, treat-
ment, and a follow-up process. Screening, treatment, locator form completion, and random selection
were performed by health educators (HEs) at the two sites while most participant tracking proce-
dures were handled by the evaluation team. Most HEs had bachelor’s or master’s degrees in coun-
seling or related fields and were not licensed at the time they were hired. HEs received 3 days of
training on data collection, consent protocols, and motivational interviewing (MI). Coaches also pro-
vided additional training on more advanced MI topics quarterly. In addition, an evaluation team
member served as a liaison to each of the hospital-based teams and attended monthly meetings to
provide ongoing training and share feedback on the follow-up process. Fidelity of MI implementa-
tion was monitored by Motivational Interviewing Network of Trainers (MINT)-certified supervisors
who reviewed audiorecorded MI’s for each HE twice monthly. This ensured that HEs achieved and
maintained MI proficiency over the course of the intervention.

Screening, treatment, and random selection. HEs administered a brief screening tool measuring alcohol
and drug use (described subsequently) to determine participant eligibility for treatment. Based on the
screening score, patients received either a BI, brief therapy (BT), or a referral to treatment (RT) from
HEs on site. BIs consisted of a single MI session, while BTs consisted of a series of MI sessions
scheduled over a period of weeks, at the convenience of patients and HEs. HEs also conducted a
brief MI session with RT patients who were then referred to treatment programs where their stay
was paid for by the project. We drew a 10% random sample from patients who scored high enough
to receive treatment (BI, BT, or RT) using the last two digits of their Social Security number.
Patients who we randomly selected and who consented to participate in the follow-up study were
given a full assessment, which includes the Alcohol, Smoking, and Substance Involvement Screen-
ing Test (ASSIST) and other measures such as support for sobriety, readiness to abstain from alcohol
and drugs, and quality of life. HEs provided participants with an appointment card with the project’s
logo and contact information to remind them of their enrollment in the follow-up study.

Follow-up procedures. The evaluation team received weekly packets of information from study parti-
cipants, which contained signed consent forms, locator forms, and all participant intake data. Within
approximately 1 week, callers attempted to confirm participants’ contact information and remind
them of their participation. We sent participants a US$5 gift card for verifying this information.
Three months after intake, we called participants again to ensure their contact information was still
up to date. We contacted participants to complete their follow-up interview 5–8 months after intake,
for which they received a US$15 gift card (US$20 total).

Measures
Here, we discuss measures of follow-up process completion, participant characteristics, and partici-
pant circumstances. Follow-up process completion includes the amount of contact information col-
lected at intake, and whether we were able to contact participants at each of two scheduled
verification calls and the follow-up interview. Participant characteristics include stable traits: age,
472 American Journal of Evaluation 35(4)

race, gender, and years of education. Participant circumstances include substance abuse severity,
housing stability, and support for sobriety.
Three spaces on the locator form were designated as participant home, cell, and work numbers.
These were considered primary numbers. Four spaces for friends, family members, and institutional
contacts were designated in the locator form as collateral numbers. Locator form completion was
calculated as the sum of primary and collateral telephone numbers at intake.
Completion of verification calls and the follow-up interview were measured dichotomously (1 ¼
completed, 0 ¼ not completed). The first verification call completed up to 3 months postintake was
counted as a completed 7-day verification contact.
Once participants completed their 7-day verification, evaluation team staff stopped all contact
attempts until 3 months postintake. When staff members were unable to reach a participant for a
7-day verification, they continued contact attempts until the verification was completed or until a
decision was reached to discontinue contact attempts (e.g., a family member asked us not to call
again). Verification calls completed between 3 and 5 months postintake or completed after the
7-day verification contact were counted as a completed 3-month verification contact. The procedure
for completing the 3-month verification was the same as the 7-day verification, except that no gift
card was sent to participants who already received a US$5 gift card. If participants completed all or
part of their follow-up interview (conducted 5–8 months after intake), it was counted as completed
for the purpose of this analysis.

Housing stability. Previous studies have identified consistency and independence of housing as key
factors underlying overall housing stability (Bebout, Drake, Xie, McHugo, & Harris, 1997; Shinn
et al., 1998). A 4-point index (0–3) was created to assess housing stability by summing responses
from the following three questions on the locator form in reference to the address the participant
provided. The first question, ‘‘Whose place is it?’’ assessed whether patients reported their primary
address to be their own residence, or someone else’s (0 ¼ someone else’s residence; 1 ¼ patient’s
own residence). Staying at one’s own residence is considered independent housing. The second
question, ‘‘For about how long have you been at this address?’’ was derived from Shinn et al.’s
(1998) definition of housing stability and was used as a measure of residential consistency. Staying
at the listed address for 1 year or more was rated as consistent, and any length of time less than 1 year
was rated as inconsistent. The four answer choices were transformed to a dichotomous variable (0–6
months or 6–12 months ¼ 0; 1 year or more ¼ 1). The third question, ‘‘Do you stay at this address
every night?’’ assesses residential consistency (0 ¼ no; 1 ¼ yes).

Substance abuse severity. The ASSIST, version 3.0 is a screening tool developed by the World Health
Organization for use by health professionals in assessing a patient’s level of substance-related risk.
The ASSIST consists of seven questions about frequency of and problems related to use of each of
10 different types of substances: tobacco, alcohol, cannabis, cocaine, opiates, inhalants, sedatives,
hallucinogens (LSD, PCP, etc.), amphetamine-type stimulants, and ‘‘other’’ drugs that do not fit into
the above categories (World Health Organization, 2006). The scores for each substance-specific
question except for frequency of usage are summed and a numerical score is produced, representing
addiction severity and risk for each substance. This score is referred to as the Specific Substance
Involvement score or simply ASSIST score (Humeniuek & Ali, 2006). This score was used to deter-
mine level of treatment: Patients scoring from 4 to 19 received a BI from HEs, patients scoring
20–26 received a BI plus up to 12 additional follow-up appointments (BT) with the HE, and patients
scoring 27 or higher received a BI plus referral to more intensive treatment (RT). For the purposes of
this analysis, the highest ASSIST score for alcohol or any drug except tobacco was used to represent
overall substance use severity. Substance-specific alcohol, cannabis, and cocaine scores, as well
as treatment assignments based on ASSIST scores, were included in preliminary model-building
Gilmore and Kuperminc 473

procedures. Information on the length and frequency of treatment utilization of patients in the BT
and RT groups was not available for all patients in the present analysis; however, data from Site
B indicated that 15–25% of patients in the RT group were engaging with the treatment providers
they were referred to.
The ASSIST has shown good internal consistency and construct validity (World Health Organi-
zation, 2006) and concurrent validity when compared with other established measures of substance
abuse such as the Mini International Neuropsychiatric Interview (Sheehan, Lecrubier, & Sheehan,
1998), r ¼ .76, p < .01, and measures of alcohol abuse such as the Alcohol Use Disorders Identifi-
cation Test (Saunders, Aasland, Babor, de la Fuente, & Grant, 1993), r ¼ .82, p < .001. In the current
study, internal consistencies for ASSIST subscales for the three most commonly used substances
were alcohol (a ¼ .76), cannabis (a ¼ .71), and cocaine (a ¼ .91).

Support for sobriety. Participants’ perceptions of the social support for reducing their drinking or drug
use were assessed with the question ‘‘How supportive are the following people in helping you reduce
your drinking or drug use?’’ Participants rated spouse/partner, parents, siblings, other family,
friends, neighbors, and other important people. Response choices were coded numerically as not
at all (0), a little (1), and a lot (2). The responses were summed to produce a score for overall support
for sobriety. This scale is based on items from the Ecological Assessment of Substance-abuse
Experiences (EASE; Matto, Miller, & Spera, 2005), and demonstrated adequate internal consistency
in the current study (a ¼ .87).

Demographic variables. Dichotomous demographic variables included gender (1 ¼ male, 0 ¼ female),

race (1 ¼ Black or African American, 0 ¼ White or Other), and intake site (1 ¼ Site A, 0 ¼ Site B).
We measured age and years of education as continuous variables.

Analytic Approach
Preliminary analysis focused on item nonresponse that ranged from 0 to 20%. Items with the highest
rates of missing data (e.g., ‘‘Do you stay there every night?’’) were part of the housing stability scale.
The pattern of nonresponse indicated that some interviewers left the question blank on most of the
intake surveys they completed. The three HEs with the highest rate of missing data on the item skipped
it on 51% of the surveys they completed, compared with a skip rate of 7.5% among all other HEs. It
seemed unlikely that this pattern of nonresponse was strongly correlated with participant homelessness
across the entire data set, as would be expected if the missing data resulted from participants’ hesitancy
to answer. To determine whether the missing data mechanism was dependent on the observed vari-
ables, the data were tested using Little’s (1988) missing completely at random (MCAR) test. The null
hypothesis of MCAR was rejected (w2 ¼ 2453.9, df ¼ 2252, p ¼ .002), so missing data methods under
the less restrictive missing at random assumption were used. Accordingly, multiple imputation was
used, creating 10 data sets with missing data imputed using Markov chain Monte Carlo methods with
Bayesian estimation in Mplus version 6.1 (Muthén & Muthén, 2010). Results were averaged across
data sets to produce the reported parameter estimates and standard errors.
For the main analysis, we constructed a follow-up process model in which we tested the effects of
each step (locator form completion, 7-day verification, and 3-month verification) on subsequent
steps. We controlled for previous steps and participant circumstances (support for sobriety, housing
stability, and substance abuse severity and demographic characteristics (age, gender, race, intake
site, and years of education).
We used OLS regression to test effects of participant characteristics on locator form completion.
We used logistic regression to examine models of 7-day/3-month verification and follow-up com-
pletion, which were binary outcome variables. We examined potential covariates for inclusion based
474 American Journal of Evaluation 35(4)

on the purposeful variable selection procedures (Hosmer, Lemeshow, & Sturdivant, 2013), in which
a series of univariate and multivariate analyses were conducted, with the goal of including in the
final model only variables that significantly predict the outcome variable and/or contribute to the
explanatory power of other variables as moderators. We tested our hypotheses about moderation
by creating interaction terms from mean-centered independent variables (Aiken, West, & Reno,
1991). We added interaction terms to the main effects models one at a time and tested them for sta-
tistical significance. These terms included hypothesized follow-up process (locator form completion
and 7-day/3-month verifications) by participant circumstance (support for sobriety, substance abuse
severity, and housing stability) interactions. Further, we tested all variables that were significant in
each main model for interaction effects with all other significant variables, even if they were not
specified by the main hypotheses (e.g., 3-month verification by site).
We tested mediation effects by estimating all main models simultaneously using a single path
analysis. This approach allows the testing of indirect effects through multiple dependent variables.
We used probit regression with a robust weighted least squares estimator. Muthén, du Toit, and Spi-
sic (1997) found in simulation studies that this estimator generally performed well with path analysis
models that include dichotomous or ordered categorical variables. The probit model uses a standard
normal distribution, allowing simultaneous estimation of multiple binary variables as dependent
variables and predictors, which is not possible using logistic regression.

Results
Results from the main effects models were consistent with our hypotheses, with each milestone sig-
nificantly associated with subsequent milestones. However, contrary to our hypotheses, the results
from moderation analysis indicated that these associations were not attenuated for participants with
low housing stability, low support for sobriety, or high substance abuse severity. We explored sig-
nificant associations between covariates, including race and site, and completion of milestones in
post hoc moderation and mediation analysis.

Descriptive Analysis
Of the 475 participants, 69% were contacted for the 7- to 30-day verification, 40% were contacted
for the 3-month verification, and 68% completed their follow-up interview (see Table 1). Seven- to
thirty-day verification was positively associated with 3-month verification and follow-up comple-
tion. Three-month verification was positively associated with follow-up completion. As shown in
Table 2, only about half of all participants stayed at their own residence, and over half were unem-
ployed at intake. Most participants earned no money from any source in the past month, and only
14% earned more than US$1,000 in the past month. Plots of the standardized residual variance in
locator form completion revealed no significant deviations from normality.

Predictors of Follow-Up Process Completion

Predictors of locator form completion. Predictors of locator form completion, including support for
sobriety, housing stability, substance abuse severity, intake site, age, gender, race, and years of edu-
cation were analyzed using linear regression (see Table 3). Intake site was the only significant pre-
dictor of locator form completion, with intake at Site A associated with .68 fewer pieces of locator
form information than Site B.

Predictors of 7-day verification completion. Each piece of locator form information increased the odds of
completing a 7-day verification by 53%, and each one-unit increase in housing stability increased the
odds by 30%.
 Table 1. Sample Means, Percentages, and Zero-Order Correlations.

Mean/% 1 2 3 4 5 6 7 8 9 10 11 12

1. Age (years) 42.22 — 0.10* 0.11* 0.00 0.11* 0.14** 0.01 0.12** 0.04 0.08 0.10* .16**
2. Race (1 ¼ Black) 77% — — 0.01 0.03 0.09 0.04 0.09 0.03 0.04 0.03 0.13** 0.33**
3. Gender (1 ¼ male) 71% — — 1 0.04 0.01 0.08 0.01 0.10* 0.03 0.02 0.06 0.01
4. Years of education 11.69 — — — 1 0.01 0.00 0.11 0.04 0.09 0.04 0.04 0.05
5. Locator form 1.82 — — — — 1 0.03 0.13** 0.02 0.16** 0.20** 0.19** 0.39**
6. Housing stability 2.03 — — — — — 1 0.05 0.21** 0.11* 0.05 0.08 0.11*
7. Support for sobriety 9.31 — — — — — — 1 0.05 0.11* 0.08 0.16** 0.25**
8. Substance abuse 16.83 — — — — — — — 1 0.06 0.04 0.09 0.15**
9. 7-day verification 69% — — — — — — — — 1 0.37** 0.45** 0.09
10. 3-month verification 40% — — — — — — — — — 1 0.41** 0.08
11. Follow-up interview 68% — — — — — — — — — — 1 0.01
12. Site (1 ¼ Site A) 51% — — — — — — — — — — — 1

Note. N ¼ 475.
*p < .05. **p < .01.

475
476 American Journal of Evaluation 35(4)

Table 2. Participant Characteristics by Follow-Up Completion Status.

Follow-up completion status

Not Complete (N ¼ 152) Complete (N ¼ 323)

Variable N % N %

Locator form completion

0 numbers 3 2 0 0
1 number 77 51 113 36
2 numbers 51 34 136 42
3þ numbers 19 13 74 23
Verifications completed
7-day verification only 44 29 114 35
3-month verification only 1 <1 17 5
Both completed 15 10 155 48
Neither completed 92 61 37 11
Employment status
Employed 54 36 96 31
Unemployed 96 64 94 69
Housing stability
Lives at own residence 65 47 152 52
Stays there every night 105 85 217 86
One year or more at residence 67 56 210 70
Monthly income
US$0 95 63 181 56
US$1–500 16 11 46 14
US$501–1,000 15 10 54 17
More than US$10,00 26 17 42 13
Substance abuse severity
Brief intervention 90 59 218 68
Brief treatment 24 16 47 15
Referral to treatment 38 25 51 16
Note. N ¼ 475.

Predictors of 3-month verification completion. Participants who were contacted for a 7-day verification
were approximately 7 times more likely be contacted for a 3-month verification than those who were
not, and each piece of locator form information increased the odds of a 3-month verification by 49%.

Predictors of follow-up completion. Participants were about 5 times more likely to be reached for
follow-up when they had been reached for the 7-day and 3-month verifications, while each piece
of locator form information increased the odds of completing a follow-up interview by 37%. Support
for sobriety also significantly increased the odds of follow-up completion, while African American
participants were more than twice as likely as White participants to complete their follow-up
interview.

Effects of the Follow-Up Process on Probability of Follow-Up Completion

To illustrate the effects of follow-up process characteristics at various points in the follow-up
process, statistically significant predictors of follow-up completion were converted to probabilities.
Figure 1 illustrates the increase in the probability of a follow-up interview as more locator form
information is collected and more verification contacts made. With both verification contacts
 Table 3. Predictors of Locator Form, 7-Day/3-Month Verification, and Follow-Up Completion in Linear and Logistic Regression.

Dependent variable

Locator form completion 7-day verification 3-month verification Follow-up completion

Predictor B 95% CI OR 95% CI OR 95% CI OR 95% CI

Locator form completion 1.53** [1.19, 1.97] 1.49** [1.18, 1.89] 1.37** [1.01, 1.86]
7-day verification 7.18** [4.17, 12.36] 4.64** [2.86, 7.52]
3-month verification 6.03** [3.29, 11.06]
Housing stability 1.30* [1.04, 1.61]
Support for sobriety 1.08* [1.02, 1.13]
Race (1 ¼ Black) 2.66* [1.51, 4.69]
Site (1 ¼ Site A) 0.68** [0.82, 0.54]
Note. N ¼ 475.
*p < .05. **p < .01.

477
478 American Journal of Evaluation 35(4)

100%

90%

No verification
Probability 80%

70%
7-day only
60%

50%
Both verifications
40%
1 2 3 4
Locator Form Completion

Figure 1. Probability of contacting participants at follow-up by locator form completion and number of
verification contacts.

completed, the probability of follow-up completion approaches 95%, independent of the amount of
locator form information collected.

Moderation Analysis
Housing stability, support for sobriety, and substance abuse severity were tested as potential mod-
erators of the effects of locator form completion on 7-day verification, 7-day verification on 3-
month verification, and 3-month verification on follow-up completion. Substance abuse severity
was tested as a moderator using the patient’s highest ASSIST score, substance-specific raw scores,
and dummy-coded treatment classification scores (i.e., BI, BT, and RT). Because race and site
emerged as statistically significant predictors of follow-up process completion, post hoc modera-
tion analyses were conducted to test for their potential moderating effects at each step of the
follow-up process. For the model predicting 3-month verification completion, there was a signif-
icant 7-day verification by race interaction (odds ratio [OR] ¼ .19, p ¼ .044). The interaction
effect was probed by reverse coding the race variable and creating a new interaction term. For
White/other participants, 7-day verification completion was more strongly associated with 3-
month verification (OR ¼ 27.5, p < .001) than for Black participants (OR ¼ 5.17, p < .001).
No other interaction terms reached significance, so it was concluded that the effects of locator
form completion, 7-day and 3-month verifications did not differ as a function of support for sobri-
ety, housing stability, or substance abuse severity.

Mediation Analysis
Locator form completion increased the probability of 3-month verification through its effect on
7-day verification (B ¼ .18, p ¼ .003) and also increased the probability of a follow-up interview
through its effect on both 7-day and 3-month verifications (B ¼ .09, p ¼ .008). Seven-day verifi-
cation had a significant indirect effect on follow-up completion through 3-month verification com-
pletion (B ¼ .37, p < .001), while intake at Site A was associated with a decreased probability of
follow-up completion through its effect on locator form completion and both verifications (B ¼
.06, p ¼ .014).
Gilmore and Kuperminc 479

Discussion
Individuals presenting to EDs and struggling with substance abuse are a difficult group to retain in
longitudinal research, due in part to factors related to the substance abusing lifestyle, such as housing
instability and social disconnection. Evaluators have developed follow-up protocols to anticipate
and prevent study attrition by emphasizing the collection of as much contact information as possible
from participants at intake (Dennis et al., 2002) and continuing contact with them throughout the
study (Scott, 2004; Wright, Allen, & Devine, 1995). The present study lends empirical support to
the effectiveness of these techniques. Specifically, each telephone number obtained at intake inde-
pendently increased the odds of completing 7-day and 3-month verifications by about 50%, while
these verifications each independently made completing a follow-up interview about 5 times more
likely. Even after controlling for the effects of verification contacts, each telephone number
increased the odds of completing a follow-up interview by 37%. Considering that the population
in the present study had moderate to high substance abuse severity and low housing stability, we
expect these findings to apply to a broad range of longitudinal outcome evaluations conducted with
populations where attrition remains a serious issue, such as those struggling with homelessness or
housing instability, in treatment for substance abuse, and those with limited social networks.
Even when the effects of locator form completion and verification contacts were accounted for,
analysis of the predictors of follow-up completion revealed that certain participant circumstances
continued to play a significant role in attrition. Participants with high support for sobriety were sig-
nificantly more likely to complete their follow-up interview than those with low support, even after
controlling for the effects of locator form completion and verification contacts. This indicates that
friends and family who are supportive of a participant’s recovery may be more helpful to the follow-
up process than those who are not supportive.
Follow-up rates in studies of ED-based substance abuse interventions vary widely, ranging from
approximately 20 to 80% (Cunningham, Walton, Outman, Murray & Booth, 2008). Studies with
lower follow-up rates have tended to have high proportions of hard-to-reach participants. For exam-
ple, Bernstein, Bernstein, and Levenson (1997) reached 22% of a sample of over 1,000 ED patients
60–90 days after intake, with homeless rates over 30%. Neuner et al. (2007) reached 63% of over
2,500 trauma patients and reported high rates of attrition among participants with high alcohol use
severity and low education. In contrast, studies with follow-up rates of around 80% have often had
smaller samples and excluded homeless or alcohol-dependent participants (Mello et al., 2005), and
others have not reported rates of homelessness in their sample (Cunningham et al., 2008). Clearly,
obtaining representative samples of hard-to-reach participants remains a challenge for longitudinal
substance abuse research.

Practical Implications for Substance Abuse Research in the ED Setting

These findings indicate that the collection of locator form information drives each step of the follow-
up process. Additionally, ongoing verification is necessary to retain participants. Successful verifi-
cation increases engagement of participants in the follow-up process, while also allowing the team to
identify participants who are not responsive to contact more quickly, so that intensive tracking pro-
cedures can be used. Taken together, these findings highlight the crucial importance of active, com-
prehensive efforts to motivate and engage participants throughout the study. For example,
participants may be reluctant to provide telephone numbers for friends or family members if (a) they
do not understand some part of the study (confidentiality, compensation, time, and commitment), (b)
do not want to participate in the study, or (c) do not have contact information available for those
individuals. Given the importance of locator form information, the initial engagement, motivation,
and informed consent stage of the study warrants a considerable investment of time and resources.
For other health interventions where the initial contact with participants is brief, this is an especially
480 American Journal of Evaluation 35(4)

important consideration. Without a long period of time to establish rapport with participants and
learn about their circumstances, this initial contact may be the only opportunity to engage partici-
pants and collect adequate locator form information for follow-up.
Ongoing monitoring and feedback is required to manage the follow-up process effectively. How-
ever the follow-up process is configured, effectively managing it requires continual information on
all aspects of the process from locator data collection to verification calls and housing status. Each
step of the process is important; it is not enough to complete a full locator form but fail to verify the
information later in the study. For example, mediation analyses confirmed that an incomplete locator
form affects the entire follow-up process by decreasing the probability of subsequent verification
contacts, which in turn make a follow-up contact less likely. When allocating time and resources
for longitudinal research, it is important to consider how each step of the follow-up process contri-
butes to subsequent steps. While it may be tempting to reduce staff workload by collecting minimal
locator information, attempting to reach participants with limited contact information can become
prohibitively expensive. Additional telephone calls, Internet searchers, and street outreach are likely
to be much more costly and time-intensive than extra time spent collecting detailed locator form
information. This issue is especially problematic for large samples, since coordinating street out-
reach for dozens of participants can quickly become overwhelming, even for large teams.
In the present study, the team had weekly updates on hard-to-find participants, their statuses
(incarcerated, no working number, homeless, etc.), as well as a detailed description of recent
efforts to track participants. These efforts are critical to ensure the follow-up process is being
implemented effectively and that participants are not ‘‘slipping through the cracks.’’ However,
it is also important to consider that participants and their family members may become uncomfor-
table with the more intensive tracking procedures, and this monitoring is also used to ensure that
tracking efforts do not become intrusive. It is recommended that follow-up teams pay close atten-
tion to participants with low support for sobriety or who are not responsive to verification contacts,
so that they can be reached using alternative strategies. Identifying and focusing special effort on
these participants as soon as possible will minimize the ripple effect that results from a missed
verification call or a lack of contact information. Many potential problems can be avoided by
keeping detailed records of participant contact, so that team members are informed of previous
attempts whenever they seek to contact a participant. For example, participants who wish to with-
draw from the study may repeatedly reschedule their follow-up interview instead of specifically
requesting to withdraw.
Evaluations of programs where treatment staff participate in the follow-up process must ensure
that engagement, motivation, consent, locator form completion, and verification procedures fit into a
cohesive follow-up process. In our project, the logistics of screening and treatment prevented the
evaluation team from having a direct role in the initial contact with participants, thus making the
treatment staff responsible for the initial steps of motivation, consent, and locator form completion.
Thus, it was necessary to dedicate substantial effort to maintaining full contact and coordination
between the sites and the evaluation team and ensuring that procedures were being followed
consistently.

Race Effects
Past research has not revealed a consistent link between contact difficulty and the age or race of
participants in substance abuse research. In the present study, African American participants were
more likely than others to complete their follow-up interview. However, this finding should be
interpreted with caution, since the wide confidence interval around the estimate suggests that some
unexplained source of variability may be contributing to the effect. Investigating a possible
Gilmore and Kuperminc 481

interaction between staff and participant race and its effect on attrition rates is an interesting ave-
nue for future research.
Moderation analysis revealed that while all participants were more likely to complete a 3-month
verification if they had completed a 7-day verification, this effect was stronger for White partici-
pants. As with the main effect of race, this effect should be interpreted with caution. Among all the
models tested, this interaction was the only one to reach conventional levels of statistical signifi-
cance. Nevertheless, these results suggest that there may be differences in how racial groups respond
to the follow-up process used in the present study. Future research is needed to replicate this finding
and inform modifications to ensure increased retention among all racial groups participating in long-
itudinal substance abuse research.

Limitations
An important limitation is that intensive follow-up procedures such as Internet searches, letters,
and street outreach were not measured in the current analysis. It has been our experience that these
methods contributed to successful contact with participants, but it is difficult to determine whether
a letter was actually received by a participant, unless the participant mentions the letter at some
point or the letter is returned unopened. One approach taken by Hobden et al. (2011) is to assign
importance ratings to these different methods to determine how helpful each technique is for dif-
ferent groups of participants. This technique is promising, since more than one technique may con-
tribute to follow-up success, and different raters can describe the relative priority of each.
Irrespective of the research method used, the effective study of the follow-up process requires sys-
tematic documentation of processes such as calls, letters, and Internet searches, as well as out-
comes such as verification calls, completed interviews, and contact with friends/family.
Researchers focused explicitly on studying the follow-up process should plan ahead for the diffi-
culties in measuring these outcomes and be able to account for the unique or unexpected events
that occur during the follow-up process.
Capturing the role of the changing circumstances of participants in this study was difficult as
well. Some participants were incarcerated during the study; however, accounting for this effect
was not possible since arrest data in the state were not consistently available. A similar issue
was raised when participants enrolled in long-term treatment or transitional housing, where the
team was often unable to contact them. Thus, some participants who became incarcerated or
enrolled in long-term residential treatment were lost to follow-up. Since the research design
used an intent-to-treat approach that assumed patients received the treatment they were
assigned to, the team was not able to determine the extent to which attrition was related (para-
doxically) to the project’s successfully linking participants to needed services. Future evalua-
tions of programs that refer participants to outside programs should anticipate the possible
effects of treatment engagement on attrition. If researchers can work with participants to plan
for follow-up contact before they enter long-term treatment, problems coordinating with mul-
tiple treatment providers can be avoided.

Future Directions
Additional research is needed to identify how existing tracking protocols can be adapted for use in
the ED setting. The main challenge for tracking this population is collecting sufficient locator infor-
mation in a time-constrained setting, while also providing effective treatment services. Various
approaches should be compared across different settings and populations to determine which prove
most effective. For example, an abbreviated locator form can be used to collect important informa-
tion quickly, while frequent incentives for participation could improve engagement. Studies system-
atically comparing the effectiveness of these techniques could prove valuable for researchers
482 American Journal of Evaluation 35(4)

designing longitudinal studies with limited resources. Randomizing participants into different track-
ing processes could facilitate the comparison of different techniques in terms of cost, effort, and
reduced attrition.
Longitudinal research could also benefit from an improved ability to predict dropout in advance.
Using statistical techniques such as survival analysis could provide a more detailed model of parti-
cipant dropout by showing when certain participants are likely to dropout, and detailed information
on contact with participants throughout the study could be used to identify tracking techniques that
are effective for those at high risk for dropout. For example, Internet searches for family members
may be an effective way to reach participants who remain engaged with their social support network.
More detailed information on hard-to-reach ED patients could help improve and standardize the
tracking protocols used in research with this population.

Conclusion
In conclusion, the present study has provided empirical evidence supporting the effectiveness of a
standardized follow-up protocol when implemented in an abbreviated format in a hospital emer-
gency room setting. These findings demonstrate that collecting adequate amounts of locator form
information and performing regular verification calls can help minimize attrition even when parti-
cipants are transient and have severe substance abuse problems, but extra effort may be required to
contact participants who lack social connections that support sobriety. This extra effort should entail
collecting more information on the locator form and paying special attention to participant engage-
ment throughout the study. By adapting the follow-up protocol to suit specific settings and partici-
pants, researchers can analyze different aspects of the process over time to better understand the
interaction between the follow-up process, participants, and their social networks.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publi-
cation of this article.

Funding
The authors declared the following financial support for the research, authorship, and/or publication of this arti-
cle: This research was funded by SAMHSA grant #TI019545. The views and opinions contained in this pub-
lication do not necessarily reflect those of SAMHSA or the U.S. Department of Health and Human Services,
and should not be construed as such.

References
Aiken, L. S., West, S. G., & Reno, R. R. (1991). Multiple regression: Testing and interpreting interactions.
Thousand Oaks, CA: Sage.
Bale, R. N., Arnolduseen, B. H., & Quittner, A. M. (1984). Follow-up difficulty with substance abusers:
Predictors of time to locate and relationship to outcome. Substance Use and Misuse, 19, 885–902.
Bebout, R. R., Drake, R. E., Xie, H., McHugo, G. J., & Harris, M. (1997). Housing status among formerly
homeless dually diagnosed adults. Psychiatric Services, 48, 936–942.
Bernstein, A., Bernstein, J., & Levenson, S. (1997). Project ASSERT: An ED-based intervention to increase
access to primary care, preventive services, and the substance abuse treatment system. Annals of Emergency
Medicine, 30, 181–189.
Blow, F. C., Walton, M. A., Barry, K. L., Murray, R. L., Cunningham, R. M., Massey, L. S., . . . Booth, B. M.
(2011). Alcohol and drug abuse among patients presenting to an inner-city emergency department: A latent
class analysis. Addictive Behaviors, 26, 793–800.
Gilmore and Kuperminc 483

Brown, C. H. (1990). Protecting against nonrandomly missing data in longitudinal studies. Biometrics, 46,
143–155.
Cherpitel, C. (1998). Differences in performance of screening instruments for problem drinking among blacks,
whites, and Hispanics in an emergency room population. Journal of Studies on Alcohol, 59, 420–426.
Claus, R. E., Kindleberger, L. R., & Dugan, M. C. (2002). Predictors of attrition in a longitudinal study of
substance abusers. Journal of Psychoactive Drugs, 34, 69–74.
Coen, A. S., Patrick, D. C., & Shern, D. L. (1996). Minimizing attrition in longitudinal studies of special
populations: An integrated management approach. Evaluation and Program Planning, 19, 309–319.
Compton, W. M., Glantz, M., & Delaney, P. (2003). Addiction as a chronic illness—putting the concept into
action. Evaluation and Program Planning, 26, 353–354.
Cottler, L. B., Compton, W. M., Ben-Abdallah, A., Horne, M., & Claverie, D. (1996). Achieving a 96.6 percent
follow-up rate in a longitudinal study of drug abusers. Drug and Alcohol Dependence, 41, 209–217.
Cunningham, R. M., Bernstein, S. L., Walton, M., Broderick, K., Vaca, F. E., Woolard, R., & D’onofrio, G.
(2009). Alcohol, tobacco, and other drugs: Future directions for screening and intervention in the emergency
department. Academic Emergency Medicine, 16, 1078–1088.
Cunningham, R., Walton, M. A., Outman, R., Murray, R., & Booth, B. M. (2008). Tracking inner city substance
users from the ED: How many contacts does it take? Academy of Emergency Medicine, 15, 136–143.
Dennis, M., Titus, J. C., Diamond, G., Donaldson, J., Godley, S. H., Tims, F. M., . . . Scott, C. K. (2002). The can-
nabis youth treatment (CYT) experiment: Rationale, study design, and analysis plans. Addiction, 97, 16–34.
Hansen, W. B., Collins, L. M., Malotte, C. K., Johnson, C. A., & Fielding, J. E. (1985). Attrition in prevention
research. Journal of Behavioral Medicine, 8, 261–275.
Hansen, W. B., Tobler, N. S., & Graham, J. W. (1990). Attrition in substance abuse prevention research: A
meta-analysis of 85 longitudinally followed cohorts. Evaluation Review, 14, 677–682.
Hobden, K., Forney, J. C., Durham, K. W., & Toro, P. (2011). Limiting attrition in longitudinal research on
homeless adolescents: What works best? Journal of Community Psychology, 39, 443–451.
Hosmer, D. W., Lemeshow, S., & Sturdivant, R. X. (2013). Applied logistic regression. Hoboken, NJ: John
Wiley.
Howard, K. I., Krause, M. S., & Orlinsky, D. E. (1986). The attrition dilemma: Towards a new strategy for
psychotherapy research. Journal of Consulting and Clinical Psychology, 51, 106–110.
Humeniuek, R., & Ali, R. (2006). Validation of the alcohol, smoking and substance involvement screening test
(ASSIST) and pilot brief intervention [electronic resource]: A technical report of phase II findings of the
WHO ASSIST Project. Retrieved November 9, 2010, from http://www.who.int/substance_abuse/activities/
assist_technicalreport_phase2_final.pdf
Institute of Medicine. (1998). Executive summary. In S. J. Lamb, M. R. Greenlick, & D. McCarty (Eds.),
Bridging the gap between practice and research: Forging partnerships with community-based drug and
alcohol treatment. Washington, DC: National Academy Press.
Kleschinsky, J. H., Bosworth, L. B., Nelson, S. E., Walsh, E. K., & Shaffer, H. J. (2009). Persistence pays off:
Follow-up methods for difficult-to-track longitudinal samples. Journal of Studies on Alcohol and Drugs, 70,
751–761.
Little, R. (1988). A test of missing completely at random for multivariate data with missing values. Journal of
the American Statistical Association, 83, 1198–1202.
Matto, H., Miller, K., & Spera, C. (2005). Ecological assessment of substance-abuse experiences (EASE):
Findings from a new instrument development pilot study. Addictive Behaviors, 30, 1281–1289.
Mello, M. J., Nirenberg, T. D., Longabaugh, R., Woolard, R., Minugh, A., Becker, B., . . . Stein, L. (2005).
Emergency department brief motivational interventions for alcohol with motor vehicle crash patients.
Annals of Emergency Medicine, 45, 620–625.
Meyers, K., Webb, A., Frantz, J., & Randall, M. (2003). What does it take to retain substance abusing
adolescents in research protocols? Delineation of effort required, strategies undertaken, costs incurred, and
6-month post-treatment differences by retention difficulty. Drug and Alcohol Dependence, 69, 73–85.
484 American Journal of Evaluation 35(4)

Mowbray, C. T., & Luke, D. (1996). Introduction to the special issue: Innovative methodologies for longitudinal
evaluation of human service programs. Evaluation and Program Planning, 19, 279–285.
Muthén, B. O., du Toit, S. H. C., & Spisic, D. (1997). Robust inference using weighted least squares and quad-
ratic estimating equations in latent variable modeling with categorical and continuous outcomes. Retrieved
May 19, 2012, from http://www.gseis.ucla.edu/faculty/muthen/articles/Article_075.pdf
Muthén, L. K., & Muthén, B. O. (2010). Mplus version 6.1. Los Angeles, CA: Author.
National Institute on Drug Abuse. (2004). Report of the blue ribbon task force on health services research
at the National Institute of Drug Abuse. Retrieved April 2, 2010, from http://www.drugabuse.gov/
about/organization/nacda/HSRReport.pdf
Nemes, S., Wish, E., Wraight, B., & Messina, N. (2002). Correlates of treatment follow-up difficulty. Substance
Use and Misuse, 37, 19–45.
Neuner, B., Fleming, M., Born, R., Weiss-Gerlach, E., Neumann, T., Rettig, J., . . . Spies, C. (2007). Predictors
of loss to follow-up in young patients with minor trauma after screening and written intervention for alcohol
in an urban emergency department. Journal of Studies on Alcohol and Drugs, 68, 133–140.
Saunders, J. B., Aasland, O. G., Babor, T. F., de la Fuente, J. R., & Grant, M. (1993). Development of the
alcohol use disorders identification test (AUDIT): WHO collaborative project on early detection of persons
with harmful alcohol consumption—II. Addiction, 88, 791–804.
Scott, C. K. (2004). A replicable model for achieving over 90% follow-up rates in longitudinal studies of
substance abusers. Drug and Alcohol Dependence, 74, 21–36.
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for
generalized causal inference. Boston, MA: Houghton-Mifflin.
Shaffer, H. J., Nelson, S. E., LaPlante, D. A., LaBrie, R. A., Albanese, M., & Caro, G. (2007). The epidemiol-
ogy of psychiatric disorders among repeat DUI offenders accepting a treatment-sentencing option. Journal
of Consulting and Clinical Psychology, 75, 795–804.
Sheehan, D. V., Lecrubier, Y., & Sheehan, K. H. (1998). The mini international neuropsychiatric interview
(MINI): The development and validation of a structured psychiatric diagnostic interview for DSM-IV and
ICD-10. Journal of Clinical Psychiatry, 59, 22–33.
Shinn, M., Weitzman, B. C., Stojanovic, D., Knickman, J. R., Jiménez, L., Duchon, L., . . . Krantz, D. H.
(1998). Predictors of homelessness among families in New York City: From shelter request to housing sta-
bility. American Journal of Public Health, 88, 1651–1657.
Vitale, S., & Van de Mheen, H. (2006). Illicit drug use and injuries: A review of emergency room studies. Drug
and Alcohol Dependence, 82, 1–9.
Walton, M. A., Ramanathan, C. S., & Reischl, T. M. (1998). Tracking substance abusers in longitudinal
research: Understanding follow-up contact difficulty. American Journal of Community Psychology, 26,
233–253.
World Health Organization. (2006). Validation of the alcohol, smoking and substance involvement screening
test (ASSIST) and pilot brief intervention: A technical report of phase II findings of the WHO ASSIST
project. Retrieved October 12, 2010, from http://www.who.int/substance_abuse/activities/assist_technical
report_phase2_final.pdf
Wright, J. D., Allen, T. L., & Devine, J. A. (1995). Tracking non-traditional populations in longitudinal studies.
Evaluation and Program Planning, 18, 267–277.
Zand, D., Thomson, N. R., Dugan, M., Braun, J. A., Holterman-Hommes, P., & Hunter, P. L. (2006). Predictors
of retention in an alcohol, tobacco, and other drug prevention study. Evaluation Review, 30, 209–222.
Ziek, K., Beardsley, M., Deren, S., & Tortu, S. (1996). Predictors of follow-up in a sample of urban crack users.
Evaluation and Program Planning, 19, 219–224.