Discussion

The aim of this study was to quantitatively review all available randomized
controlled trials of OST. To the best of our knowledge, this is
the first study to employ a quantitative review of organizational skills
interventions for children with ADHD.We located 14 randomized controlled
studies of OST that met our inclusion criteria. Twelve studies involving
1054 children (576 treatment, 478 control) provided data that
was included in the quantitative syntheses (i.e., meta-analyses).
We evaluated outcomes in three domains of interest: (a) organizational
skills (parent- and teacher-rated), (b) inattention (parent-and
teacher-rated), and (c) academics (teacher-rated academic performance
and student's GPA). Results showed significant effects of OST
across all outcomes. Overall, significant effects of large to moderate
magnitude were found for organizational skills outcomes for children
when assessed by teacher-rated (g = 0.54) and parent-rated (g =
0.83) measures, and for inattention outcomes for children when
assessed by parent-rated measures (g=0.56). The effects for teacherrated
inattention and academic performance, and student's GPA were
statistically significant, although smaller in magnitude: g = 0.26, g =
0.33, and g = 0.29, respectively. It is likely that the magnitude of ES
was related to differences among the studies in terms of specific features
of OST treatments, outcome measures, informants, and control
conditions. A relatively small number of studies available for this
meta-analysis did not allow us to investigate potential mediator variables
of treatment effects. Thus, we discuss possible differences among
the studies that might have contributed to the differences in magnitude
of the observed ES.
The ES were the highest for the domain of parent-rated organizational
skills. Four of six studies (Abikoff et al., 2013; Langberg et al.,
2008b; Langberg et al., 2012; Pfiffner et al., 2014) reported large ES
(range g = 0.85 to 2.00). It is possible that the smaller ES reported by
Evans et al. (2016) and Power et al. (2012), g=0.20 and g = 0.27, respectively,
were due to a sole focus on organizational skills required
for homework (e.g., using a homework checklist) while organizational
skills in the home were not directly targeted during the intervention.
The duration of treatment in the Power et al. (2012) study was also relatively
short: 12 sessions versus 16 to 20 sessions in other studies. The
aspect of duration of OST was not addressed in this meta-analysis or
in the included individual randomized controlled trials, but future studies
might test the effects of treatment duration/intensity on the outcomes.
Most importantly, the Power et al. (2012) and Mautone et al.
(2012) studies compared OST to an active control condition, “Coping
with ADHD through Relationships and Education” (CARE). The 12-session
CARE program provided support and education to parents including:
(a) discussing children's progress at home and school, (b)
establishing a context within which parents can support each other in
coping with their children's difficulties, and (c) providing education to
parents about ADHD. The ES might be smaller because the CARE controlled
for treatment factors such as attention from the therapists and
education about ADHD.
There was greater heterogeneity in ES in the domain of the teacher
ratings of organizational skills, ranging from large (Abikoff et al.,
2013) to moderate (Pfiffner et al., 2014) and small (Langberg et al.,
2012; Power et al., 2012). OST interventions in the studies with large
and moderate ES (Abikoff et al., 2013; Pfiffner et al., 2014) consisted
of approximately 20 sessions and had more extensive teacher components
than in the interventions in the Langberg et al. (2012) and
Power et al. (2012) studies, which might explain the differences in ES
across studies. However, all OST packages can be viewed as multicomponent
interventions and the difference in relative contributions
of child-, parent- and teacher-focused components to the overall ES, cannot be discerned fromthe
existing studies. Future studies can utilize
dismantling designs to evaluate relative effects of the unique components
and techniques that comprise multi-componential OST interventions.
Furthermore, organizational skills are complex behaviors and not
unitary skills. It is unknown how the OST interventions affect specific
subsets of organizational skills as very fewstudies reported specific subscales
and including subscales in meta-analyses is methodologically
challenging.
The ES of parent-reported improvement in attention was moderate
while the ES of teacher-reported gain in attention was small. The ES
for parent-rated attention were highly heterogenic across studies ranging
fromminimal (g=0.08) to large (g=1.02). The majority of studies
that used waitlist control conditions found moderate to large effects of
OST on inattention symptoms. In contrast, two studies that compared
OST to a community control condition (Evans et al., 2011; Evans et al.,
2014) reported small effects. In these studies families received a packet
of contact information for providers in their local community and a
summary of the intake evaluation was sent to the school psychologists
at their respective schools. However, it is unknown how many participants
in the control group initiated treatment during the intervention.
Three of the six studies that reported teacher-rated outcomes of inattention
also incorporated school personnel directly or indirectly into
theOST treatment process. Thus, in the “ChallengingHorizons Program”
(Evans et al., 2011), undergraduate counselors communicated with the
teachers on a biweekly basis about students' progress and behavior in
classroom. The “Supporting Teens' Academic Needs” program (Sibley
et al., 2013), included one school consultation and parents were taught
howto bemore involved with the schools and engage teachers. Despite
these teacher-directed portions of the OST, the teacher-report did not
reveal significant improvements in ADHD symptoms (while parent-reports
did). It is possible that improvements in attention noted by the
parents were not potent enough to manifest in the classroom or be salient
to the teachers. However, it is worth commenting that one study
that reported the largest effect on the teacher-rated attention (Pfiffner
et al., 2014) (g=0.79) also included a comprehensive training component
for teachers. Specifically, teacherswere taught to scaffold and support
attention toward classroom goals, implement a school-home
report card, and promote the generalization of skills across classes.
Discrepancies between parent- and teacher-ratings of inattention
may be also due to the difference in their involvement in the interventions.
A noteworthy bias in clinical trials of behavioral interventions is
that the outcome measures are not blinded for participants (children
and parents) because participants know that they are receiving treatment.
In OST, parents are active participants of the treatment and therefore
they may be biased to report larger improvement on parent-rated
questionnaires (although, as noted above, some OST interventions also
included teacher components). This limitation can be addressed in future
studies by including additional outcomes such as naturalistic observation
of organizational skills and behaviors at school and at home,
report cards, or academic tests. Three trials in this meta-analysis
(Langberg et al., 2012; Sibley et al., 2013; Sibley et al., 2016) have
used objective outcomes such as planner use and materials organization
checklists rated by research assistants. Of note, while observational
measures may reduce the rater bias, they are harder to standardize
and costlier to administer, which limits application of these measures
in clinical trials.
Furthermore, parent and teacher ratings are only weakly correlated
for inattention symptoms and moderately correlated for hyperactivity/
impulsivity symptoms (Narad et al., 2015). Parents report greater severity
of ADHD symptoms than teachers, but the magnitude of parentteacher
agreement does not vary across development (Narad et al.,
2015). Due to the difference in parent- and teacher-ratings of ADHD,
we opted to analyze the parent and teacher ratings separately in this
study.
Another consideration for the assessment of treatment outcomes in
OST studies is that four of nine DSM inattention symptoms (i.e., often
fails to finish schoolwork and chores, often has difficulty organizing
tasks and activities, often loses things, and often forgetful in daily
activities) describe organizational behavior, which make it difficult to
separate change in ADHD symptoms of inattention from change in
organizational skills ratings. This overlap in symptoms of inattention
and organizational skill can also contribute to high levels of correlations
between parent and teacher ratings of these constructs. For example,
the correlations of the total scale COSS score with the inattention subscale
of the Conners were 0.66 and 0.67 for the parent and teacher ratings,
respectively (Abikoff & Gallagher, 2009). To examine if and how
the OST affects symptoms of inattention, future OST studies can report
change in the four inattention symptoms that overlap with organizational
behavior separately from change in the five inattention symptoms
without such overlap.
We found that OST had a modest ES in the area of academic performance.
Although relatively small in magnitude, this ES was statistically
significant. Given the correlation between academic performance and
organizational abilities (Langberg et al., 2011a; Langberg et al., 2011b),
this finding suggests that organizational skills interventions can benefit
academic performance for children with ADHD. The small ES are difficult
to detect in clinical trials of behavioral interventions evenwith samples
that would be considered large in our field (e.g., N200 subjects).
Nevertheless, even a small improvement in academic performance can
be clinicallymeaningful and practically important for children in general
and for children with ADHD in particular. By aggregating results
across studies, meta-analyses can estimate the magnitude of ES and
confirm statistical significance of smaller effects of active treatments
versus control conditions, thus informing the field of evidence-based
practice in ways that are complementary to the information provided
by the randomized controlled trials.
4.1. Differences across studies
There were also differences among treatments that might have
contributed to heterogeneity of ES. Although a small number of individual
studies precluded analyses of treatment characteristic as possible
mediators of treatment response, several key treatment variables can
be considered in future studies. As noted above, OST aims to improve
organizational skills by defining criteria for organizing time and
materials and setting realistic goals for staying organized. However,
the OST approaches varied across the studies in terms of the specific
treatment contents, the number and duration of sessions, the session
format (individual vs. group), and the extent of teacher and parent
involvements.
Parents played an important role in most of the trials as they were
asked to reinforce the targeted organizational skills at home. For the
twelve studies included in the meta-analysis, a parent component was
incorporated in different forms including: the parents joining their
child at the end of the session to review behavioral techniques such as
prompting, praise, and rewards (Abikoff et al., 2013; Langberg et al.,
2012); individualized parent sessions that addressed parenting skills
(Evans et al., 2014; Pfiffner et al., 2014); elements of family therapy
(Power et al., 2012); group parent sessions to discuss common behavioral
strategies such as positive reinforcement of expected behavior
(Langberg et al., 2008b; Sibley et al., 2013); and family sessions with
the parent and child together that address problem solving (Evans et
al., 2011; Evans et al., 2014; Sibley et al., 2013). Clearly, parent components
are important in any child-focused behavioral interventions but
the exact nature of the amount and content of parenting component
in OST cannot be easily identified from the existing studies. Similarly,
teacher involvement and school components varied across studies and
ranged from one consultation with the teacher, to facilitating the parent-
teacher meeting, to direct training of the teacher in prompting
and rewarding children's organizational skills during classes. It is likely
that teacher components are easier to incorporate in OST treatments
conducted in schools and parent components are more feasible when OST is conducted in
outpatient mental health settings. Table 1 shows
that six studies were conducted in school, five in clinics and three in
both clinics and schools.
Another notable difference among the studies was the inclusion
criteria.Only two studies required organizational deficits as an inclusion
criterion (Abikoff et al., 2013; Power et al., 2012). Most studies used academic
impairment for inclusion (Evans et al., 2011; Evans et al., 2014;
Evans et al., 2016; Sibley et al., 2016) and some also used social impairments
(Evans et al., 2011; Evans et al., 2014; Evans et al., 2016). This is
an important consideration for OST as not all children with ADHD
have organizational deficits. Including children without organizational
skills deficits into the studies might cause a ceiling effect where no further
improvementmight be possible. Future studies of OSTmay consider
a priori inclusion criteria for organizational skills to assure that the
intervention iswell-matched to the treatment needs of the study participants.
Age and developmental characteristic of OST participants is another
important consideration for future research as the number of
available studies precluded examination of participant characteristics
in this meta-analysis. Most OST studies included children 8 years of
age and older,with the exception of Mautone et al. (2012),who studied
children in kindergarten and first grade. Some OST interventions described
in the meta-analysis (Abikoff et al., 2013; Pfiffner et al., 2007;
Pfiffner et al., 2014; Power et al., 2012) were developed for elementary
school students, while others (Evans et al., 2014; Evans et al., 2016;
Langberg et al., 2012; Sibley et al., 2014b; Sibley et al., 2016)were developed
for middle school students. These treatments differ from each
other because they address organizational skills required for age appropriate
academic tasks.
Given that impairments in attention and organizational skill increase
in severity with age, due to the increase in academic demands and
decrease in adult supervision, clinical judgment suggests that earlier
interventions may promote the acquisition of skills and avert future
problems. However, behavioral interventions for young children with
ADHD are shown to reduce impulsivity and noncompliance (Barkley,
2013; M.T.A., 1999) and deficits in attention and organization may be
overlooked until older ages when compounded by functional impairments.
OST specifically addresses deficits in organization, timemanagement,
and planning and it is important to recognize that these skills are
related to age and cognitive development. Complex executive functions
such as planning continue to mature during adolescence and young
adulthood, and many of the organizational skills taught to adolescents
are developmentally inappropriate for younger children.When children
enter middle school the demands on organizational skills change dramatically
leading some children to exhibit greater difficulties in middle
school. Our review suggests that OST has been best testedmostly in 8 to
12 year-old children. Future studies might consider downward extensions
of OST approaches for younger children that might include ageappropriate
organizational skills that are required not only for school,
but also for play activities and daily living tasks.
Neurocognitive deficits in sustained attention,working memory, response
inhibition, and planning seen in children with ADHD (Nigg,
2005; Willcutt et al., 2005) are connected to organizational skills, and
future studies might test the effects of OST on neurocognitive profiles
as well as the utility of neurocognitive testing in predicting response
to OST. Also, children with ADHD show delays and heterogeneity in
brain maturation trajectories (Gopin & Healey, 2011; Shaw et al.,
2007). This calls for testing neural mechanisms and biomarkers of OST
toward a long-term goal of matching patients to treatments that can
engage neural mechanisms of attention and executive functions to
improve organizational skills.
Two studies of OST that met our a priori search criteria did not provide
data for calculating ES in the domains of interest in this meta-analysis.
Molina et al. (2008) reported the percentages of grades as a
measure of academic performance, a metric that could not be converted
to ES value. The study also reported the total BASC externalizing scale
that combines ratings of inattention with ratings of aggression,
hyperactivity, and conduct problems. We also could not include the
Pfiffner et al. (2007) study because the outcomes were averaged across
teacher and parent ratings and we were interested in evaluating ES
separately for the parent and teacher ratings. These two studies can be
included in future meta-analyses that may focus on other outcome
domains.
Regarding durability of OST treatment effects, five studies provided
follow-up data in a range from one to six months after treatment.
Three studies reported significant differences in parent-reported
inattention at 6 month follow-up in almost moderate (g = 0.48)
(Evans et al., 2016), moderate g=0.72 (Pfiffner et al., 2014) and large
ES g = 0.80 (Sibley et al., 2016). Parent-rated organizational skills
were significant (g = 0.56) at the 6-months follow-up in one study
(Pfiffner et al., 2014). Teacher-rated organizational skills were significant
(g = 0.32) at the 3-month follow-up in another study (Power
et al., 2012). These results suggest that OST treatment gains are robust,
at least in the short term.
4.2. Limitations
Our review also revealed several limitations of the randomized
controlled studies of OST, most notably lack of consistency in outcome
assessments and failure to use independent or “blinded” outcome
assessment. This underscores a need for outcomemeasures with strong
psychometric properties and clinical utility in the area of testing organizational
skills interventions. One point to convey is that all outcome
measures (i.e., parent ratings, teacher ratings, observations, or grades)
are likely to have their unique strength and limitations. We note that
parent and teacher ratings in clinical trials of behavioral interventions
are not blinded and can be affected by expectation biases, but this
does not mean that they are “inferior” to observational measures.
Counting items in the backpack or number of turned-in homework assignments
may be more objective (because the count can be done by
a researcher unaware of treatment assignments). However, behavioral
counts are subject to measurement error due to situational variability
and are hard to standardize across children. Also, in contrast to some
fields of behavioral research that have widely accepted “gold-standard”
measures (such as the Yale-Brown Obsessive Compulsive Scale in studies
of behavioral therapy for OCD), the field of OST research is relatively
new and it appears that no single measure has emerged as a possible
gold standard. We suggest that future studies of OST clearly designate
one primary outcome and use a comprehensive set of secondary and
exploratory outcome measures.
The majority of studies of OST used the waitlist or treatment-asusual
(TAU) as control conditions, but the description of allowed concomitant
treatments was inconsistent across publications. The terms
“waitlist-control”, “treatment-as-usual”, and “community control” are
sometimes used interchangeably in studies of psychosocial interventions
and imply that subjects are allowed to continue mental health
and/or educational services that they have been receiving prior to enrolment
in the study. For example, clinical trials may include subjects on
concomitantmedication but require thatmedication is stable for a period
of time. An important distinction iswhether or not subjects are asked
not to initiate newtreatments for the duration of the active phase of the
study. Two studies in our dataset (Evans et al., 2011; Evans et al., 2014)
provided subjectswith a list of resources in community and encouraged
themto pursue care. Other studies did not encourage or discourage subjects
in control conditions from pursuing additional services while in
the active phase of the study. Some control subjects initiated newtreatments,
which might have led to greater symptom reduction compared
to control conditions that required no initiation of new treatments.
The results of this meta-analysis might have also been affected by
several limitations of the body of available randomized trials, including
uneven sample sizes (sometimes treatment was twice as large as control
conditions), small number of studies to meta-analyze, high risk of
bias due to lack of blinding of outcome assessors, and few trials conducting intent to treat
analysis. The small sample of studies in this
meta-analysis did not allowus to explore possible causes of heterogeneity,
which for our primary outcomes was quite large. Also, the small
number of studies precluded our ability to examine the presence of publication
bias in all but one meta-analysis (parent-rated attention),
which revealed the possibility of publication bias using the Trim and
Fill method. A limitation of our meta-analysis is that we did not include
a meta-analysis of observational measures of organizational skills, as
only three studies provided data for each of the measures,which is considered
a small number to combine statistically.We alsowere unable to
synthesize data on parent-rated academic performance because this
outcome was not included in any study. Another limitation is that we
used HPC-I as a proxy measure for inattention, despite its specific
focus on behaviors related to homework. However, HPC-I shows a correlation
of 0.67 with BASC-PRS Inattention subscale (Power et al.,
2006), which is why it was used as a measure of attention. Among the
DSM symptoms of Inattention, four are indicative of organizational
problems. The attention improvements reported by parents and
teachers in the studies might be influenced by improvement of these
four organizational-related symptoms. However, it was not possible to
dismantle the improvement in specific attention symptoms as none of
the studies reported items beyond subscale scores. The same is true
for the specific effect on the organizational skill measures, where total
scoreswere preferably reported. The goal ofmeta-analysis is to estimate
treatment effects aggregated across studies and domains of
measurement, and this approach does not disentangle the complexity
of psychological functions such as organizational skills. Organizational
functioning involves a complex set of skills and as more clinical trials
of OST become available, future meta-analyses may investigate relative
effects on specific organizational skills, for example, as reflected by the
COSS subscales of organized actions, task planning, and materials management.
It is unknown whether the improvements in organizational
functioning were uniform across the specific subskills in these studies.
Future studies should examine the effects of OST on the specific items
of attention and organizational skills measures.
5. Conclusion
OST leads tomoderate improvements in organizational skills of children
with ADHD as rated by teachers and large improvements as rated
by parents. Modest improvements were also observed on the ratings of
symptoms of inattention and academic performance. The review notes
methodological limitations of studies to date including relatively small
samples of the majority of studies, heterogeneity of the outcome measures,
and lack of attentional control comparison conditions. The clinical
implication of this meta-analysis is that OST is a helpful treatment for
organizational skills deficits in ADHD and it could be considered as
part of comprehensive treatment approaches for the core ADHD symptoms
as well as associated functional impairments.
Roles of each author
Aida Bikic and Denis Sukhodolsky designed the study and wrote
the protocol. Aida Bikic and Spencer McCauley conducted literature
searches, provided summaries of previous research studies, and extracted
data. Brian Reichow conducted the statistical analysis and wrote the
results section.Aida Bikic,Denis Sukhodolsky, Brian Reichow, and Karim
Ibrahim wrote the manuscript and all authors read and approved the
final manuscript.
Conflict of interest
The authors have no conflict of interest.
Funding
Dr. Aida Bikic received support from Region of Southern Denmark
Psychiatry Research, TrygFonden (J.nr. 7-12-1137), The Region of
Southern Denmark's PhD pool (2011), University of Southern
Denmark (12/2/2011) and Lundbeck Foundation (j.nr. R169-2014-38).
Acknowledgments
We thank the founding organizations.We thankMs. Shivani Kaushal
for her help with the literature search in preparation of revisions of this
paper and for proofreading the final manuscript.
Appendix A. Supplementary data
Supplementary data to this article can be found online at http://dx.
doi.org/10.1016/j.cpr.2016.12.004.
References
Abikoff, H., & Gallagher, R. (2009). The children's organizational skills scales: Technical
manual.
North Tonawanda, NY: Multi-Health Systems.
Abikoff, H., Gallagher, R.,Wells, K. C., Murray, D. W., Huang, L., Lu, F., & Petkova, E. (2013).
Remediating organizational functioning in childrenwith ADHD: Immediate and longterm
effects from a randomized controlled trial. Journal of Consulting and Clinical
Psychology, 81(1), 113–128. http://dx.doi.org/10.1037/a0029648.
Abikoff, H., Nissley-Tsiopinis, J., Gallagher, R., Zambenedetti, M., Seyffert, M., Boorady, R., &
McCarthy, J. (2009). Effects of MPH-OROS on the organizational, time management,
and planning behaviors of children with ADHD. Journal of the American Academy of
Child and Adolescent Psychiatry, 48(2), 166–175. http://dx.doi.org/10.1097/CHI.
0b013e3181930626.
American Psychiatric Association, APA (2013). Diagnostic and statistical manual of mental
disorders (5th ed.) (Washington DC).
Anesko, K. M., Schoiock, G., Ramirez, R., & Levine, F. M. (1987). The homework problem
checklist: Assessing children's homework problems. Behavioral Assessment, 9,
1979–1985.
Barkley