Chemistry Education

Research and Practice

View Article Online
PAPER View Journal | View Issue

Guiding teaching with assessments: high school

chemistry teachers use of data-driven inquiry
Cite this: Chem. Educ. Res. Pract.,
2015, 16, 93
Jordan Harshman and Ellen Yezierski*

Data-driven inquiry (DDI) is the process by which teachers design and implement classroom assessments and
use student results to inform/adjust teaching. Although much has been written about DDI, few details exist
Published on 30 October 2014. Downloaded on 25/09/2016 11:51:40.

Received 8th September 2014,
Accepted 21st October 2014
DOI: 10.1039/c4rp00188e
www.rsc.org/cerp
about how teachers conduct this process on a day-to-day basis in any specific subject area, let alone
chemistry. Nineteen high school chemistry teachers participated in semi-structured interviews regarding how
they used assessments to inform teaching. Results are reported for each step in the DDI process. Goals
there was a lack in specificity of learning and instructional goals and goals stated were not conducive to
informing instruction. Evidence at least once, all teachers determined student learning based solely on
scores and/or subscores, suggesting an over-reliance on these measures. Conclusions many teachers
claimed that students either did or did not understand a topic, but the teachers did not adequately describe
what this means. Actions very few teachers listed a specific action as a result of analyzing student data,
while the majority gave multiple broad pedagogical strategies to address content deficiencies. The results
of this study show the limitations of teachers DDI practices and inform how professional development on
DDI should have a finer grain size and be based in disciplinary content.

Introduction
High school chemistry teachers have daily access to assessment
data from their students. Whether from homework assignments,
quizzes, informal questions, classroom discussions, or lab activities,
teachers have virtually endless ways of eliciting student feedback
and using the information gained in the process to evaluate and
adjust their instruction if necessary. Throughout the process of
informing their teaching via assessment results, teachers make
multiple inquiries about teaching and learning, using data to drive
any decisions made, which is why we refer to it as data-driven
inquiry (DDI). This process goes by many other names in the
literature (Calfee and Masuda, 1997; Brunner et al., 2005;
Datnow et al., 2007; Halverson et al., 2007; Gallagher et al.,
2008; Hamilton et al., 2009), but all describe a very similar
framework (Harshman and Yezierski, in press).
DDI is carried out daily in the teaching of chemistry and
teachers are expected to use student data as a basis for improving
the eectiveness of their practice (Means et al., 2011). All decisions
that teachers make about what they will do dierently or similarly
next year, next topic, next class, and even next minute is
in some way dictated by information that teachers consider
(Calfee and Masuda, 1997; Suskie, 2004; Hamilton et al., 2009).
Department of Chemistry and Biochemistry, Miami University, Hughes Hall,
501 East High Street, Oxford, OH 45056, USA. E-mail: yeziere@maimioh.edu
Electronic supplementary information (ESI) available. See DOI: 10.1039/
c4rp00188e
Even so, previous research has failed to characterize this process at
a level that is sucient to inform advances teachers can make in
their own DDI processes (Harshman and Yezierski, in press).
Herein lies the purpose of our study; to thoroughly investigate
how teachers use (or do not use) their classroom data to inform
their teaching. Addressing this problem will allow teachers to tailor
their instruction to meet the needs of their specific classroom
environments with their specific content areas to promote lasting
learning (Suskie, 2009). By enhancing assessment analysis and
interpretation skills, teachers are better able to accommodate their
individual students (Means et al., 2011). Further, the National
Academy of Science reported in 2001 that classroom assessments
are not being used to their fullest potential (National Research
Council, 2001). Eight years later, the Institute of Education
Sciences remarked that despite a keen interest in DDI, questions
about how educators should use data to make instructional
decisions remain mostly unanswered (Hamilton et al., 2009)
and three extensive studies from the Department of Education
(Gallagher et al., 2008; Means et al., 2010, 2011) suggest a similar
need for investigation. Characterizing DDI practices in chemistry
classrooms is a necessary first step in determining what reforms in
assessment and teacher learning might be necessary to optimize
such DDI practices which have been deemed critical to eective
teaching and student learning.
For any one unit, topic, or concept, teachers provide instruction
to students and then assess students via the variety of assessment
types (e.g. informal, diagnostic, formative, summative, etc.). We
focus here on formative assessments, the definition of which

This journal is The Royal Society of Chemistry 2015 Chem. Educ. Res. Pract., 2015, 16, 93--103 | 93

Paper
revolves more around what one does with the results as opposed
to what it physically is (Bennett, 2011; Wiliam, 2014). When we
use the word assessment, we are referring to anything that
elicits student responses regardless of whether it is an actual
object or not (informal questioning, for example). Throughout
this manuscript, we will detail the process of DDI into four
distinct steps as described by general education research, provide
specific science education research of DDI, describe our methods,
and present results from the study according to each of the
four DDI steps. The research presented here was born from
an extensive literature review we conducted (Harshman and
Yezierski, in press), reviewing over 80 sources on the topic of
using assessment results to inform instructional practice. In a
general sense, DDI is simple to explain because it closely
mimics the familiar scientific process of inquiry. Specifically
Published on 30 October 2014. Downloaded on 25/09/2016 11:51:40.
applied to high school chemistry education, we describe DDI in
terms of four steps: determining goals, collecting evidence,
making conclusions, and taking actions.
Goals are what teachers are hoping to find out via the results
of their assessments. Goals can range from strictly content
goals (e.g. Students will be able to explain why electrons transfer
in a single displacement reaction.) to more abstract ones (e.g.
I want my students to better appreciate the energy loss asso-
ciated with internal combustion engines.). Similarly, teachers
can set the goal to evaluate the eectiveness of an instructional
strategy (e.g. Does my implementation of the smores lab
promote understanding of mole-to-mole ratios?). Although this
particular aspect of aligning assessments with goals is important
to the entire process of assessment in chemistry, it is largely
outside the scope of this study but investigated using the same
interviews presented here (Sandlin et al., in preparation).
Once the goal(s) for an assessment is (are) set, teachers design
an assessment that will elicit the data needed to evaluate the
goals. After implementation and collection of this assessment,
teachers seek out the relevant information that will inform their
goal(s), referred to as evidence. Like goals, sources of evidence
range widely from simple, objective scores and sub-scores (e.g.
90% students responded correctly) to complex aective char-
acteristics such as the disposition of specific students on the
day of assessment (e.g. this student didnt really try). Much
like in research fields (Linn et al., 1991; American Educational
Research Association, 1999; Schwartz and Barbera, 2014), the
validity and reliability of sources of evidence used in educa-
tional environments should be considered (Popham, 2002;
Datnow et al., 2007; McMillan, 2011; Witte, 2012). Often in
conjunction with choosing valid and reliable evidence, teachers
make conclusions relevant to the goals of the assessment.
Conclusions are any declarative statements that a teacher
makes about teaching and learning, the assuredness of which
is dependent upon what can be supported by the evidence
examined. Of note, it is mostly unavoidable that personal and
professional experience play a role in making conclusions
about teaching and learning, which is not always a bad thing
(Suskie, 2004). Once data are analyzed and interpreted, teachers
address issues, if any arise, through proposed actions. From the
teachers perspective, all actions are instructional actions.
94 | Chem. Educ. Res. Pract., 2015, 16, 93--103
View Article Online
Chemistry Education Research and Practice
Finally, that action (or inaction) can also be assessed with the
same or a dierent assessment, making the DDI process cyclical.
The impetus for the investigation lies in two crucial ideas.
Firstly, teachers will not adopt new pedagogical ideas if the task of
translating those ideas into practice is the primary responsibility
of the teachers (Black and Wiliam, 1998). Secondly, previous
formative assessment research has been largely generalized, and
the specifics of a given content area have not been adequately
addressed when considering design and analysis of formative
assessments (Coey et al., 2011). Both ideas suggest the necessity
to study formative assessment practices while the content,
chemistry in this case, is fully integrated into the investigation.
As an example of this integration of chemistry, instead of just
asking why a teacher designed an assessment as they did, this
study probed chemistry-specific considerations, such as why the
teacher chose a certain mole-to-mole ratio, why s/he included
unit conversions or not, and what the rationale was for providing
additional information such as molar masses and chemical
equations. Thus, the findings presented here were generated
because the study was designed around high school chemistry
teachers course-specific assessments and the central tenet of
data analysis was chemistry content learning.
Background
In our literature review, we found that research on DDI is
not specific enough to guide day-to-day practice and that a
vast majority of the research does not specify a discipline or
education level (Harshman and Yezierski, in press). Although
they do not directly call out a DDI process, there are a handful
of projects that have investigated analogous data-analytic
frameworks or various aspects of DDI in chemistry/science.
Focusing on the practices of teachers informal, formative
assessments, Ruiz-Primo and Furtak (2007) reported the
number of complete and incomplete ESRU cycles (Elicits a
question, Student responds, Recognize the response, and Use
the information) as well as individual components of each step
in the process (e.g. teacher revises students words was one
specific thing the researchers coded for). In this study, three
middle school science teachers assessment practices were
described according to steps of the ESRU framework. While
this method provides valuable information of how these teachers
implement ESRU in their classrooms, focusing only on theorized
steps of ESRU neglects specific aspects not predicted by the
ESRU model.
More specific to the secondary science setting, Tomanek et al.
(2008) investigated several factors that explained the reasoning
used to select tasks for students in science courses. This directly
relates to the goal step of DDI as it dictates the alignment
between the goals and the task. The major finding of this study
was a detailed framework that delineates how teachers (experi-
enced and preservice) choose a specific assessment task. While
these two studies from Ruiz-Primo and Furtak (2007) and
Tomanek et al. (2008) add valuable understanding to teachers
assessment process, much more is left to be investigated.
This journal is The Royal Society of Chemistry 2015

Chemistry Education Research and Practice
Research questions
Although assessment also serves the very important purposes of
evaluating and providing feedback to students, the focus of our
research is how teachers use evaluation of students to inform
their teaching. The research questions are aimed at closing a gap
in the knowledge base about specific ways in which chemistry
teachers inform their teaching with assessment results:
1. How do high school chemistry teachers guide their
teaching via the design, implementation, and analysis of class-
room assessments?
a. What goals do teachers hope to achieve throughout the
assessment process in general and in one specific assessment?
b. What sources of evidence do teachers consider when
determining eect of their teaching and/or evaluation of learning?
Published on 30 October 2014. Downloaded on 25/09/2016 11:51:40.
c. What types of conclusions are made about students and
as teachers themselves as a result of analyzing student assess-
ment data?
d. What kind of actions do teachers take to address the
results of assessments?
Methods
Sample
A total of 19 (12 female, 7 male) high school chemistry teachers
from 10 dierent states participated in an interview conducted
either in-person or via Skype. The sampling method used
is dicult to label definitively, but had aspects of criterion
and random purposeful sampling techniques (Miles and
Huberman, 1994). The list of teachers invited to participate in
interviews was compiled by selecting states from the low
(approx. 140), middle (approx. 150), and high (approx. 160)
performance on the science portion of the National Assessment
of Educational Progress standardized tests. Using a random
number generator, 3 states from each category were chosen,
followed by 10 high schools in each state (found by randomizing
a list of each states counties available online and then searching
for schools within those counties), and finally 1 high school
chemistry teacher from each school was selected using school
web directories. In addition to the 90 teachers chosen this way,
more were added by listing acquaintances of the authors of this
paper for a total of 126 teachers invited (15% response rate).
Science Teacher Ecacy Beliefs Instrument (STEBI, Bleicher,
2004) distribution scores and years of teaching experience are
available in Appendix A (ESI). We purposefully did not collect
any information regarding previous professional development
programs regarding chemistry-specific DDI for two reasons.
First, we assumed that few professional development programs
(and no chemistry-specific ones) directly address DDI, and it is
even less likely that our sampled teachers had gained skills
or knowledge from such a program (and our results support
this). Secondly, we took great strides to discourage teachers
from responding based on what they thought the interviewer
wanted to hear rather than what they actually do in their
classes, which could be prompted by even a rudimentary
description of DDI.
This journal is The Royal Society of Chemistry 2015
View Article Online
Paper
Interviews
After informed consent (research was fully compliant with IRB) was
collected, each of the 19 teachers participated in a semi-structured,
three-phase interview that lasted between 3090 minutes. Next,
the teachers provided a homework assignment, quiz, laboratory
experiment, or in-class activity (collectively referred to as an
assessment) that had already been administered, collected, and
evaluated by the teacher prior to the interview. This assessment
was sent a day before the interview so that the interviewer could
customize questions for specific items on the assessment
provided. In Phase 1 of the interviews, teachers were asked
about the definitions and purposes of assessment in general
and in their own classrooms. For each assessment as a whole
(Phase 2) and two to four individual items on the assessment
(Phase 3), teachers were asked (1) what the goal(s) of the
assessment/item was (were), (2) what conclusion(s) could be
made about those goals, (3) how the teacher determined these
conclusions, (4) what the teacher would do in light of the
conclusions, (5) how the material was originally taught, and
(6) any questions specific to the content area assessed by the
item or assessment. For the complete interview protocol with
an example assessment, see the Appendix B (ESI).
Coding
All transcripts were transcribed verbatim and all coding was
carried out in NVivo 9.2 software (QSR International, 2011).
Analysis of data went through multiple iterations within each of
five methodological steps: (1) data were open coded (as described
by Creswell, 2007) into 13 major categories. (2) Horizonalization
to develop clusters of meaning (Moustakas, 1994), which were
transformed to specific codes. (3) Data were coded according to
these codes. (4) Individual codes and associated references were
organized and coded into the four main aspects of DDI frame-
work (closed coding). (5) The final organization of the codes was
created by aligning the clusters of meaning hierarchically under the
four main steps of DDI. See Appendix C (ESI) for an illustration
of this entire process.
Reliability
Two separate stages of inter-rater reliability were conducted
to determine the reliability of the coding performed. In both
stages, Krippendors alpha (Krippendor, 2004a) statistic was
used. In opposition to percent agreement, Krippendors alpha
corrects for chance agreement among raters. Although kappa
statistics also correct for chance agreement (Cohen, 1960;
Fleiss, 1971), Krippendors alpha calculates chance agreement
via the proportion of individual codes applied globally versus
proportions of agreements and disagreements between raters.
This attenuates the dierences that individual raters bring to
coding and emphasizes the reliable use of codes given any rater
(Krippendor, 2004b). Also, kappa statistics have been shown
to suer from a base rate problem (Ubersax, 1987), which is to say
that the interpretation of kappa may depend on the proportions
of agreement/disagreement. In the first stage, approximately
one third of two dierent interviews were coded for the four
Chem. Educ. Res. Pract., 2015, 16, 93--103 | 95

Paper
main steps of DDI (goals, evidence, conclusions, actions) by
three chemistry education researchers plus the first author of
this paper (a = 0.889). In the second stage, one full transcript was
coded by the same four researchers in accordance with the full
list of codes (81 total, a = 0.787). In addition, three researchers
applied these 81 codes to three other full transcripts (one for
each of the three, the first author coded all three transcripts);
the three pairwise alpha values were 0.848, 0.874, and 0.729.
Cutos for Krippendors alpha have been documented between
0.6670.800 (Krippendorff, 2004b). These results indicate that
the codes were reliably applied throughout the data. None-
theless, the 81 individual codes were revised as needed from
this process and the entire data set was recoded according to
these revisions.
Published on 30 October 2014. Downloaded on 25/09/2016 11:51:40.
Results and discussion
Results will be presented in terms of the DDI steps, since this
framework informed the interview protocol and analysis.
Within each DDI step, major themes and representative quotations
are given. To more eciently present the results in the narrative,
the number of teachers (out of 19) that were coded for a particular
Fig. 1 Coding organization chart.
96 | Chem. Educ. Res. Pract., 2015, 16, 93--103
View Article Online
Chemistry Education Research and Practice
idea will be shown in boldface followed by the total number of
times that idea was coded across all interview in parentheses
(NVivo terms sources and references, respectively). Fig. 1
shows a graphical representation of the ideas coded along with
the frequencies that they manifested in the data. Appendix D
(ESI) contains a list of all codes along with their descriptions.
Goals
Given the importance of delineating a goal for each item on any
assessment, it is surprising that many teachers did not clearly
articulate goals when asked (see Fig. 1). Opposed to stating
objectives those in teacher education might be accustomed to
seeing (i.e. students will be able to. . .), many teachers listed
the required knowledge, equations, and concepts to solve the item
or use colloquial terms like just see if they get it to describe their
goals for an assessment or assessment item. Anecdotally, it
seemed as if the teachers were unaccustomed to describing goals
for specific assignments despite the heavy emphasis on lesson
planning in high schools. Just because teachers did not delineate
their goals clearly does not mean that they do not have well-
defined goals for their assessments. However, it does indicate
that they may not be consciously thinking about the specific
This journal is The Royal Society of Chemistry 2015

Chemistry Education Research and Practice
Fig. 2 Example of one of Mandisas assessment item with goals.
reason for assessing students on a regular basis. Fig. 2 shows
an example problem provided by Mandisa:
Interviewer: So what were the goals for this specific item?
Mandisa: Number one, they had to remember that delta H was
the sum of the products minus the sum of the reactants and they
had to multiply those delta H values by the coecients, they forget
Published on 30 October 2014. Downloaded on 25/09/2016 11:51:40.
that. . . Then they had to understand how to take that data, and
plug it into the Gibbs free energy. . . they dont look at units. That
delta S was in joules, not kilojoules. . .
The goals given by the teacher represent the steps that
students must complete to arrive at the correct solution. The
analysis of this goal, therefore, will only ever tell if students are
or are not following the correct steps to achieve the right
solution, a conclusion that does not readily inform instruction
or the degree students have learned chemistry concepts.
In Phase 1 of the interviews (which does not allude to any
specific assessment), 11 (13) teachers expressed at least one goal
that was directly tied to their instruction. Examples of this were
teachers that wished to use results of assessments to know if
what Im doing is working (Adie), make the teaching better describe their content-oriented goals in a manner that neither
(Chris), or explain this better the next time (Mark). However,
in Phases 2 and 3 of the interview that probed the goals of
specific assessments, only one explicitly stated that she designed
and used her specific assessment to inform her instruction:
Alyssa: But almost every day I give, its kind of a video quiz, an
assessment of what they saw last night. . . Sometimes that informs
me as a teacher, did they pick up what I wanted them to get out of
that video. If not then its my job to fill the gaps. . . am I doing what
I need to do to support them?
None of the teachers stated instructional improvement as a
goal for an individual assessment item, but just because this
goal was not explicitly stated does not mean the results did not
inform instruction. However, other evidence presented suggests
that these teachers spent little time thinking about their instruc-
tion in light of assessment results.
While no specific instructional goals were reported by the
teachers, two types of broad instructional goals can be seen in
Fig. 1. First, 9 (14) teachers expressed their interest in using
assessments to determine how to make the teaching better
(Chris) or see what areas I need to improve (Mandisa). Second,
Mark and 9 (12) others discussed using their assessments to
determine whether or not to continue on with the curriculum
versus going back and reteaching in some way. Although these
goals have a growth mindset, they are very limited as they do
not inform a specific conclusion or action because of their lack in
specificity. Being able to conclude one or two specific aspects of
teaching lead to extremely vague suggested actions (detailed in
This journal is The Royal Society of Chemistry 2015
View Article Online
Paper
further sections). For example, when Adie wanted to use assess-
ments to know if what [shes] doing is working, there are many
inquiries that naturally follow: How did you teach it originally?
How have students responded to that teaching style previously?
How does what is measured (student understanding) translate to
what you wish to conclude (eectiveness of teaching)? How
do you measure student understanding of the chemistry pheno-
menon? Yet, these questions were not discussed by Adie or
any of the other teachers that had similar instructional goals.
In the data, setting general instructional goals lead to broad
and oftentimes not very helpful actions.
Not surprisingly, every teacher at some point defined goals
for their specific assessment and assessment items pertaining
to their students understanding of certain chemistry and math
concepts. This is consistent with a DDI model because, in order
to make decisions about instruction, evaluation of student-level
aspects must be made. However, just because a goal is about
chemistry content does not mean that a meaningful interpreta-
tion can be gained from results. Surprisingly, 2 (2) teachers
leads to a clearly assessable outcome nor yields valuable
information, exemplified by Chris and Jess:
Chris: So the goal is basically to expose them to every type of
chemical reaction that the CollegeBoard could possibly ask them
and to try to give them a format exactly as theyre going to see it.
Jess: Um, and so that was kind of the intent. So to give them
practice with a variety of the problems and then in kind of a
structured way.
Exposing formats and giving are verbs associated with
teachers actions that require no student feedback whatsoever.
These teachers may not literally mean that their only goal is to
expose content or give practice. However, their elaboration
provides evidence that they are teacher-centric goals because
they do not discuss anything related to students learning when
making conclusions. Because such goals do not depend on student
performance, the results cannot directly inform teaching and
learning, thus bringing the process of DDI to a halt.
Finally, 10 (18) teachers goals for items and assessments revolve
around the idea of understanding content. Unfortunately, the
word understand appears to have many dierent meanings to
many dierent teachers and will be discussed below.
Evidence
Chronologically, after teachers design and implement an assess-
ment, evidence must be considered in order to make conclu-
sions about teaching and learning. To make these conclusions,
teachers used both evidence that can be measured with a high
Chem. Educ. Res. Pract., 2015, 16, 93--103 | 97

Paper
degree of certainty (calculated scores, attendance records, etc.)
and evidence that is measured with a lesser degree of certainty
(eort, confidence, paying attention, etc.). We refer to these as
suppositional (lesser certainty) and non-suppositional (greater
certainty) evidences. The primary source of non-suppositional
evidence was the use of a percent correct on an item or whole
assessment. In this logic, teachers concluded that because a
high percentage of the class got an item correct, they must
understand the content. The problem with this logic is that it
heavily relies on the assumptions that (1) the assessment is
designed to appropriately elicit chemistry understanding; and
(2) the responses from students represent those students
mental models. Neither researchers nor teachers can guarantee
that responses to assessment items absolutely reflect students
mental models (Pelligrino, 2012). According to Fig. 1, this over-
Published on 30 October 2014. Downloaded on 25/09/2016 11:51:40.
simplified view of assessment occurred in 16 (40) of the
teachers the three teachers that did not exhibit this code
either never collected their assessment or took completion
points and did not ever calculate percent correct. Bart provides
an example of this code:
Interviewer: . . .do the assessment results seem to support or
refute this goal, that youve met this goal, or your students have?
Bart: It seems to support it.
Interviewer: And then why would you believe that?
Bart: Because the vast majority of the students got most of that
problem right if not completely right.
Beyond simple descriptive statistics of scores on an assess-
ment, the only non-suppositional measures of evidence used
by teachers were the current students performance on future
(13 (20)) and past (5 (6)) assessments with similar content
and performance of previous classes on the same or similar
assessments (9 (12)). This means that teachers will use past
assessments or look ahead to assessments that are not yet given
to students:
Matthew: . . . are the kids really paying attention or do they
not know, and I typically find that out on the final assessment
for that unit.
Nichole: Our final in this unit, one of my classes the average
was 89% and the other was 80%, and last year it was down in the
lower 70s. And Ill be honest with you, these kids struggle more
than the kids from last year. Just, theyre not as bright, and theyre
not as motivated. . .
In the two examples, both teachers use performance on a
final exam as a source of evidence to make their conclusions,
but Nichole also compares current students performance to
previous years performance (a second source of evidence).
Teachers also used the same sources of evidence in very
dierent ways. For instance, consider how Natalie used the
previous performance of her students compared to Britt:
Natalie: [The students] definitely have met [my goals]. I did
have a wide range from As to Fs, but knowing the students
background, of course the students that made the As are the ones
I anticipate getting extremely high scores on the national exams,
those that made Fs are thinking they are getting out of the class by
transfer anyways so they have not been trying, so I did not expect
my two Fs to do well. . .
98 | Chem. Educ. Res. Pract., 2015, 16, 93--103
View Article Online
Chemistry Education Research and Practice
Interviewer: . . .was [conclusion that students could not con-
struct particulate drawings from scratch] based on what you saw in
this assessment?
Britt: No, just previous knowledge of because Ive done this, Ive
used these particle drawings before so my expectation is ya know I,
from previous assessments, I know that if I were, would have
thrown something more complicated on there, I ask them to um,
draw something that we havent seen yet, they might not be able to
do that. . . I think its uh, ya know theyve seen this before, but
theyve never had to draw them themselves. . .
Natalie makes the assumption that if a student performs well
in the past they should perform well in the future and vice versa
for those that did not perform well. While arguments can be made
for the validity of this claim, its use as evidence in data analysis is
built on faulty logic, as a wide variety of factors that aect student
performance on her assessment were not taken into account by
Natalie. Alternatively, Britt not only realizes her assessment (which
requires students to recognize features of particulate drawings but
does not require them to draw them) cannot answer the question
posed by the interviewer, but she also relies on the performance of
previous classes and states that students lack of familiarity with
this type of problem may mask conclusions she can make about
her students understanding of particulate images.
Alternatively, suppositional sources of evidence were used
abundantly throughout all portions of the interviews. Compar-
isons using non-suppositional evidence generally led to more
thoughtful conclusions in the sample of teachers whereas the
use of more suppositional measures lead to greater uncertainty
in conclusions. Of these suppositional sources, the most pre-
valent were observations from classroom discussions (13 (22)),
teacher experience or instinct (9 (14)), and the level to which
students were paying attention during classes (6 (8)). Teachers
also used evidence that has both suppositional and non-
suppositional characteristics. An example of these sources
of evidence were the previous math and chemistry courses of
students and what that means (7 (8)) and the motivation of
students (3 (4)). In this category of codes, special attention is
given to teachers who made conclusions about students based
on how much practice the teacher perceived the student
completed. This idea can be seen in Fig. 1 as a goal (purpose
of assessment or item was to give students practice, 4 (8)),
source of evidence (conclusions made were based on amount of
practice, 10 (16)), conclusion (students did not get enough
practice, 9 (17)), and an action (give students more practice
opportunities, 12 (18)). Many times, it seemed as though the
teachers were entirely focused on getting practice with solving a
particular type of problem instead of anything to do with
understanding the chemistry concepts:
Alyssa: Yeah, I think that for the most part, um, I think that
students were able to do exactly what I had asked them to do
because they had that practice in advance.
Britt: Um, but Im happy with where were at so far because
theyre going to be getting more practice on it.
Bart: . . .they usually from enough practice can tell me what the
limiting reagent is, they can dierentiate from the limiting and
actual and find the percent yield. . .
This journal is The Royal Society of Chemistry 2015

Chemistry Education Research and Practice
The use of suppositional evidence in making instructional
decisions is as unavoidable as it is encouraged (Hamilton et al.,
2009). However, when these teachers used more suppositional
sources of evidence, it was often in isolation of as opposed to in
tandem with non-suppositional evidence. Making decisions in
this way can be detrimental to the process of DDI.
Conclusions
By far, codes that represent conclusions that teachers made about
students were the most abundant throughout the interviews.
Looking at the blue regions in Fig. 1, it is clear that student-
centered conclusions (any conclusions about students or learning)
grossly outnumber teacher-centered conclusions (any conclusions
about teachers or teaching). This suggests that no matter how
much the teachers set out to use the results of their assessments
Published on 30 October 2014. Downloaded on 25/09/2016 11:51:40.
for instructional modification, few actually concluded anything
about their instruction, and instead made many declarative
statements about their students or their students learning. Not
all questions can be appropriately used to inform instruction, but
the high number of conclusions that the teachers made about
students that were unrelated to their understanding of chemistry
content was of interest. These conclusions included making
aective judgments and determining student characteristics:
Michael: Yeah, because as I was saying these are capable
students and highly motivated, theyre very math oriented. . . these
are students who have a very high math capability and so once they
understand the methods here, they can just go run by themselves
after that.
Britt: . . .ya know it really wasnt anything new as far as the
actual content or discussion but that they actually went and took
some measurements, just engaged them more so that they, um, they
understood it better.
The two examples from Britt and Michael above could just as
easily be displayed in the evidence section of this paper, as the
teachers made conclusions to be used as evidence of under-
standing and learning. Michael informs his instructional decision
(deciding that he will let them run by themselves) by stating dierent models of bonding. Lastly, Amy both provides scarce
that his students are capable, motivated, and math-oriented.
Similar to previous discussion, these may not be the best sources
of evidence for which to base what a teacher should do next, as it
could lead to false-positive (teacher believes instruction promoted
learning when it did not) or false-negative (teacher believes
instruction did not promote learning when it did) results. Beyond
representing evidence, these and other quotes like it reveal what
teachers were aiming to conclude about with their assessments.
Teachers that commented on aspects about students that were
not tied to the chemistry content were comparable in abundance
to aspects that were tied to chemistry content. This finding is very
interesting because it demonstrates the value of evidence that can
only be obtained by looking outside of the responses to the
assessment and outside of considerations from the chemistry
content. It was dicult to know if the teachers were determining
these characteristics based on the results of the assessment or
simply based on their experiences as teachers. Regardless of
which, these conclusions regarding students came up during
interview prompts that specifically asked them to consider what
This journal is The Royal Society of Chemistry 2015
View Article Online
Paper
information could be gleaned from the results, indicating that ideas
not directly from assessment results were a large part of the
teachers data analysis process.
Conclusions tied to the chemistry content are integral to the
DDI process, as they inform the specific content area weak-
nesses of students, which in turn helps identify the pedagogical
strategy used so that it also may be evaluated. As alluded to
previously, a majority of the teachers mentioned conclusions
revolving around the idea of student understanding (10 (18)). As
an example, Mark used the root understand and its associated
tenses 37 times in a 64 minute interview (not counting colloquial
phrases such as get it and on the same page). Below is a
variety of dierent ways in which teachers concluded that their
students understood something:
Adie: . . . if they can represent the situation in more than one
way, it shows me that they actually understand whats going on. . .
they could participate in discussion, which tells me that at least
they understood enough to discuss it.
Laura: . . .I dont think theyre really understanding the dier-
ence between that that does not represent necessarily two mole-
cules of hydrogen and two molecules of oxygen and then making
the mole thing. . .
Nichole: . . .we just were not getting any clear understanding, what
was the dierence? What were the dierences and what were the
similarities of the four models [of bonding], we werent getting those.
Amy: . . .they did pretty successfully, they understood the question,
they understood the concept, they understood the math, and like I said
thats what its all about.
Here, a range of meanings of understand can be observed. To
Adie, understanding means that students can represent phenomena
in dierent ways with an emphasis on being able to talk about it,
although she does not specify how the students talk about it,
only that they do talk about it. Laura diers from this by equating
understanding with demonstrating rote knowledge (of atomic
versus molecular hydrogen and oxygen). Nicole likens under-
standing to being able to identify dierences and similarities in
details on what it means to understand something, but only
that they do or do not understand it. The ambiguity of under-
standing something is just as detrimental to conclusions as it is
to goals: without the specification dictated by detailed models of
chemistry phenomena, the teachers gathered very little useful
information from assessments.
Actions
In contrast to conclusions, actions that teachers would take in
light of assessment results were scarce. This is most likely a
result of not having specific instructional goals and conclu-
sions, because any prescribed action needs to address the
conclusions made from assessment data. Similar to conclu-
sions, teachers proposed actions were often vague and marked
with indecision:
Adie: Well, if they cant do those things then I need to go back
and present the material again or do it in a dierent way.
Laura: So obviously they didnt get the concept, so I need to
reteach it somehow, so if it doesnt work by just lecturing and
Chem. Educ. Res. Pract., 2015, 16, 93--103 | 99

Paper
putting an example on the board and doing a couple of them with
it, Ill have to come up with a manipulative something. . .
Adie and Laura both desire to change their instruction, but do
not detail what changes are going to be made or how the content
deficiencies will be addressed. These were both coded as
ambiguous actions (13 (36)) because it is unclear as to what
the teachers will actually do. In addition, 10 (14) teachers
responded with multiple options for instructional adjustments
as a sort of laundry list of suggestions, but never committed to
one even after follow-up questions trying to pinpoint which
action(s) the teacher deemed necessary. While coming up with
multiple options for how to adjust instruction is a good thing,
teachers must eventually decide on one or more actions, or else
no changes will be made. Because of the limited actions brought
up, the data corpus holds limited findings on the decision-
Published on 30 October 2014. Downloaded on 25/09/2016 11:51:40.
making processes of teachers course of action.
Another noteworthy action exhibited by the teachers was the
notion of reteaching a concept. While many more teachers
used the words reteach or recover, only 7 (10) gave evidence number (approximately half, although this was never specifically
to suggest they would physically do or say the same exact things
to address a content deficiency. Alternatively, many teachers
implied that they would actually teach with a dierent pedagogy
even though they used the word reteach and were thus coded
as ambiguous as described above. As an example of the former
group of teachers, Michael stated that he originally taught
stoichiometry to his students using an analogy of turkey sand-
wiches and bicycles. When the assessment yielded less-than-
expected results from his students, Michaels response indicates
that he will do exactly as he did previously:
Interviewer: . . .what exactly are you going to do or what have
you done to address the mole to mole ratio?
Michael: Well I go back to bicycles and turkey sandwiches. How
many seats do you need for each bike, so thats a one to one ratio.
How many wheels do you need for each bike, two, thats a two to
one ratio, so now lets come back to the chemical substances and
its the same methodology.
Our hypothesis to explain this finding is that these teachers
adhere to the belief that students need to hear things multiple
times before being able to understand the content and/or meet
the teachers goal(s). We argue for this belief because all but one
of the teachers who said they would reteach exactly as originally
taught also had heavy emphasis on practice inside and outside
of class, indicating the value of repetition in student work.
Exemplary DDI
Throughout this project, it was never our intention to criticize
high school teachers or to actively search for things they were
doing wrong. So far, we have purposefully shown places
where teachers DDI practices limit their ability to inform their
instruction so that we can promote future research to focus on
interventions in these areas. Despite the previous results, we
are happy to report just a sample of the promising aspects seen
throughout the data. For example, with a heavy emphasis on
mathematical assessments and items, 9 (17) teachers noted
that just because students were able to complete algorithmic
100 | Chem. Educ. Res. Pract., 2015, 16, 93--103
View Article Online
Chemistry Education Research and Practice
problems did not mean that they understood the concepts
behind them:
Laura: I consider chemistry and application so I feel like, hey,
doing math problems is applying it but again, everybody can plug
and chug if they know how to do it, but do they understand why
theyre doing it, thats, Ive been asking a lot of why questions
lately, which I think is helping.
Quotes like the one above from Laura indicate at least
partial alignment between goals and items on assessments
because the teachers demonstrate that they interpret results
knowing that correct mathematical answers do not necessarily
indicate (mis)understanding of chemistry concepts.
Additionally, 5 (7) teachers specifically sought feedback from
their students by use of metacognitive questions beyond what
are you struggling with? When teachers invest in collecting
more information from their students, they are gathering more
data than just the results of the assessment and are better able to
inform appropriate instructional modifications. Lastly, a fair
coded) of the sample of teachers mentioned writing notes for the
next time they would implement an assessment or activity. This
common practice aligns well with DDI as the teacher uses data to
drive what changes (if any) should be made to instruction.
Conclusions
In our response to specific research questions 1a1d, major
themes were identified in each subheading above. The high
school chemistry teachers in this study set goals that were not
very conducive to instructional improvements largely due to their
ambiguous nature, as was determined by analyzing the chemistry
content the teachers stated as goals. In evaluating these goals,
teachers often used suppositional or non-suppositional evidence,
but not both in conjunction with the one another, thus missing
opportunities to cross-reference sources of evidence upon
which to draw conclusions. As a result, many of the conclusions
about student learning were based on potentially faulty evidence,
compromising the integrity of the DDI process. In addition,
conclusions regarding the teachers themselves as effective educa-
tors were not reported in the same quantity as the goals that were
made for instructional improvement. This led to an even lesser
number of tangible, specific actions that teachers said they would
take to improve their instructional effectiveness.
Our study also shows several overall themes associated with
how teachers enact the DDI process (response to broad research
question 1). First, in every step teachers demonstrated a lack of
elaboration when responding to interview prompts. Teachers
may be oversimplifying the processes of learning and measuring
learning, which are both extremely complex, by reducing complex
chemical phenomenon to algorithms for which understanding
can be determined by simple correct/incorrect dichotomies.
Secondly, in several of the specific assessment items that
teachers provided for the interviews, the DDI process was
stunted by having dichotomous goals (e.g. see if the students
got it or not) or by teachers dismissal of assessment results
This journal is The Royal Society of Chemistry 2015

Chemistry Education Research and Practice
because of aective aspects. Lastly, based on the results dis-
cussed, the sample of high school chemistry teachers in this
study demonstrated limited DDI practice. We label it limited
because the teachers implementation of DDI constrains what
information teachers could draw from their assessment results
and what pedagogical strategies could be employed to support
eective pedagogy and improve upon weaker pedagogy.
Limitations
To exhibit transparency in our work, we have provided a
discussion of some limitations with our study. First, we asked
several questions in a format that implies a yes/no answer so
that teachers would be prompted to commit to explicitly stated
conclusions as opposed to supplying responses that make it
Published on 30 October 2014. Downloaded on 25/09/2016 11:51:40.
unclear as to what the teacher actually thinks. This encouraged
shorter responses to our prompts and as a result, could have
increased the prevalence of codes that targeted dichotomous
responses. For this reason, the interviewer always asked
unscripted questions to elicit more detail and codes were given
more stringent descriptions so that what was coded was a result
of what the teachers expressed as opposed to a product of the
prompt format. Another limitation was the variety of teacher-
provided assessment content. To rule out dierences and find
similarities in DDI, we originally limited the content to a few
topics, but ended up with nine chemistry topics throughout the
interviews due to participant availability. While this provides
breadth, we recognize that the DDI process may look dierent
depending on the topic being assessed because pedagogical
strategies and types of assessment may also change depending
on these topics. This limitation has theoretical backings that
content knowledge interacts with pedagogical knowledge to
engender pedagogical content knowledge (Shulman, 1987). In
response to this issue, we sought information specific to the
topic and later generalized the findings into themes that cut
across multiple topics such as conceptual vs. mathematical
understanding. Lastly, the data used in this project was self-
reported data from the teachers that participated and findings
are not based on observational evidence. We do not see this as
much as a limitation, but more as our focus on what teachers
were thinking during assessment interpretation an aim that
observational data would not be able to validly measure.
Implications
This study generated many implications for chemistry teachers,
professional development providers, and chemistry education
researchers. The implications for teachers and professional
development providers are organized by the steps of DDI,
followed by the implications for researchers.
Goals
When teachers set goals for assessments, caution should be used
when trying to assess students understanding of a certain topic.
As was seen in our study, many vague goals of understanding did
This journal is The Royal Society of Chemistry 2015
View Article Online
Paper
not lead to fruitful conclusions regarding the quality of learning or
instruction. A great way for teachers to do this is to take extra time
to ask what does it mean to understand this topic and what
evidence would show this? and then ensure that assessment
items can actually elicit such evidence. High quality assessments
available to teachers such as Page Keelys assessment probe books
from NSTA Press (Keely and Harrington, 2010), concept inventories
(a sample of inventories in redox, Brandriet and Bretz, 2014;
bonding, Luxford and Bretz, 2014; general chemistry, Mulford
and Robinson, 2002), and the American Association for the
Advancement of Sciences Assessment web site (http://assess
ment.aaas.org/pages/home) can alleviate the burden on teachers
to generate items so that teachers can focus more on aligning
assessments with specific learning goals. Secondly, determining
how eective a lesson was in encouraging learning should be a
goal of specific assessments and items, not just a general concern
as many teachers thought of it as. Teachers that actively consider
how their teaching has impacted learning, which is at least to a
degree displayed in student responses, will have a heightened
ability to reflect on their practice (for reflection theory, Schon,
1983; Cowen, 1998; for example implementation in science educa-
tion, Danielowich, 2007). We wish to encourage teachers to make
specific conclusions tied to their instruction; without this reflec-
tion/consideration, very few changes to teaching will be occur, as
was the case in our sample.
Evidence
The most pressing issue observed at least once from every single
teacher (except three that did not collect their students assessments)
is the unquestioned assumption that student performance on
assessments unequivocally correlates to understanding of the con-
tent. At a minimum, the use of scores and/or sub-scores on
assessments needs to be considered in light of the design of the
assessment and should be considered alongside other sources of
evidence. For example, a teacher should consider more than a class
average score to determine student understanding and gather
evidence outside of scores, such as similar assessments given
previously and/or students questions during work time and discus-
sions. Alternatively, teachers overly favored suppositional evidence.
While professional experience certainly has a place within DDI
(Suskie, 2004), it should be noted that judgments of the aective
characteristics of students does not come without bias, and caution
is warranted when making such judgments. We suggest that
teachers have at least 23 sources of evidence when making an
affective judgment. For example, considering in conjunction a large
absence/tardy record, multiple conversations revealing apathy, and
unsuccessful attempts to get a certain group of students to partici-
pate strongly suggests a lack of motivation where only one of these
sources of evidence would weakly suggest motivational issues. In
many cases, use of only one source of evidence (student perfor-
mance data) will not enable the teacher to determine if one goal is
fully met. The obvious solution to this is to collect more evidence,
but this suggestion wasnt mentioned frequently by the teachers.
Scaffolding problems is a way to access more information while
providing feedback to students throughout the learning process (see
Chapter 5: scaffolding learning in chemistry, in Taber, 2002).
Chem. Educ. Res. Pract., 2015, 16, 93--103 | 101

Paper
Conclusions
As was observed in the data, without a great deal of conclusions
about teaching, there are few teaching actions that can be
specified. Therefore, we reiterate the importance that teachers
consider the impact that pedagogical strategies have on learning
as measured by assessments. Also, whether students results are
good or bad, teachers should attempt to specify the why the alignment between stated goals and assessment items,
behind what was observed. Asking clarifying questions such as
what was it specifically about this lesson that worked so well in
helping them learn? or I think that showing a guided example
of the whole stoichiometry process followed by an explanation of
each step did not work as well as first explaining each step and
then showing the whole process. In these examples, ideas
directly tied to teaching are being assessed by the assessment
as opposed to solely assessing student learning. Also, when
Published on 30 October 2014. Downloaded on 25/09/2016 11:51:40.
student learning is being assessed, we suggest (as we did in
the goals section) to avoid making conclusions such as the
students understand this content. Teachers should clarify both
what chemistry the students specifically understand and what
it means to understand that specific chemistry content.
Actions
Consistent with our theme of calling for detail, we find it
important that chemistry teachers choose a specific action for
a specific purpose. If a teacher suspects that one type of activity
did not promote learning, it is better to uncover (or at least
hypothesize) a reason why it did not than to try something
dierent just because it is dierent. Targeting a reason for
less-than-expected performance specifically informs a new
pedagogical strategy needed that could address the reason that
the strategy was less eective.
Research
In addition to the above implications for teachers, this study
also has implications for researchers studying high school
chemistry teachers and instruction. Because the teachers from
our sample spoke ambiguously about their assessment pro-
cesses, other studies investigating DDI or general assessment
characteristics may find a similar lack of specificity. As we have
reported, this does not mean that teachers neglect to think
about their assessment practices, but a carefully constructed
data solicitation method is required to prompt teachers to
elaborate on their assessment practices. We found that the
discussion of assessment practices in situ with their chemistry-
specific curriculum, assessments, and instruction worked well
and would serve future inquiries. This outcome emphasizes the
necessity of studying assessment in a disciplinary context.
Future work
While this study investigates the DDI process of 19 high school
chemistry teachers, it does not necessarily translate to the
majority of high school chemistry teachers. Future studies
should focus on characterizing which of these traits exist in a
representative sample of high school chemistry teachers to see
102 | Chem. Educ. Res. Pract., 2015, 16, 93--103
View Article Online
Chemistry Education Research and Practice
how widespread these themes are. After characterization of a
large sample, professional development and pre-service educa-
tional programs should be developed to address limitations in
the process found in larger scale studies. Also, our qualitative
study focused on a bigger picture of the entire DDI process
with details at each step. Additionally, an investigation into
teachers process while developing assessments, and compar-
ing assessment beliefs to assessment practices would be very
valuable for both researchers and practitioners.
Acknowledgements
We would like to thank the teachers that participated in this
study as well as the Yezierski and Bretz Research Groups at
Miami University. Specifically, we thank Justin Carmel, Sara
Nielsen, and Sarah Erhart for their assistance in qualitative
coding for this project.
References
American Educational Research Association, American Psycho-
logical Association, National Council on Measurement in
Education and Joint Committee on Standards for Educa-
tional and Psychological Testing (U.S.), (1999), Standards
for educational and psychological testing, Washington, DC:
American Educational Research Association.
Bennett R. E., (2011), Formative assessment: a critical review,
Assess. Educ.: Princ., Pol., Pract., 18(1), 525.
Black P. and Wiliam D., (1998), Inside the black box: Raising
standards through classroom assessment, Phi Delta Kappan,
80(2), 139144, 146148.
Bleicher R., (2004), Revisiting the STEBI-B: Measuring Self-
Ecacy in Preservice Elementary Teachers, Sch. Sci. Math.,
104(8), 383391.
Brandriet A. R. and Bretz S. L., (2014), The development of the
Redox Concept Inventory as a measure of students symbolic
and particulate redox understandings and confidence,
J. Chem. Educ., 91, 11321144.
Brunner C., Fasca C., Heinze J., Honey M., Light D., Mandinach
E., et al., (2005), Linking data and learning: The Grow
Network study, Journal of Education for Students Placed at
Risk, 10(3), 241267.
Calfee R. C. and Masuda W. V., (1997), Classroom assessment
as inquiry, in Phye G. D. (ed.), Handbook of classroom
assessment. Learning, adjustment, and achievement, San Diego:
Academic Press.
Coey J. E., Hammer D., Levin D. and Grant T., (2011), The
missing disciplinary substance of formative assessment,
J. Res. Sci. Teach., 48(10), 11091136.
Cohen J., (1960), A coecient of agreement for nominal scales,
Educ. Psychol. Meas., 20, 3746.
Cowen J., (1998), On Becoming and Innovative University Teacher,
Buckingham: Open University Press.
This journal is The Royal Society of Chemistry 2015

Chemistry Education Research and Practice
Creswell J. W., (2007), Qualitative inquiry and research design:
Choosing among five traditions, 2nd edn, Thousand Oaks,
CA: Sage.
Danielowich R., (2007), Negotiating the conflicts: Reexamining
the structure and function of reflection in science teacher
learning, Sci. Educ., 91(4), 629663.
Datnow A., Park V. and Wohlstetter P., (2007), Achieving with
data: How high-performing school systems use data to improve
instruction for elementary students, Los Angeles, CA: University
of Southern California, Center on Educational Governance.
Fleiss J. L., (1971), Measuring nominal scale agreement among
many raters, Psychol. Bull., 76(5), 378352.
Gallagher L., Means B. and Padilla C., (2008), Teachers use
of student data systems to improve instruction, 2005 to 2007,
U.S. Department of Education, Oce of Planning, Evalua-
Published on 30 October 2014. Downloaded on 25/09/2016 11:51:40.
tion and Policy Development, Policy and Program Studies
Service.
Halverson R., Prichett R. B. and Watson J. G., (2007), Formative
feedback systems and the new instructional leadership, Madison,
WI: University of Wisconsin.
Hamilton L., Halverson R., Jackson S., Mandinach E., Supovitz
J. and Wayman J., (2009), Using student achievement data to
support instructional decision making (NCEE 2009-4067),
Washington, DC: National Center for Education Evaluation
and Regional Assistance, Institute of Education Sciences,
U.S. Department of Education.
Harshman J. and Yezierski E., (in press), Assessment data-
driven inquiry: A review of how to use assessment results
to inform chemistry teaching, Sci. Educ., Summer, 2015.
Keely P. and Harrington R., (2010), Uncovering Student Ideas in
Physical Science, Washington, DC: National Science Teachers
Association Press, vol. 1.
Krippendor K., (2004a), Content Analysis: An Introduction to its
Methodology, 2nd edn, Thousand Oaks: Sage.
Krippendor K., (2004b), Reliability in content analysis:
some common misconceptions and recommendations,
Hum. Commun. Res., 30(3), 411433.
Linn R. L., Baker E. L. and Dunbar S. B., (1991), Complex,
performance-based assessment: Expectations and validation
criteria, Educ. Res., 20(8), 1521.
Luxford C. J. and Bretz S. L., (2014), Development of
the Bonding Representations Concept Inventory to Identify
Student Misconceptions about Covalent and Ionic Bonding
Representations, J. Chem. Educ., 91, 312320.
McMillan J. H., (2011), Classroom assessment: principles and
practice for eective standards-based instruction, 5 edn,
Pearson.
Means B., Chen E., DeBarger A. and Padilla C., (2011), Teachers
ability to use data to inform instruction: challenges and supports,
Oce of Planning, Evaluation and Policy Development, U.S.
Department of Education.
Means B., Padilla C. and Gallagher L., (2010), Use of education
data at the local level: From accountability to instructional
improvement, Oce of Planning, Evaluation and Policy
Development, U.S. Department of Education.
This journal is The Royal Society of Chemistry 2015
View Article Online
Paper
Miles M. B. and Huberman A. M., (1994), Qualitative Data
Analysis: An Expanded Sourcebook, Thousand Oaks, CA: Sage.
Moustakas C., (1994), Phenomenological Research Methods,
Thousand Oaks, CA: Sage.
Mulford D. and Robinson W., (2002), An inventory for alternate
conceptions among first-semester general chemistry students,
J. Chem. Educ., 79, 739744.
National Research Council, (2001), Knowing what students know:
The science and design of educational assessment. Committee
on the Foundations of Assessment, in Pelligrino J., Chudowsky
N. and Glaser R. (ed.), Board on Testing and Assessment, Center for
Education. Division of Behavioral and Social Sciences and Educa-
tion, Washington, DC: National Academy Press.
Pelligrino J., (2012), Assessment of science learning: living in
interesting times, J. Res. Sci. Teach., 49(6), 831841.
Popham W. J., (2002), Classroom Assessment: What Teachers
Need to Know, 3 edn, Allyn and Bacon.
QSR International (2011), NVivo qualitative data analysis soft-
ware. Version 9.2.
Ruiz-Primo M. A. and Furtak E. M., (2007), Exploring teachers
informal formative assessment practices and students
understanding in the context of scientific inquiry, J. Res.
Sci. Teach., 44(1), 5784.
Sandlin B., Harshman J. and Yezierksi E., (2014), Formative
assessment in high school chemistry teaching: investigating
the alignment of teachers goals with their items, J. Chem.
Educ. Res., in preparation.
Schon D. A., (1983), The Reflective Practitioner: How Professionals
Think in Action, USA: Basic Books.
Schwartz P. and Barbera J., (2014), Evaluating the Content and
Response Process Validity of Data from the Chemistry Con-
cepts Inventory, J. Chem. Educ., 91(5), 630640.
Shulman L. S., (1987), Knowledge and teaching: Foundations of
the new reform, Harvard Educ. Rev., 57(1), 122.
Suskie L., (2004), What is assessment? Why assess? In Assessing
student learning: A common sense guide, San Francisco:
Jossey-Bass Anker Series, pp. 317.
Suskie L., (2009), Using assessment results to inform teaching
practice and promote lasting learning, in Joughin G. (ed.),
Assessment, Learning, and Judgement in Higher Education,
Springer Science.
Taber K., (2002), Chemical Misconceptions Prevention, Diag-
noses, and Cure Volume I: Theoretical Background, Piccadilly,
London: Royal Society of Chemistry, pp. 6784.
Tomanek D., Talanquer V. and Novodvorsky I., (2008), What do
science teachers consider when selecting formative assess-
ment tasks?, J. Res. Sci. Teach., 45(10), 11131130.
Ubersax J. S., (1987), Diversity of decision-making models and
the measurement of interrater agreement, Psychol. Bull.,
101(1), 140146.
Wiliam D., (2014), Formative assessment and contingency in
the regulation of learning processes, paper presented at
Annual Meeting of American Educational Research Associa-
tion, Philadelphia, PA.
Witte R. H., (2012), Classroom assessment for teachers, McGraw-Hill.
Chem. Educ. Res. Pract., 2015, 16, 93--103 | 103