Running head: CRITIQUE OF AN ASSESSMENT

!
!
!
!
!
!
!
!
!
!
Critique of an Assessment
Julie K. Marsh
College of William and Mary

CRITIQUE OF AN ASSESSMENT

Critique of an Assessment
It is important for instructional leaders to have a full understanding of curriculum,
instruction, and assessment as well as how they relate to one another. Curriculum is a set of
intended learning outcomes (Grant, 2015). Intended learning outcomes (ILOs) are statements of
what students should know, be able to do, or value after instruction is given (Grant, 2015).
Effective assessment aligns with clear ILOs (Waugh & Gronlund, 2011, p. 21). Creating
effective and reliable assessments begins with understanding ILOs (Gareis & Grant, 2015, p. 52).
The following paper critiques a teacher-made assessment in Grade 7 Life Science. The
assessment itself is described as well as the creator of the assessment, the ILOs and their
alignment to curriculum standards are analyzed, a reflection on the assessments validity and
reliability is provided, as well as a discussion on the effects of instructional leadership when
looking at this assessment.
The Assessment
The creator of the assessment is a Life Science and Physical Science teacher in a middle
school in Richmond, Virginia. In the interest of protecting her identity, I shall refer to her as
Linda. Linda teaches both seventh and eighth grade students, and she has been teaching for nine
years. She has always loved exploring science, and she truly loves helping her students find a
love for science as well. Linda and I worked together at her middle school many years ago, so
when I needed a quality assessment for this particular class assignment, I reached out to her.
Linda was eager to help, and she provided me with the assessment critiqued below. Linda gave
me permission to use the assessment in any way I needed.
The assessment is a seventh grade Life Science test intended to be administered as a
checkpoint to make sure students are on-track for both the midterm and the Standards of

CRITIQUE OF AN ASSESSMENT

Learning (SOL) exam at the end of the year. In terms of consequential validity, both the
intended and unintended consequences are appropriate for this assessment (Waugh & Gronlund,
2013). The assessment helped Linda see where students needed more instruction and possibly
different types of instruction in order to help them prepare for the midterm and the SOL.
The assessment is a 33 question multiple choice test covering 22 ILOs. It is organized in
number order and includes some options for matching as well as visual examples. The ILOs
include a focus on having the student investigate and understand the basic physical and chemical
processes of photosynthesis and its importance to plant and animal life, understanding all living
things are composed of cells, and knowing all living things show patterns of cellular organization
(Virginia Department of Education, 2010).
Cogent Set of ILOs and Alignment to Curriculum Standards
The more specific ILOs are listed below in terms of performance:

Describe the process of photosynthesis of raw materials and products generated

Identify and describe cellular organelles involved in the process of photosynthesis
Explain how organisms utilize energy stored from the products of photosynthesis
Compare and contrast processes of photosynthesis and cellular respiration
Relate the importance of photosynthesis to the role of producers as the foundation of food
webs
Design an investigation from a testable question related to photosynthesis. The investigation
may be a complete experimental design or may focus on systematic observation, description,
measurement, and/or data collection and analysis
Distinguish the cell membrane, cytoplasm, nucleus, cell wall, vacuole, mitochrondrion,
endoplasmic reticulum, and chloroplast
Correlate the structures of cell organelles with their functions
Compare and contrast examples of plant and animal cells, using the light microscope and
images obtained from other microscopes
Describe and sequence major points in the development of the cell theory
Identify three components of the cell theory
Sequence the steps in the cell cycle, including the phases of mitosis
Differentiate the purpose of mitosis and meiosis
Design an investigation from a testable question related to animal and plant cells. The
investigation may be a complete experimental design or may focus on systematic
observation, description, measurement, and/or data collection and analysis. An example of

CRITIQUE OF AN ASSESSMENT

such a question is: Do onion cells vary in shape or structure depending on where they are
found in the plant?
Explain the relationship among cells, tissue, organs, and organ systems
Differentiate unicellular organisms and multicellular organisms and name common
examples of each
Compare and contrast unicellular and multicellular organisms perform various life
functions. This includes the application of knowledge about systems in organisms
Explain the role that each life function serves for an organism: ingestion, digestion and
removal of waste, stimulus response, growth and repair, gas exchange, and reproduction
Explain there is a specific range or continuum of conditions that will meet the needs of
organisms
Model how materials move into and out of cells in the processes of osmosis, diffusion, and
selective permeability. This includes creating and interpreting three-dimensional models
and/or illustrations demonstrating the processes involved. Students should be able to analyze
the components of these models and diagrams and communicate their observations and
conclusions
Create plausible hypotheses about the effects that changes in available materials might have
on particular life processes in plants and in animals
Conduct basic investigations related to understanding cellular organization, with emphasis
on observations of cells and tissue. This investigation should focus on the skills developed in
LS.1
(Virginia Department of Education, 2010)

These ILOs are directly from the Life Science Curriculum Framework from the Virginia
Department of Education (2010), specifically the Essential Knowledge, Skills, and Processes
sections for standards LS.2, LS.3, and LS.5. These ILOs are a cogent set of learning objectives
that directly tie to the SOLs taken at the end of the academic year. The middle school where
Linda works has a designed curriculum directly stemming from the SOLs and Curriculum
Framework, so the schools mission aligns as well. The school also focuses on 21st century
learning skills in its Life Science classrooms as well as college and career readiness. The school
is located in a county with a 1:1 laptop initiative, meaning every student and teacher has access
to a laptop 24/7. Linda makes a point to implement technology in her students science
exploration on a daily basis. Her students learn skills that not only align with the designed
curriculum for both the school and the states SOLs, but also emphasizes communication,
collaboration, and creativity, skills her students will need in their future lives.

CRITIQUE OF AN ASSESSMENT

Construct and Content (Sampling) Validity and Repeated Trials Reliability

Using the ILOs from the Essential Knowledge, Skills, and Processes under standards
LS.2, LS.3, and LS.5, I created a Table of Specifications (see Table 1) in order to map where the
content aligned with each cognitive level.
Table 1
Table of Specifications for Cell Processes Test
Cognitive Level
Content

Process of
photosynthesis of raw
materials and
products generated

Remember Understand

Importance of
photosynthesis to the
role of producers as
the foundation of food
webs
Investigation from a
testable question
related to
photosynthesis. The
investigation may be a
complete

Analyze

Evaluate

Create

describe

identify;
describe
1

Cellular organelles
involved in the
process of
photosynthesis
How organisms
utilize energy stored
from the products of
photosynthesis
Processes of
photosynthesis and
cellular respiration

Apply

explain
2, 7, 26

9, 10

compare;
contrast
5
23, 25

relate

design

CRITIQUE OF AN ASSESSMENT

experimental design
or may focus on
systematic
observation,
description,
measurement, and/or
data collection and
analysis
Cell membrane,
cytoplasm, nucleus,
cell wall, vacuole,
mitochrondrion,
endoplasmic
reticulum, and
chloroplast
Structures of cell
organelles with their
functions
Examples of plant and
animal cells, using the
light microscope and
images obtained from
other microscopes
Major points in the
development of the
cell theory
Three components of
the cell theory
Steps in the cell cycle,
including the phases
of mitosis
Purpose of mitosis
and meiosis
Investigation from a
testable question
related to animal and
plant cells. The
investigation may be a
complete
experimental design
or may focus on
systematic
observation,

6, 21

distinguish
29

22

correlate

compare;
contrast

describe

sequence

identify
11, 12, 19

27, 28, 31,

33

15, 20

sequence
30

differentiate

design

CRITIQUE OF AN ASSESSMENT
description,
measurement, and/or
data collection and
analysis. An example
of such a question is:
Do onion cells vary
in shape or structure
depending on where
they are found in the
plant?
Relationship among
cells, tissue, organs,
and organ systems
Unicellular organisms
and multicellular
organisms and name
common examples of
each
Unicellular and
multicellular
organisms perform
various life functions.
This includes the
application of
knowledge about
systems in organisms

14

explain
32

differentiate

13, 18

16, 17

compare;
contrast
24

Role that each life

function serves for an
organism: ingestion,
digestion and removal
of waste, stimulus
response, growth and
repair, gas exchange,
and reproduction

explain

There is a specific
range or continuum of
conditions that will
meet the needs of
organisms

explain

Materials move into

and out of cells in the

model

CRITIQUE OF AN ASSESSMENT

processes of osmosis,
diffusion, and
selective
permeability. This
includes creating and
interpreting threedimensional models
and/or illustrations
demonstrating the
processes involved.
Students should be
able to analyze the
components of these
models and diagrams
and communicate
their observations and
conclusions
Plausible hypotheses
about the effects that
changes in available
materials might have
on particular life
processes in plants
and in animals

create

Basic investigations
related to
understanding cellular
organization, with
emphasis on
observations of cells
and tissue. This
investigation should
focus on the skills
developed in LS.1

conduct

Note: Using a teachers list of ILOs and the test relating to the ILOs, a table of specifications was
made in order to show where the content aligned with the various cognitive levels.
Construct validity. In order for an assessment to have construct validity, the assessment
must align with the ILOs. In the case of the Life Science assessment, there is not much
alignment between the assessment and ILOs. In fact, the assessment only aligns with the ILOs

CRITIQUE OF AN ASSESSMENT

nine times in the 33 question assessment. So the other 2/3 of the time, the assessment is not in
alignment. The assessment may align with what was taught, but that also is a validity issue: if
the teacher is giving instruction that does not align with the curriculum, and the assessment does
not align with the ILOs, then the students are not learning what they should be learning to be
successful on the upcoming midterm or the SOL at the end of the year. This could also introduce
issues with consequential validity because the students may not be ready for the next level of
science at the end of the year. Additionally, there are issues with the assessment in light of the
cognitive levels. The ILOs are spread evenly across the six cognitive levels; however, the
assessment only tests students in the Remember and Understand levels. In fact, out of 33
questions, only one question is on the Apply level while the other 32 questions are split between
Remember and Understand.
Content (sampling) validity. The assessment is not a lengthy one, though it is
attempting to assess many ILOs. There are either too few questions or too many ILOs; either
way, there is clearly an issue with sampling validity in this assessment. It would be impossible
to assess all 22 ILOs in only 33 questions. While the ILOs that require investigations and
designs of projects (e.g., Design an investigation from a testable question related to animal and
plant cells) were never likely thought to be included on this exam, but rather in other class
projects to assess those skills and knowledge, there are still too many ILOs to be assessed.
Considering both construct and content validity. There are issues with both construct
and content validity in this assessment. The creator either asked too few questions or intended to
assess far too many ILOs. The creator either needs to expand the assessment or, better yet, have
shorter assessments attempting to test fewer ILOs at one time. The creator also needs to look at
the alignment between the ILOs and what the assessment is actually testing. I have great

CRITIQUE OF AN ASSESSMENT

10

concerns over the construct validity of this assessment and how far off alignment it truly is based
on the schools curriculum and the Curriculum Framework.
Repeated trials reliability. Students are provided many opportunities to show their
learning across nine of the ILOs, so roughly half of what the test should have assessed. For
instance, students were asked to explain how organisms utilize energy stored from the products
of photosynthesis, and they were given four different opportunities across Remember and
Understand to show their learning. Students were asked to compare and contrast processes of
photosynthesis and cellular respiration, and they were given three different opportunities across
Remember and Understand to show their learning. Finally, the best example was when students
were asked to sequence the steps in the cell cycle, including the phases of mitosis, and they were
given eight different opportunities across Remember and Understand to show their learning.
These are only three examples of nine in this particular assessment showing a high level of
repeated trials reliability.
Reliability: Sources of Measurement Error
There are multiple issues with measurement error for this particular assessment,
specifically visual issues in both text and pictures. The text is quite small on the printed page
and also uses serif fonts. Serif can be difficult to read for some students, especially if the text
size is small. So, for instance, question #1 uses small text and serif font, plus it also employs the
use of a bolded word. Due to the size and the font, it is difficult for the student to see one of the
words, specifically the word except, is different from the others. This can cause the student to
not realize the question is asking for her to find the one that does not belong. It also does not
help that the question is in a negative form, which can cause confusion for some students
(Waugh & Gronlund, 2013).

CRITIQUE OF AN ASSESSMENT

11

Another visual issue is the pictures used throughout the assessment. The assessment was
printed for students on paper in black and white, causing some of the pictures to be blurry or
grainy. For instance, question #17 uses a picture of a wilting plant; however, it is difficult to
even see the picture is of a wilting plant. The question asks, When this organelle is not full of
water, plants tend to wilt as seen in the picture below. Due to the wording of the question,
which does not use clear and simple language (Waugh & Gronlund, 2013), it is difficult to know
if the question is saying the picture is of the organelle or a wilting plant, which can cause
confusion for the student when answering the question. A final issue with the pictures used is
the placement above and below questions. Some images are used above the question while
others are used below; in fact, there are two questions that ask the question, provide the picture,
and then give the answer choices. It can be confusing to a student and cause issues in answering
the questions if she is unsure which picture matches which question and which answer choices.
Scoring System
The 33 question multiple choice assessment is scored on an evenly weighted scale,
meaning that each question is worth exactly the same: one point. There is no rubric for this
assessment, and I believe there should be. I think it would be good for the students to understand
the importance of each question and how the questions relate to the ILOs. I also believe each
question should have a different weight since clearly, according to the Table of Specifications
above, there were certain questions the teacher was weighing more over others.
The current scoring system for this assessment will not help the teacher see if there is
consistency across her students or whether or not she is asking enough questions in many
different ways to truly assess her students knowledge and learning. As discussed before, there
are either too few questions or too many ILOs being assessed, so the teacher needs to make a

CRITIQUE OF AN ASSESSMENT

12

decision as to whether she should expand the test or have different assessments for all of the
ILOs.
Instructional Leadership
If I were an administrator working with Linda, there would be a few areas I would like to
focus on as she grows as a teacher. I would first want to meet with Linda to go over the
assessment and compare it with the Table of Specifications. In fact, it might be helpful to ask
her to create her own Table of Specifications prior to our first meeting so we can both have an
understanding of where we are and where we need to go. Together we can then unpack the
standards and discuss goals for her students including daily goals, unit goals, and long-term
goals, ensuring all goals align with the department, school, and state curriculum based on the
Curriculum Framework.
Once we had a shared understanding of the curricular needs for her students, then I would
want to look specifically at the assessment and teacher-created Table of Specifications. Together
we can walk through and discuss what she observes based on the cognitive levels as well as ILOs
assessed. I would point out she has demonstrated strength in providing many opportunities for
her students to show their learning by asking different questions on the same ILO. I would then
point out the need to increase the cognitive level across all of her questions. Then I would
discuss the need to either expand the test to more than 33 questions or limit the amount of ILOs
assessed in one test. Once we explore both of those options together including the pros and cons
to both, we can create a game plan for changing this assessment as well as a plan for creating
future assessments. Finally, I would point out some of the physical features of the assessment
that could cause issues (e.g., blurry pictures and serif font) and see if she has ideas for making
those aspects better for her students for greater accessibility.

CRITIQUE OF AN ASSESSMENT

13

I would continue to follow up with Linda by conducting walk-through observations and

individual conferences to provide support and encouragement to her while also monitoring her
growth and successes.
Conclusion
Educators need to teach the proper curriculum as stated by their department, school, and
state while also using strategies that align with the curriculum. Educators need to assess the
correct ILOs in reliable and valid ways in order to improve their teaching and support their
students learning. Instructional leaders must oversee all of this. Instructional leaders need to
understand the purpose of each assessment, how the assessment connects to the curriculum, and
how instruction connects to both. Instructional leaders also need to know what to do with the
results of assessments and how those results can help improve teaching and learning in their
schools.

CRITIQUE OF AN ASSESSMENT

14
References

Gareis, C.R., & Grant, L.W. (2015). Teacher-made assessments: How to connect curriculum,
instruction, and student learning. Larchmont, NY: Eye on Education.
L. Grant (personal communication, September 2, 2015)
Virginia Department of Education. (2010). Virginia science standards of learning curriculum
framework 2010: Life science. Retrieved from
http://www.doe.virginia.gov/testing/sol/standards_docs/science/2010/curriculum_framew
k/life_science.pdf
Waugh, C., & Gronlund, N. (2013). Assessment of student achievement (10th ed.). Saddle River,
NJ: Pearson Education, Inc.

Appendix A