Food Science Assessment Report

Draft
Spring 2008
May 8, 2008

Review This Report

Introduction
This report of Food Science’s assessment focuses on the Program’s use of its Critical
and Integrative Thinking rubric as applied in a set of courses taken by Food Science
students and other students. The Food Science faculty have a variety of goals. To
address departmental diversity, meet process strategies for accountability required by
the professional accrediting association (The Institute of Food Technologists), WSU and
UI strategic plan and WSU’s regional accrediting body’s requirements, and, most
importantly, to improve the education of future Food Scientists, faculty in the Food
Science programs at WSU and at the UI have identified a single, comprehensive
outcome: Students who complete the program will demonstrate professional level
competence in scientific reasoning in Food Sciences. This is the fifth assessment cycle
for the program.

Goals and Outcomes

This assessment has three goals:

1. The assessment of student performance in Food Science’s overarching

outcome— demonstrate competent scientific reasoning —in order to improve the
ability of students to integrate information from multiple disciplines and to reason
scientifically and critically.

2. The assessment supports WSU and UI institutional goals—quantitative and

symbolic reasoning, critical and creative reasoning, information fluency, students’
communication skills and their understanding of self and society.

3. The assessment will help identify weaknesses and strengths to help improve and
promote the Food Science program.

Procedures and Methods

To assess scientific reasoning — A comprehensive sample of 48 student papers were

gathered from various courses at strategic points in the curriculum and rated by 10
participating faculty. Student papers from FS 470 (n=13), FS 303 (n=15), FS 220 UI
(n=5), FS 220 WSU (n=15) were selected and scored based on six criteria identified by

Food Science Outcomes - 1- May 8, 2008

Food Science faculty and an assessment consultants from CTLT to comprise the
scientific reasoning construct.

Faculty agreed that the double review process used by the Educational Testing
Services would be appropriate for establishing and monitoring inter-rater reliability and
consensus necessary for improving program coherency and faculty community. To
establish reliability, raters participated in a norming session, rating an identical paper
based on the scoring criteria, and then negotiated a consensus on how the criteria
would be applied. Further, by adopting the rigor embodied in the double review
approach faculty establish expert validity in the assessment process.

In the rating process, participants agreed that competent performance would be

reflected in a score of 4 or better on the 6-point rubric. Each paper was assessed by two
raters on each criterion of the rubric using a scale that ranges from 1-6. A score of 4
designates the desired level of competence of a WSU graduate; 6 represents
exceptional or mastery level performance. A competent student paper therefore
requires consensus of two independent raters. Scores that differed by more than one
point were reconciled by a third rater and the papers were further tagged for additional
review.

Performance Criteria

In order to effectively demonstrate competent scientific reasoning in FS, students

should be able to demonstrate the six essential subordinate or supporting outcomes
that are aligned with WSU’s institutionally generated student outcome goals and the UI
5 program learning outcomes. Students who complete the FS curriculum will therefore
demonstrate the ability to:

1. Identify, summarize, and define the issue or problem at hand.

2. Clearly state purposes, objectives, or hypotheses.
3. Present observations and results in a complete, logical and clear fashion.
4. Assess, discuss, and reconcile the supporting data/evidence acquired from the
exercise/experiment in relation to the existing scientific literature.
5. Identify and assess conclusions, implications, and consequences.
6. Effectively organize and articulate information to promote understanding and
communicate significance of the issue or problem.

The six criteria can be mapped to the WSU’s Six Goals of the Baccalaureate by
association as illustrated in Figure 1 and Figure 2:



Food Science Outcomes - 2- May 8, 2008

Figure 1. Critical and Integrative Thinking in Food Science

Figure 2 illustrates how the 6 institutional outcomes—critical thinking, quantitative and

symbolic reasoning, information literacy, communication, specialty, and self in society—
are embedded in the Food Science assessment strategy.

Figure 2. Mapping to the WSU’s Six Goals of the Baccalaureate

Food Science Outcomes - 3- May 8, 2008

FS Outcomes Mapped to the WSU Six Goals of the Baccalaureate

The outcomes assessed in FS implicitly map to WSU’s six goals of the baccalaureate.
The correspondence between WSU’s six goals and the FS assessment process has
been used here to provide a view of how FS’ students are performing on institutional
goals. Table 1 reflects a distribution of rubric dimensions to the WSU Six. Figure 3
represents the results of that distribution.

FS Outcomes Allocation to WSU6

Critical Quant & Information Communication Self & Specialty
Thinking Symbolic Fluency Society
Reasoning
Problem 3.49 3.49 3.49 3.49
Hypothesis 3.34 3.34
Observation 3.38 3.38 3.38 3.38 3.38
Data 3.23 3.23 3.23
Conclusions 3.18 3.18 3.18 3.18 3.18
Communication 3.40 3.40
WSU 6 3.32 3.28 3.32 3.40 3.35 3.34
Table 1. A distribution of rubric dimensions to the WSU Six

Figure 3. A Radio Graph of the Distribution of Outcomes to the WSU 6

Food Science Outcomes - 4- May 8, 2008

 Results for 2008

Inter-rater Reliability

Inter-rater reliability was determined using the ETS method—a calculation of the
percentage of overall agreement—which was used to monitor the reliability of the raters.
Acceptable levels of reliability are indicated by 75% or better using the ETS method.
The overall percentage agreement among raters in this session was 74%.

Rubric Dimension Course – Student Papers Inter-Item Reliability

FS 470 FS 303 FS 220 UI

(n=13) (n=15) (n=20)

ETS ETS ETS
Problem 85% 73% 75% 78%
Hypothesis 77% 80% 70% 76%
Observations 87% 73% 65% 75%
Data 100% 93% 90% 94%
Conclusions 69% 67% 65% 67%
Communication 62% 60% 85% 69%
Average 80% 74% 75% 76%

Table 2. 2008 Inter-rater Reliability by Course and Dimension

Figure 4. Inter-rater reliability by dimension

Food Science Outcomes - 5- May 8, 2008

Student Performance

Student performance is reported as an average paper rating by class and overall.

Future reports might include percentage of papers rated as competent at the level of 4
or better.

Course
FS 470 FS 303 FS 220 Overall
Rubric Dimension WSU and UI
(n=13) (n=15) (n=20) (n=48)
Mean Mean Mean Mean
Problem 3.81 3.10 3.57 3.49
Hypothesis 3.65 3.27 3.10 3.34
Observations 4.04 2.97 3.12 3.38
Data 4.00 2.73 2.97 3.23
Conclusions 3.77 2.77 3.00 3.18
Communication 4.00 2.97 3.23 3.40

Total Average 3.88 2.97 3.17 3.34

Table 3. 2008 Critical and Integrative Thinking Outcomes by Dimension and Course

Figure 5. Overall Ratings by Class

Assignment Assessment

In addition to assessing student performance in written papers, raters used the same
criteria to assess the assignments that students were responding to. The scoring
method assessed the extent to which the assignment expectations related to the six
dimensions of the rubric: explicit, implicit, or absent. These ratings are necessarily

Food Science Outcomes - 6- May 8, 2008

approximate, as they do not reflect the spoken instructions / discussions that proceeded
or accompanied the assignment or feedback students may have received as they
worked or revised (with or without a rubric). Nonetheless, the results are reported here
to inform faculty reflection on the alignment of the assignments with program goals.

Figure 6. Comparison of FS 303 Assignment Ratings

Faculty Feedback on the FS 303 Assignment Prompt

Dimension 1: Problem

• Explicit in works and phases but no explanation given (R1)

Food Science Outcomes - 7- May 8, 2008

• Hard to tell if student is to write a 'term paper' or 'proposal' these are different things.
Assuming that the topic is pretty open ended (but has to have a theoretical and
applied component), and this was probably presented in class, but is not clear from
the writing here. (R3)
• Does not clearly explain the need to identify or embedded or implicit aspects. (R4)
• Not entirely clear what you want. No introduction to the assignment given. I could
not do this assignment without instructor explaining verbally. (R5)

Dimension 2: Hypothesis

• Explicit in works and phases but no explanation given (R2)

• This is pretty clear from the outline, but presumes that a student knows what this
means already. I presume that there was explanation of this in class. (R3)
• Purpose or hypotheses not requested explicitly. (R 5)

Dimension 3: Observations

• Explicit in works and phases but no explanation given (R2)

• Noted in outline but no explanation provided (R3)
• More detail of expectations needed.(R5)

Dimension 4: Data

• A little hard to figure out how to do this in light of the first sentence (R3)
• More detail of expectations is needed. (R5)

Dimension 5: Conclusions

• Explicit in works and phases but no explanation given (R3)

• hard to figure out what the student is supposed to be concluding (results of
earlier studies) or push ideas and convince reader that this work would be worth
funding. (R3)
• Does not address consequences (R4)
• More detail of expectations needed. (R5)

Dimension 6: Communication
• I would have trouble trying to organize my thoughts for this, the structure is clear,
but would be difficult to implement. (R3)
• This really lacks detail on expectations. I could not do this assignment without
instructor explaining verbally. (R5)

• Assignment does not express the requirement for:

o Title is sufficiently descriptive and informative.

Food Science Outcomes - 8- May 8, 2008

 o Text is clear and well organized.
o Text contains few or no spelling or grammatical errors.
o The overall style and organization consider the audience, follow a logical
order, and unify all aspects of the project to maximize understanding of
the problem or issue.
o Supporting literature is fully and accurately cited (R4)

Figure 7. Comparison of FS 220 Assignment Rating

Food Science Outcomes - 9- May 8, 2008

Faculty Feedback on the FS 220 Assignment Prompt

Dimension 1: Problem

• Written English is a bit difficult to follow and some instructions are not all that clear
(R3)

Dimension 2: Hypothesis

• Critique does not usually require that there be a hypothesis (for sure). An outline
should include objectives or a roadmap for the paper, but whether this would be
understood by students would depend on what level they are at in college.(R3)

Dimension 3: Observations

• Could explain more (R5)

• Critique of a current topic does not necessarily involve analysis of any type of
data. If I had this assignment, I would be a bit confused as to what type of data I
would need to be evaluating. It is possible that the student would not know how
the data were collected - since their primary source documents could be review
articles and not reports in the peer reviewed literature. (R3)

Dimension 4: Data

o This implies that the paper will involve an analysis of primary literature and not
review literature and also that the paper will focus on scientific reports rather
than policy studies.(R3)

Dimension 5: Conclusions

• Explain more here (R5)

• Charts and tables would not belong in this section, but instead in the results. This
should be a written conclusion. (R3)

Dimension 6: Communication
More detail here would help. Depending on a student’s perspective of what this
assignment is supposed to be about (e.g. science vs policy)they could have
major problems with trying to write a paper with the proposed structure. The
assignment is "to critically analyze information about a topic related to food
safety and quality and to write a paper which critiques, summarizes and
articulates the current level of knowledge concerning that topic". This is pretty
vague. (R3)

Food Science Outcomes - 10- May 8, 2008

Figure 8. Comparison of FS 470 Assignment Rating

Faculty Feedback on the FS 470 Assignment Prompt

Dimension 1: Problem

• Pretty clear (R3)

Dimension 2: Hypothesis

• Objectives are clear but request for hypothesis is not (R3)

Dimension 4: Data

• No mention of the use of statistics (R4)

Dimension 6: Communication

Food Science Outcomes - 11- May 8, 2008

 • Assignment does not fully express the requirement for:
o Title is sufficiently descriptive and informative.
o Text is clear and well organized.
o Text contains few or no spelling or grammatical errors.
o The overall style and organization consider the audience, follow a logical
order, and unify all aspects of the project to maximize understanding of
the problem or issue.
o Supporting literature is fully and accurately cited (R4)
• Good assignment to fit the rubric. (R5)

Comparison of Student Work and Assignment Prompt Ratings

Figure 9. Comparison of Ratings for Student Work and the Assignment Prompt for FS
202

Food Science Outcomes - 12- May 8, 2008

Figure 10. Comparison of Ratings for Student Work and the Assignment Prompt for FS
303

Figure 11. Comparison of Ratings for Student Work and the Assignment Prompt for FS
470

Food Science Outcomes - 13- May 8, 2008

 Discussion and Next Steps for 2008 – 2009

Make the Most Out of Results

1. The next step will be to meet to discuss what the assessment data means and to
identify and implement program changes. This meeting will solicit faculty
reflections on strategies for revising assignments to improve program outcomes.
(The reflections are invaluable evidence of quality enhancement essential for
accreditation as well as program improvement.)

The program has made solid progress in formalizing the assessment of the three
goals—student competence in scientific reasoning, relating the assessment of that
competence to the WSU Big Six, and implementing a process that will help establish
Food Science.

Program Innovations

Throughout the study, noteworthy innovations have been integrated into Food Science:

1. A formalized, annual time-line for program assessment management was

developed. This model is a one year cycle oriented toward student assessment in
selected classes.

2. A schedule of ad hoc faculty meetings with participating faculty to discuss

assessment issues and strategies for revising assignments and prompts to improve
program outcomes was implemented.

3. Changes were incorporated in Food Science syllabi to expand the scientific

reasoning concept and to incorporate it as a key aspect to the assignments.

4. The assessment of student work and assignment prompts was completed online
using the Skylight survey tool. This innovation offers the option of more convenient
asynchronous rating after a brief face to face norming session.

Selecting Next Steps

We make suggestions below for next steps but without investment from faculty progress
is unlikely. Ultimately the Food Science faculty will have to make a commitment to
improvements of their choice, identify the players needed to implement them, and
create a timeline with milestones. Such choices, even if limited in number, will guide
future assessment efforts in the classroom settings experienced by Food Science
students.

Food Science Outcomes - 14- May 8, 2008

Suggestions for next steps

Measure what you value and value what you measure. If the learning outcomes
identified by faculty are important enough to guide the assessment of the program, they
are important enough to communicate to students as a guide for their learning.

There are many ways to communicate the value of the learning outcomes to the
students.
• Give the rubric to students when they enter the program. Make it clear to them that
these criteria will be the measure of their success in the program and of their
readiness to enter the professional community. . The grading scale is absolute—
professional competence—faculty, professionals in the industry, peers, and they
will continuously assess their performance, using that absolute standard, as they
move through the program

• Let students know that through ongoing and consistent feedback they will
improve in their understanding and practice of these criteria. Coach the students
in using the rubric to self assess and assess peers. (CTLT can help with this)

• Involve Students in the Process: On the first day of classes, give a copy of the
rubric to your students along with the syllabus.
o Conduct a short norming session with students early in the semester.

o Active/Collaborative Learning: Incorporate assignments into the course

that require students to give each other feedback on their use /
interpretation of the rubric.

o Encourage Revisions: Allow students to revise written assignments in

consultation with the rubric and have them turn in both first and revised
drafts with indications of where the rubric was used to revise.

• Coach faculty in using the rubric to self assess, assess peers, and students.
(CTLT can consult with faculty individually or in group workshops)

• Alignment: Monitor the alignment of assignment prompts with the outcomes

criteria as identified in the rubric.

• Invite students, TA’s, professionals, community colleagues, and

administration to participate in rating sessions. Involve them in assessment of
student work, assessment and revision of assignments, and discussion of
innovations.

Food Science Outcomes - 15- May 8, 2008

Make Use of Rating Session Results

• Meet regularly to identify and implement changes. Identify trends and problem
areas in student performance. Solicit faculty, professional and student reflections
on strategies for revising assignments to improve program outcomes (The
reflections are invaluable evidence of quality enhancement essential for
accreditation as well as program improvement).

• Develop a formalized timeline for rating sessions and the use of their results.
The model below repeats as part of an ongoing process

Model Time-line for one Semester in an Assessment Cycle

Pre-term Mid-semester Post-term

Design course, Provide Provide explicit Continue using Conduct outcomes

assignments and outcomes criteria mid-term criteria to assessment of student
activities. to students and feedback to support student work and of assignments.
opportunity to students using learning.
Collaboratively norm on sample criteria and Use results to enhance
review these work. collect feedback alignment of outcomes ,
materials to from students assignments, activities,
identify and Use criteria to about criteria and and to identify
enhance the support student assignments. refinements for next
alignment of learning (faculty cycle.
embedded feedback, peer Examine how the
outcomes criteria. feedback, self- criteria are being
reflection). used; faculty
share
Provide observations and
opportunities for make
students to adjustments to
revise their work rubric,
based on assignments,
feedback. processes, etc.

Food Science Outcomes - 16- May 8, 2008

 2008
Food Science, with assistance from the Center for Teaching, Learning, & Technology
W
Waasshhiinnggttoonn S
Sttaattee U
Unniivveerrssiittyy

Food Science Outcomes - 17- May 8, 2008

Food Science Outcomes - 18- May 8, 2008