Rubric Assessment of Student

Responses to an Information
Literacy Tutorial
Megan Oakleaf
Librarian for Instruction & Undergraduate
Research
Steve McCann
NCSU Libraries Fellow
M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004
Presentation Objectives
Share our experiences.
In changing the way we assess our program
In adapting ACRL outcomes to our project
In selecting a learning artifact to assess
In piloting our assessment plan
Facilitate your experience with this
type of assessment.
Provide take away ideas.



NCSU Libraries
Instruction Program
M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004
Information Literacy
Information literacy is a set of abilities
requiring individuals to "recognize when
information is needed and have the
ability to locate, evaluate, and use
effectively the needed information."

Association of College and Research Libraries
http://www.ala.org/ala/acrl/acrlstandards
/informationliteracycompetency.htm
M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004
How have we
measured our success?
M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004
The Decision to Change
Goal: To assess student learning of
information literacy skills using
outcomes-based assessment.

Need 2 things
An Artifact to Assess
Outcomes to Measure
M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004
What artifact will we assess?
Interactions with students that could
yield assessment artifacts

50-minute one-shot workshops

Library Online Basic Orientation (LOBO)
M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004
Disadvantages of Using
Workshops for Assessment
Perceived lack of time for open-ended
responses, only m.choice and T/F are
options.
Taught by numerous librarians who lack
assessment knowledge.
Inconsistent audiences & content.
Incomplete spectrum of outcomes
addressed.
M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004
Advantages of Using
LOBO for Assessment
Forms basis for IL instruction at NCSU.
Reaches virtually all incoming freshmen.
Recently redesigned, includes open-
ended questions.
Captures student responses in a
searchable database.
Potential for rich data.
M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004
What outcomes will we assess?
Information Literacy Competency
Standards for Higher Education
5 Standards
22 Performance Indicators
87 Outcomes

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004
What outcomes will we assess?
Objectives for Information Literacy
Instruction: A Model Statement for
Academic Librarians
Standards = 5
Perform. Indicators = 14
Outcomes = 35
Bullets = 133

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004
What outcomes will we assess?
LOBO Objectives & Outcomes

Objectives = 5

Measurable
Outcomes= 45

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004
First Steps
Set up DB to access student answers.

Match outcomes to questions.

How will we know the outcomes been met?
Beginning, Developing, Exemplary

Pilot test a section of LOBO.
M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004
The Pilot Test
Problem: Students use web sources for
academic purposes without evaluating
their quality.

Are they duped by low-quality sites?
Can we teach them to be more
critical consumers of information?
M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004
Evaluating Web Sites
What criteria are you looking for?
What clues can you find?
What specific example can you give
from the web site at hand?
Is the web site a good one for
you to use?
Evaluating Web Sites
M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004
Pilot Test Results
What we found out from 50 randomly
selected student accounts
M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004
LOBO Sections - %Completed
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1. The
Research
Process
2. Defining
Research
Needs
3. Developing
a Research
Strategy
4. Conducting
the Search
5. Evaluating
Resources
6. Using
Resources
Lobo DataJan-April, 04
1,830 total accounts new
accounts
46% of total questions
answered
Evaluated 50 accounts
Sections Total Answers Ttl Quest. Poss. Ans. % Complete
1. The Research
Process
7,031 7 13,230 53%
2. Defining Research
Needs
9,056 10 18,900 48%
3. Developing a
Research Strategy
9,798 11 20,790 47%
4. Conducting the
Search
11,984 12 22,680 53%
5. Evaluating
Resources
11,582 16 30,240 38%
6. Using Resources 4,692 6 11,340 41%
M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004
Objective 5: Evaluating Resources
#3 Evaluate Web Sites questions:
1. Locate a website
2. Evaluate websites authority
3. Evaluate recency/currency
4. Identify Bias/Point of View


Section 5: Evaluating Resources
Task Answered Null Possible % Complete
1. Evaluate Books 4,455 8,824 13,279 34%
2. Evaluate Articles 1,570 2,224 3,794 41%
3. Evaluate Web Sites 5,557 7,722 13,279 42%
M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004
Q1: Locate a Website
4 points possible: Average score 2.7

Question Text: Type the title and URL (web address) of the web site you will
evaluate here:
Q1: Cumulative Count of Scores
0
5
10
15
20
25
30
35
40
Total 0 3 36 0 11
Count-0 Count-1 Count-2 Count-3 Count-4
Q1: Count of Scores by Outcome
0
5
10
15
20
25
30
35
40
45
Beginning Developing Exemplary Beginning Developing Exemplary
Completion Accuracy
M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004
Q2: Evaluate a Websites Authority
8 points possible: Average score 5.0

Question Text: Answer the questions above for the web site you're evaluating.
Overall, does what you know about the URL of the web site indicate that it's a good
resource?
Answer the questions above for the web site you're evaluating. Overall, does what
you know about the authorship of the web site indicate that it's a good resource?
Q2: Cumulative Count of Scores
0
2
4
6
8
10
12
14
16
18
Total 0 5 1 2 6 16 9 7 4
Count-
0
Count-
1
Count-
2
Count-
3
Count-
4
Count-
5
Count-
6
Count-
7
Count-
8
Q2: Count of Scores by Outcome
0
5
10
15
20
25
30
35
40
B
e
g
i
n
n
i
n
g
D
e
v
e
l
o
p
i
n
g
E
x
e
m
p
l
a
r
y
B
e
g
i
n
n
i
n
g
D
e
v
e
l
o
p
i
n
g
E
x
e
m
p
l
a
r
y
B
e
g
i
n
n
i
n
g
D
e
v
e
l
o
p
i
n
g
E
x
e
m
p
l
a
r
y
B
e
g
i
n
n
i
n
g
D
e
v
e
l
o
p
i
n
g
E
x
e
m
p
l
a
r
y
Uses Criteria Terms Cites Clues Cites Examples from Source Judges Whether or Not To
Use
M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004
Q3: Evaluate a Websites Currency
8 points possible: Average score 5.6

Question Text: Answer the questions above for the web site you're evaluating.
Overall, does what you know about the currency of the web site indicate that it's a
good resource?
Q3: Cumulative Count of Scores
0
2
4
6
8
10
12
14
16
Total 0 6 0 5 4 2 12 6 15
Count-
0
Count-
1
Count-
2
Count-
3
Count-
4
Count-
5
Count-
6
Count-
7
Count-
8
Q3: Count of Scores by Outcome
0
5
10
15
20
25
30
35
B
e
g
i
n
n
i
n
g
D
e
v
e
l
o
p
i
n
g
E
x
e
m
p
l
a
r
y
B
e
g
i
n
n
i
n
g
D
e
v
e
l
o
p
i
n
g
E
x
e
m
p
l
a
r
y
B
e
g
i
n
n
i
n
g
D
e
v
e
l
o
p
i
n
g
E
x
e
m
p
l
a
r
y
B
e
g
i
n
n
i
n
g
D
e
v
e
l
o
p
i
n
g
E
x
e
m
p
l
a
r
y
Uses Criteria Terms Cites Clues Cites Examples from Source Judges Whether or Not To
Use
M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004
Q4: Identify a Websites Bias
8 points possible: Average score 4.1

Question Text: Answer the questions above for the web site you're evaluating.
Overall, does what you know about the bias of the web site indicate that it's a good
resource? Overall, is this web site a good resource to use for your assignment?
Q4: Cumulative Count of Scores
0
2
4
6
8
10
12
Total 1 8 3 4 10 11 8 3 2
Count-
0
Count-
1
Count-
2
Count-
3
Count-
4
Count-
5
Count-
6
Count-
7
Count-
8
Q4: Count of Scores by Outcome
0
5
10
15
20
25
30
35
40
B
e
g
i
n
n
i
n
g
D
e
v
e
l
o
p
i
n
g
E
x
e
m
p
l
a
r
y
B
e
g
i
n
n
i
n
g
D
e
v
e
l
o
p
i
n
g
E
x
e
m
p
l
a
r
y
B
e
g
i
n
n
i
n
g
D
e
v
e
l
o
p
i
n
g
E
x
e
m
p
l
a
r
y
B
e
g
i
n
n
i
n
g
D
e
v
e
l
o
p
i
n
g
E
x
e
m
p
l
a
r
y
Uses Criteria Terms Cites Clues Cites Examples from Source Judges Whether or Not To
Use
M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004
Pilot Test Findings
Students proven successful with
mechanical tasks like checking currency
and identifying URLs.
Students are shown as developing
with judgment tasks such as authority
and bias.
Bias is a potential target area.
M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004
How are we using our
assessment results?
Changes to LOBO
Add and Reorganize Content
Improve Question Format
Enlarge Response Space
Make Rubrics Available?
Train Course Instructors
Changes to LOBO

Answer these questions about the web site youre evaluating in the space below:

Who created the site? What point of view do they represent?
What organizations support the site? What biases might they have?
Are links included that point to other viewpoints?
Are there signs of bias included in the site?
Are you biased toward the site?

Overall, does what you know about the bias of the web site indicate that its a good resource?

Replace
Questions
with
Content
Sample Student Answers
Get Help with Your Answer
Fix
Questions,
Add Links,
Enlarge
Answer
Space
Add Viewlet
to Model
Application
of Content
Changes Coming Soon!
Changes for Instructors
Share Rubrics,
Continue Ongoing
Training,
Add Lesson Plans
M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004
How are we using our
assessment results?
Changes to the Instruction Program
Including assessment in departmental 3-yr goals
Sharing data with subject-specialist librarians
Initiating rubric assessment of advanced
instruction
Reporting results to library administration
and colleagues on other campuses
M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004
Key Learning
Rubrics require lots of revision!
Rubrics are effective in measuring
higher-level thinking skills.
Rubrics provide information
administrators can use for reporting
and instructors can use to improve
teaching and learning.
Our colleagues are interested in
our progress.
M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004
Take Aways
1. Can higher-level thinking skills like
information literacy or critical thinking be
adequately described in a rubric?
2. Are rubrics good tools for assessing student
responses to tutorials?
3. What problems did you find in your practice
with these rubrics?
4. What part of this process/project could be
applied at your institution?
M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004
Muddiest Points?
Contact Information
Megan Oakleaf

Librarian for Instruction &
Undergraduate Research

919-513-0302
megan_oakleaf@ncsu.edu
Steve McCann

NCSU Libraries Fellow


919-513-7080
steve_mccann@ncsu.edu


Rubric Assessment of Student
Responses to an Information
Literacy Tutorial
Megan Oakleaf
Librarian for Instruction & Undergraduate
Research
Steve McCann
NCSU Libraries Fellow