2/5/2014

Analysis of Open‐Ended Survey Responses –
Where to start?

Yao Zhang Hill, Ph.D.
Institutional Researcher
Kapi‘olani Community College

In Support of Assessment Office’s Mission

Improve student learning through 
program assessment

1
 2/5/2014

Workshop Outcomes
You will be able to:
• List the basic steps involved in analyzing open‐ended 
1 survey responses

• Carry out preliminary open‐ended survey responses 
2 analysis

• Know how to report the results
3

• Use results
4

Program Assessment: It’s a Cycle
Student Learning 
Outcomes

Opportunities to 
Improvement Plan
Learn

Collection & 
Interpretation of 
Analysis of 
Results
Evidence

2
 2/5/2014

Steps to take
1: Careful Reading
2: Unitization
3: Categorization
4: Coding
5: Finding patterns and trends
6: Reporting and Using

Step 1 Careful Reading

(D) Classes are small and 
fun. Many teachers care 
about their students.

3
 2/5/2014

Step 2 Unitization: Break it Apart

• Break responses into thought units/sense 
units/units of meaning
• Each unit communicates only one idea
• Can be several sentences, one sentence, text 
chunks, or one work.

How about:  , “?” or “…”

Example
(G) Class size and teachers were engaged with the 
students.

(G) Class size

(G) Teachers were engaged with the 
students.

4
 2/5/2014

Example

(A) Small class size and access to resources, like STEM

(A) Small class size

(A)Access to resources, like STEM

Step 3 Categorization
• How much material to use?
– All if possible. If not, try a balanced random sub‐
sample
• How to go about it?
– Deductive‐inductive procedure
– Skip difficult ones first
– Select a sample (30) and check your code against your 
colleagues’.
• How detailed should it be? 
– Balance between verifiability and consistency
• Aggregate or break it done?

5
 2/5/2014

Example Aspects of Categories
Accreditation 
General Learning outcomes Standards
• People • Attitudes • Instructional 
programs
• Perspective • Feelings
• Student support 
• Activities • Knowledge
services
• Processes • Skills
• Library and 
• Relationship and  learning support 
social structure services
• Physical resources
• Technology 
resources

Example: Generating categories

Class Size
Instructional Faculty
Faculty & Staff
Counselor/Advisor
Campus Resource

6
 2/5/2014

Hard to deal with categories:
• Other
• N/A
• Not relevant
• None

Step 4 Coding

• Label each comment unit with the category.
• Decide to collapse categories or break it down 
more  Step 3
• Check your work 
– with another colleague (reproducibility)
– against yourself at a different time (stability)

7
 2/5/2014

Example: Coding
Comment Category
(A) Small class size Class Size
(A) Access to resources, like STEM Campus Resource
(B) I really liked how the teachers…were.
(B) I really liked how the …counselors were.
(C) Instructor … are very easy to talk to and they are 
professional
(C) …Advisors are very easy to talk to and they are very 
professional
(D) Classes are small and fun.
(D) Many teachers care about their students
(E) Supportive faculty…
(E) Supportive … counselors
(F) Great resources
(G) Class size
(G) Teachers were engaged with the students

Example: Coding
Comment Category
(A) Small class size Class Size
(A) Access to resources, like STEM Campus Resource
(B) I really liked how the teachers…were. Instructional Faculty
(B) I really liked how the …counselors were. Counselors/Advisors
(C) Instructor … are very easy to talk to and they are  Instructional Faculty
professional
(C) …Advisors are very easy to talk to and they are very  Counselors/Advisors
professional
(D) Classes are small and fun. Class size
(D) Many teachers care about their students Instructional Faculty
(E) Supportive faculty… Instructional Faculty
(E) Supportive … counselors Counselors/Advisors
(F) Great resources Campus Resource
(G) Class size Class size
(G) Teachers were engaged with the students Instructional Faculty

8
 2/5/2014

Example: Coding Sentiment/Effect
Comment Category
(B) I really liked how the teachers…were. Instructional Faculty
(C) Instructor … are very easy to talk to and they are 
Instructional Faculty
professional
(D) Many teachers care about their students Instructional Faculty
(E) Supportive faculty… Instructional Faculty
(G) Teachers were engaged with the students Instructional Faculty
(B) I really liked how the …counselors were. Counselors/Advisors
(C) …Advisors are very easy to talk to and they are very 
Counselors/Advisors
professional
(E) Supportive … counselors Counselors/Advisors
(D) Classes are small and fun. Class size
(A) Small class size Class Size
(G) Class size Class size
(A) Access to resources, like STEM Campus Resource
(F) Great resources Campus Resource

Example: Coding Sentiment
Comment Category Sentiment
Instructional 
(B) I really liked how the teachers…were.
Faculty
(C) Instructor … are very easy to talk to and  Instructional 
they are professional Faculty
Instructional 
(D) Many teachers care about their students
Faculty
Instructional 
(E) Supportive faculty…
Faculty
Instructional 
(G) Teachers were engaged with the students
Faculty
(F) Some teachers always come to class late. I  Instructional 
have a math teacher who came to class one  Faculty
hour late one time.
… … …

9
 2/5/2014

Example: Coding Effect
Comment Category Effect
(B) My instructor’s presentation is clear. It 
Instructional 
helped me understand the course
content.
Faculty 1
(C) The advise that my instructor gave me  Instructional 
helped me through the college. Faculty 2
(D) The two courses that I took from Prof. 
XXX is so useful. The skills that I learned 
Instructional 
from these courses helped me find a job 
in the field of assessment. It changed my 
Faculty 3
life!!!

Coding Scheme
1: Small Effect
2: Medium Effect
3: Large Effect

Step 5 Identify the Patterns & Trends
• Make the obvious obvious
• Make the obvious dubious
• Making the hidden obvious 
(Schlechty & Noblit, 1982)

10
 2/5/2014

Techniques (1)
• Identify dominant themes
– “The greatest strength of the XXX department 
recognized by the respondents is [faculty]!”
– “... is another common theme raised by the 
students to demonstrate the strength of the 
department”
– “The main issues mentioned are…”
– “The most prevalent theme/factors are…”
– “To a lesser extent, … were mentioned.”

Techniques (2) 
• Identify relationships (advanced)
– Native Hawaiian students tend to believe…
– Tenure track faculty are more likely to raise the 
issue of …
– Part‐time students are in a more dire need of … 

11
 2/5/2014

The report: To Write or Not to Write

“I find in my own work that final reports 
frequently have less impact than the direct, 
face‐to‐face interactions I have with primary 
evaluation users to provide them with feedback 
about evaluation findings and to share with 
them the nature of the data.” (Pattern, M., 
2002, p. 510)

If you must write a report…
• Use executive summary for long reports
• Focus on the patterns and trends and the 
unexpected
• Use quotes to bring it to life
• Use bullet points, underlines, italics to make it 
easy to read
• Summarize the findings in tables and charts

12
 2/5/2014

Step 6 Use – To Celebrate

Finding:

85% the respondents 
demonstrated gain in civil 
knowledge and sense of 
civil responsibility after 
participating in the 
Service‐Learning Program

Step 6 Use – To Improve the Program

Finding: Diagnose 1: 
An overwhelming  Under‐Staffed
complaint emerged is 
about the transcript  Strategy 1: 
review process being too  Expedite hiring 
long (e.g., over a year).

Strategy 2: 
Diagnose 2: 
Automatic 
Manual work
Processing

13
 2/5/2014

Step 6 Use – To Improve the Instrument

• Can you think of option categories for the closed‐
ended question:
What are the important factors that helped you 
through the college? (Choose all that apply)
 Instructors/Professors
Counselors/Advisors
Campus resources (e.g., library, STEM Center)
Class size
…

Summary of the steps

Reading

Categorization
Unitization Patterns & 
Trends
Coding
Report & 
Use

14
 2/5/2014

Questions?

15
Analyzing Open-Ended Responses Workshop at UHM 9/14/12 Yao Zhang Hill
yaozhang@hawaii.edu

Sample Coding from the 2010 KapCC Professional Development Day Evaluation Survey Responses

Question: Please provide any comments on the organization of the Category Sentiment
event.
1 I was also signed up for clean-up, but since the lunch was served later Clean-up Irrelevant
than planned, I wasn't able to help because I had another meeting later.
I felt badly so I stopped by again but they did not seem to need my
help.
2 This was the best Faculty Development day we've ever had at K.C.C.. General (best) +
I felt a sense of human warmth. I also got so much more out of Collegial connections +
'playing' and 'learning with clay' and learning from Kauka de Silva Culture Learning +
than any meeting about something like assessment or some other Food +
pedagogical workshop. The reason this was the best one we've ever +
had was because it was rooted in spirit and identity of our namesake,
Queen Kapi'olani. I was so happy and honored to learn about the
roots of the Native Hawaiian culture and how it gives meaning to
everything we do here. Please make next semester or next year's
seminar similar..(copied to suggestions).the food was great!
3 very different from all previous PD days. not necessarily better, just General (different) Neutral
different.
4 The food was excellent, the taste, the presentation and the traditional Food +
preparation.
5 I didn't get word until the last minute about what time the event began. Info/communication -
The Hawaiian food was great, but once again, we were unsure what Food +
time lunch would begin, and got hungry waiting for lunch service to Activities +
begin. Working in the garden was relaxing, fun and educational!
6 It was difficult to find specific information on the options. I was Info/communication -
denied entry to the website on laulima when I tried to log in. The Registration -
event at the Arboretum was fine. Activities +
… … … …

Sample Reports
Lande, M., Parikh, S., Sheppard, S., Toye, G., Chen, H., & Donaldson, K. (2009). More to say: Analyzing
open-ended student responses to the Academic Pathways of People Learning Engineering Survey.
In American society for engineering education conference proceedings. Retrieved from
http://www.engr.washington.edu/caee/CAEE_Briefs_PDFs/MoretoSay_Lande_ASEE09.pdf
Hunter, A. (2006). Report on the qualitative analysis of five open-ended questions from the Carleton
College off-campus studies alumni survey of the Marine Biology Seminar. (Report to Carleton
College) Boulder, CO: University of Colorado at Boulder, Ethnography & Evaluation Research.
Retrieved from
http://www.colorado.edu/eer/downloads/CarletonMarineBiologyExecutiveSummary2006.pdf

1
Analyzing Open-Ended Responses Workshop at UHM 9/14/12 Yao Zhang Hill
yaozhang@hawaii.edu

Selected Sample Report from the 2010 KapCC Professional Development Day Evaluation Survey
Results

Summary of the Comments on the Evaluation of the Organization of the Event (Total comment units=45)
25 out of 54 (46%) of
Ϭ Ϯ ϰ ϲ ϴ ϭϬ
the participants provided
&ŽŽĚ ϲ feedback. In general they
ϭ
ϲ seemed to be happy with the
'ĞŶĞƌĂů
organization of the event. The
ĐƚŝǀŝƚŝĞƐ ϰ organization of the event
ŽůůĞŐŝĂůĐŽŶŶĞĐƚŝŽŶƐ ϯ received positive comments
>ĞĂĚĞƌƐ ϯ mostly in general
>ĞĂƌŶŝŶŐ Ϯ organization, food, team
activities, team leaders, and
/ŶƐƉŝƌĂƚŝŽŶ Ϯ
the feeling of collegial
/ŶĨŽͬŽŵŵƵŶŝĐĂƚŝŽŶ connection.
ϴ
ƚƚĞŶĚĂŶĐĞ ϯ The most common negative
WƵďůŝĐŝƚLJ comments were about the
ϯ lack of information on the
ZĞŐŝƐƚƌĂƚŝŽŶ Ϯ schedule and the location of
the activities, not getting the
WŽƐŝƚŝǀĞ EĞŐĂƚŝǀĞ word out enough, attendance,
and registration process.
See Appendix 1 for the full responses and their coding.

Quotes:
Food:
-“The food was excellent, the taste, the presentation and the traditional preparation.”
-" The Hawaiian food was great."
-“ …lunch was very good.”
- “Best Hawaiian food I have ever had!”
…
Collegial connection:
-“Loved the bonding that occurred”
-“ I feel enriched by this whole experience and a new fondness for my colleagues.”
…
Info/Communication:
Ͳ “… the beginning was kinda bumpy getting information about the sessions, and the times”
Ͳ “Never found where lunch was being served. Went to cafeteria, but it was not there. Couldn't find any
reference to lunch location in WILD day flyers, emails, etc. Finally gave up with no lunch.”
Ͳ “I didn't get word until the last minute about what time the event began. “

Advanced Reading with Practical Guidance and Examples

LeCompte, M. D. (2000). Analyzing qualitative data. Theory into practice, 39(3), 146-154.
Patton, M. Q. (2002). Qualitative Research & Evaluation Methods (3 ed.). Thousand Oaks, CA: Sage.
Ruona, W. E. A. ( 2005). Chapter 14: Analyzing qualitative data. In R. A. Wanson, & E. F. Holton III,
(Eds). Research in organizations: Foundations and methods of inquiry (pp. 233-263). San
Francisco, CA: Berrett-Koehler Publishers, Inc. Retrieved from
http://203.128.31.71/articles/157675314X.pdf#page=252
Thoms, D. R. (2006). A general inductive approach for analyzing qualitative evaluation data. American
Journal of Evaluation, 27(2), 237-246. doi: 10.1177/1098214005283748