Escolar Documentos
Profissional Documentos
Cultura Documentos
Monica Cano
Allison Davis
Loyola University Chicago
2
Table of Contents
Introduction ..................................................................................................................................... 4
Program Overview .......................................................................................................................... 4
Department Overview ................................................................................................................. 4
Program Context ......................................................................................................................... 4
Program Description................................................................................................................... 5
Participants ................................................................................................................................. 6
Application Process..................................................................................................................... 7
Program Purpose ........................................................................................................................ 8
Logic Model .................................................................................................................................... 8
Inputs ........................................................................................................................................... 9
Program Stakeholders ................................................................................................................. 9
Assumptions............................................................................................................................... 10
External Factors ........................................................................................................................ 10
Outcomes ................................................................................................................................... 11
Previous Evaluation ...................................................................................................................... 12
General Evaluation Approach ....................................................................................................... 12
Quantitative Approach .................................................................................................................. 13
Survey Population ..................................................................................................................... 13
Survey and Evaluation Design .................................................................................................. 14
Survey Instrument ...................................................................................................................... 15
Pilot Testing .............................................................................................................................. 15
Statistical Analysis and Reports ................................................................................................ 16
Qualitative Approach .................................................................................................................... 17
Focus Group.............................................................................................................................. 17
Focus Group Participants ......................................................................................................... 17
Protocol Description ................................................................................................................. 18
Focus Group Implementation ................................................................................................... 19
Focus Group Analysis ............................................................................................................... 20
Final Results Presentation ............................................................................................................. 20
Validity ......................................................................................................................................... 21
Limitations .................................................................................................................................... 22
Next Steps ..................................................................................................................................... 22
Appendix A: Returning Rambler Application .............................................................................. 24
Appendix B: Ramble Application................................................................................................. 28
Appendix C: Ramble Survey (Previous Evaluation) .................................................................... 32
Appendix D: Logic Model ............................................................................................................ 37
Appendix E: Quinlan Ramble Survey........................................................................................... 38
Appendix F: Initial Survey Email ................................................................................................. 45
Appendix G: Informed Consent Document .................................................................................. 46
Appendix H: Survey Construct Map............................................................................................. 49
Appendix I: Focus Group Email Invitation .................................................................................. 51
Appendix J: Focus Group Protocol and Script ............................................................................. 53
Appendix K: Consent and Demographic Form ............................................................................ 57
Appendix L: Coding Rubric-Priori Codes .................................................................................... 60
Appendix M: Evaluation Timeline ............................................................................................... 61
Appendix N: Evaluation Budget ................................................................................................... 62
Presentation Slides ........................................................................................................................ 63
References ..................................................................................................................................... 66
Introduction
The Quinlan Ramble is an immersive short term experience for business students that is
rare at the undergraduate level. In its short history, the program has received positive feedback
from participants, but has had very little formal evaluation. The goal of this evaluation project is
to provide data to show whether the program is achieving its stated outcomes, as well as
determine areas of improvement. As leadership continues to change within the university, it is
increasingly important to be able to reinforce the necessity for unique experiential programs like
the Quinlan Ramble. In this plan we have included contextual information, a logic model
outlining important stakeholders, as well as rationale for our evaluation approach. This plan also
includes detailed information on the implementation and data analysis of a mixed-methods
evaluation, which includes an online survey and a focus group.
Program Overview
Department Overview
The Quinlan School of Business offers undergraduate and graduate degrees. Quinlan
offers majors and minors in accounting, economics, entrepreneurship, finance, human resources,
information systems, international business, management, marketing, operations management,
and sport management. Quinlan also offers two dual degree programs. The student services
coordinator in charge of the Quinlan Ramble is part of the Undergraduate Programs office. The
office is overseen by the Assistant Dean of Undergraduate Programs, who reports to the Dean of
the Quinlan School of Business. Last years Ramble occurred under the leadership of an interim
dean, and a new dean was appointed over the summer of 2015.
Program Context
The Quinlan School of Business is located within Loyola University of Chicago. The
university has an enrollment of 15,902 students. The Quinlan School of Business is one of
eleven schools and colleges that make up the university. Loyola University Chicago is a Jesuit
Catholic institution. The Jesuit education system focuses on values like service, faith, and global
awareness. The climate at Quinlan is influenced by Loyolas Jesuit values. One of the five
characteristics of a Jesuit education is a values-based leadership with ethical behavior in
business and in all professions (Mission and Vision Statement, n.d.). The trips incorporate a
social justice focus through informal reflection on the values of the companies visited.
Program Description
The Quinlan Ramble is an immersive experience into the business world for
undergraduate students. The program consists of traveling to one or two major cities outside of
Illinois during Loyolas spring break. In the past, the students have visited Dallas and Austin,
Texas, and Seattle, Washington. Throughout the trip, students visit between 7 to 10 companies.
At each company, the students tour the facility and meet with leaders from the organization.
Quinlan Ramblers do research on companies and business contacts prior to departing on the trip
to prepare them for meetings with members of the organization. In addition to the business
focused activities, there are also local tourist attractions scheduled for the Ramblers. One area
the facilitators want to include in the future is intentional reflection. Currently, reflection on
company values is done informally.
The Quinlan Ramble is a unique experience because similar trips are very uncommon at
the undergraduate level among other institutions. This model is primarily seen at the graduate
level. However, there are examples of other immersive trips that focus on one specific academic
field. The Ramble has been modeled after a trip out of American University's Kogod School of
Business called the Road Scholars program (Road Scholars Program, n.d.). It is one of the only
other schools to offer a trip at the undergraduate level. Quinlan has taken aspects of this trip and
modified them to better fit its own student population.
While there is limited literature pertaining to spring break trips of this nature, there does
exist literature about alternative spring breaks, which are usually service-learning oriented.
Niehaus and Inkelas (2015) pointed to evidence in their research that indicated the way
alternative breaks play a role in students career development, which is a key outcome for the
Ramble program. The results of their study showed that students who participated in the
alternative break program shifted their career plans in some way (Niehaus & Inkelas, 2015).
Other research points to the fact that short-term immersion and study abroad programs still
contribute to student learning and development, whether or not the program meets its outcomes
(Jones, Rowan-Kenyon, Ireland, Neihaus, & Skendall, 2012). The writers suggest that, upon
their return from the trip, participants engaged in reframing to integrate what they learned during
the experience into their everyday lives and future plans.
Participants
The trip typically consists of 14-16 undergraduate business students, one professional
staff member, a graduate assistant, and one faculty member. The full-time staff member serves
as the coordinator of the program, involved in every aspect of preparation and implementation.
A critical decision facing the coordinator is what companies to visit. Ideally, the primary
stakeholders will reach out to companies and alumni by January.
In Spring 2015, the trip also coincided with a trip by a member of the Advancement team
at the Quinlan School of Business. She participated in 2 company visits, as well as helped plan
the Alumni receptions. The receptions were opportunities for the current students to network
with Quinlan alumni residing in the area.
Application Process
The Dean approves the budget between late July and early August. Once the funds have
been approved, the program director and program assistant begin sending out applications for
ramblers returning to serve as peer leaders. For the 2016 Ramble, the first round of applications
went out on August 28th and the deadline to apply was September 11th, 2015. This will be the
first year that peer leaders are incorporated into the planning and implementation process.
Notifications of acceptance will be sent to the students on October 9th. Applications for new
students to participate in the Ramble are distributed on October 19th. They are due on November
13th.
Returning Ramblers are expected to help plan the tourist outings and dining for the trip,
as well as write blogs about the experience and be model examples of how to engage with others
at company visits. For returning students, their application was more extensive (Appendix A).
There were four essay questions asking students why they were applying to experience the
Ramble again, what they will contribute to the trip as returning students, what they hope to gain
from returning on the Ramble, and what other commitments they may have in the fall and spring
semesters that will impact their ability to participate in planning meetings. They were also asked
to submit a resume along with their application.
In contrast, those applying to the trip as first-time Ramblers had a less extensive
application process (Appendix B). In addition to the submission of a resume, their application
posed two questions: Why are you applying to participate in the Quinlan Ramble to [California]
program? What do you hope to gain from this experience?
Program Purpose
The Quinlan Ramble was initiated by a former dean to allocate excess funds towards cocurricular programming for the Quinlan School of Business students. The dean wanted to create
an experience that would give students a chance to experience something that would give them a
competitive edge over undergraduate business students at other institutions. The Ramble does
this by giving undergraduate students the chance to experience career fields before graduation
and by giving them a chance to network with other professionals outside of Chicago. The
intention was to start a program that would have staying power (A. See, personal
communication, September 16, 2015). Thus, the Quinlan Ramble was born. Its main goals are
to give undergraduate students a chance to build community with other members of the Quinlan
network, explore career pathways, and provide experiences that complement business
coursework. In line with Loyolas Jesuit tradition, the Ramble encourages students to think
about how company values factor into future career choices. By participating on this trip,
students are getting a chance to meet and engage with business leaders in a variety of fields. As
our primary stakeholder explained, these opportunities allow them to learn about a pathway they
might not have considered before, build networking skills, and learn about various company
cultures (A. See, personal communication, September 16, 2015).
Logic Model
Logic models aid program planners and evaluators to eliminate the black box between
the programs objectives and its activities (Fitzpatrick, Sanders, & Worthen, 2011). The
implementation of a logic model for the use of this program is helpful when considering the
many parts of its implementation and impact. By creating one for this evaluation, we can better
explain the inputs, outputs, and impacts of the activities planned for this program (Appendix D).
Inputs
Inputs are the moving pieces needed to create the program in the first place. The inputs
for the Quinlan Ramble include important stakeholders, six or more months of planning, and a
$34,000 budget. Alumni and corporate site partners, marketing materials, students, and the
support of many offices are also integral pieces tied to the success of the Quinlan Ramble.
Program Stakeholders
The stakeholders of the program include many groups and individuals. As noted in the
physical representation of our logic model, the individuals that make the program possible
include staff partners, alumni, community partners, and student participants (Fitzpatrick,
Sanders, & Worthen, 2011).
The Quinlan School of Business Dean approves the funds for the program separately
from the budget given to Undergraduate Programs. Since they make the program possible, they
want to make sure that it is a good use of their discretionary funds. Loyola University of
Chicago and the Quinlan School of Business would benefit by learning about the effectiveness of
the program because the uniqueness of the program provides a competitive edge for these
primary stakeholders.
In addition, the people closely involved in planning the Quinlan Ramble are primary
stakeholders. The staff partners in particular put in a great deal of time and resources to make
sure the trip runs smoothly. As a result, they were able to arrange pre-trip meetings, travel and
company visits, and alumni receptions. In order to facilitate communication and networking,
they also organized a student group messaging account and a list of contact information for
professionals met on the trip. The student services coordinator responsible for all the details of
the program has a direct stake. He completely designs the experience and works to meet the
10
goals set out for the Ramble. The other professional staff member and graduate assistant are also
involved in the planning process and directing the trip. The faculty member dedicates time
outside of research and teaching to help make the program successful.
The Office of Advancement for Quinlan, as a secondary stakeholder, would like to see
the program continue and expand to strengthen existing Alumni networks. The Office of
Advancement uses the Ramble as an opportunity to reach out to Quinlan School of Business
Alumni. Alumni also would benefit from their Alma Mater producing a high achieving program
and graduates as a result of the Ramble. Similarly, Ramble participants can benefit from having
the experience if it is achieving its goal and receiving recognition from a positive evaluation.
Despite the efforts of the staff partners to make the Quinlan Ramble run smoothly, there are still
assumptions and external factors to take into consideration.
Assumptions
One critical assumption is that there will companies and alumni in the area willing to
speak to the students. The company visits are a large portion of the trip, and completely
dependent on the will of others. The students also need to research companies prior to visiting so
that they are well prepared to make the most of the experience. Similarly, the students
participating should be looking for a meaningful experience, instead of a spring break vacation
trip. When students apply to participate on the trip, the facilitators assume they are anticipating a
meaningful experience and are prepared for the fast-paced activity.
External Factors
Beyond these assumptions are a few external factors whose impacts have yet to be fully
realized. Those include the new dean and new participant pool. Moreover, the experience at the
Ramble destination can be impacted by what students are interested in, and potential natural
11
disasters or political crisis. A natural disaster, for example, could alter the flow of the trip or
cause us to unexpectedly cancel. In the spring of 2015, the destinations were Austin and Dallas,
Texas. While there, Dallas experienced its third largest snowfall in history. The unexpected
weather caused the staff to have to reschedule a company visit and incur an unplanned charge for
extra transportation.
Company interest can also make or break the trip. The program is only worthwhile if
there are businesses and companies willing to talk to the students. For some companies, these
visits are viewed as an asset and a chance to recruit new talent. For others, an undergraduate
group visit is viewed as an unnecessary distraction. Therefore, some companies do not want to
or are unable to take time out of their workdays.
Various financial factors could affect the implementation of this program. The Quinlan
School of Business has undergone major changes, receiving a great deal of attention for the
construction of an entirely new building for their students. A change in dean and budgets also
occurred between the Ramble of 2015 and the planning of the 2016 Ramble. These changes
resulted in a slightly reduced budget for the Ramble, subsequently increasing student fees for
participation. In the past, there has always been a graduate assistant to attend the trip and help
with implementation. This year it has still not been solidified if there will be room in the budget
for a graduate assistant to attend, but the student services coordinator has two graduate student
assistants that will still help to plan the trip.
Outcomes
Ultimately, all these factors matter because they influence the outcomes of the program.
Ideally, students should feel a sense of community with their fellow Ramblers that helps build a
support network lasting after the trip. One way that Quinlan has tried foster that sense of
12
community is by inviting former Rambler students to apply to help lead a future Quinlan Ramble
program. In addition to co-curricular involvement, the Quinlan Ramble staff aims to provide
students with experiences that can help them in their business coursework. The final goal centers
on helping students with their career choices. The Quinlan Ramble experience should help
students be more aware of what their options and criteria are for when they enter the workforce.
This outcome ties back to the competitive edge that the dean wanted to give undergraduate
students by better preparing them for the workforce.
Previous Evaluation
An online survey has been sent out each year to the students who have attended the trip
upon their return (Appendix C). The survey has been 21 questions long, and mostly includes
questions about satisfaction in regards to company visits and alumni receptions. There were
questions asking the students to list a take away from each visit and whether they found the visits
and presentations helpful. However, the survey never asked the students to explain why they
found them helpful or not. These surveys have not been mandatory. As a result, they were not
completed by every student who took part in the program.
General Evaluation Approach
After discussions with the student services coordinator, we decided on both outcomes and
process based evaluation methods. Since the Quinlan Ramble is relatively new, the stakeholders
want to include program outcome questions to make sure that the model is accomplishing its
goals. Specifically, the student services coordinator encouraged a strong focus on the career
knowledge outcome, regarding whether or not the experience influenced future career paths.
Some of the ways we will examine this is by determining whether students became aware of new
career possibilities and factors to consider when job hunting. In order to fully assess these
13
outcomes, we will also be asking process oriented questions related to specific activities on the
trip to see which were most impactful for the students.
This will be a formative evaluation; determining what aspects of the program are working
and which aspects need improvement (Fitzpatrick et al., 2011). However, future evaluations
could build on this work to create a summative evaluation to defend the Quinlan Ramble budget.
Especially with changes in leadership, it is critical to begin collecting data that down the line
could help support the assertion that the program is a worthwhile investment for the Quinlan
School of Business. The findings of our evaluation could also be useful to others designing
those types of programs, or considering the creation of an undergraduate business trip.
Our evaluation will be delivered via a web-based survey, and in-person focus group. We
decided on these methods to minimize the resources and time required by the evaluators, as well
as to reach the largest number of participants. Our goal is a census of all the students who have
participated thus far, and that participate in the 2016 Ramble. By including the former
participants of the Ramble, more data will be available regarding career decisions because many
students will have graduated or will be completing their final years as college students. There
have been a total of 30 past participants from the previous two years.
Quantitative Approach
Survey Population
The population studied will be all students who have participated in the Quinlan Ramble.
We are specifically interested in their perspective on the experience and their growth as a result
of the program. Since the program is only in its third year, the population of participants is still
relatively small. Therefore, we have decided to use a census sampling method which will target
the entire population of participants. This includes 30 participants from previous years, and 14
14
students participating this year. We decided on this approach because, For routine outcome
monitoring, many benefits accrue to covering all clients if the number is not too large (Wholey,
Hatry, & Newcomer, 2010, p. 274). A census sample will also help us to obtain as much data as
possible because we anticipate a response rate of approximately 60%. Given our small
population, this would be around 26 students.
Survey and Evaluation Design
The Quinlan Ramble survey is a web based pre-experimental survey designed by the
researchers (Creswell, 2014). The survey will be a cross-sectional posttest only design, therefore
we will not have a comparison group (Fitzpatrick et al., 2011, p.392). Since we are using
questions specific to the experience within the Quinlan Ramble, it would not be feasible to use a
comparison group. Some of the participants are no longer students at Loyola Chicago, but will
be asked to complete the survey. Therefore, a web-based survey emailed to participants by the
Student Services team (the professional coordinator and the graduate assistants) at Quinlan is the
best chance to reach the entire population (Appendix E). The participants will be given two
weeks to complete the survey asynchronously, on their own time.
The participants will receive an initial email sending the survey link to them and
explaining what the survey is regarding at the beginning of September 2016 (Appendix F). The
primary stakeholder Alex See will be listed as the contact person. In order to ensure that we
have a high rate of return, there will be an incentive for those who complete the survey and
complete it on time. The participants will be entered in a drawing for gift cards to popular online
shopping services with a value of $25 each. Participants will also receive a brief reminder email
halfway through the week to remind them of the surveys deadline.
15
Survey Instrument
Our survey for the Quinlan Ramble will be available on the survey platform Qualtrics. In
the survey, we ask questions that focus on three learning goals. The first learning goal, or
outcome, is that the Quinlan Ramble helped students build community within the Quinlan
population. The second outcome is that the Ramble influences the way that the participants think
or thought about their future careers. The final learning outcome we use the survey to measure is
that experiences on the Ramble positively contribute or contributed to the participants classroom
experiences.
The survey also includes an informed consent document at the start of the survey
(Appendix G). The participants can digitally agree to take the survey after reviewing the
informed consent. The document is included in the survey as a way of ensuring that participants
know that survey is voluntary, and that they are guaranteed confidentiality.
The survey should take participants no longer than five to ten minutes; the survey itself,
including the demographic questions, is seventeen questions. The demographic questions are
either multiple choice or open-ended. The questions in the survey pertaining to the learning
goals are all Likert scale responses (Appendix H).
Pilot Testing
We will pilot test the survey by sending it electronically to undergraduate ambassadors of
the Quinlan School of Business. These ambassadors work with promoting different programs
within the undergraduate programs office. Only two ambassadors have participated in a Ramble,
so out of the seventeen ambassadors, fifteen will be asked to pilot test the survey. By sending it
to them, we can be sure that the instructions are clear and easy to understand from an
undergraduate perspective. As students involved in Quinlan, they are a representative sample of
16
the students taking part in the Quinlan Ramble. We will use their feedback to help us make
revisions to the survey where necessary. In addition, we will ask our pilot test participants to
note how long the survey took to complete. This information will help us provide an accurate
estimation in emails of how long the survey will take participants.
Statistical Analysis & Reports
For our statistical analysis, we will utilize SPSS to identify trends in the data. Our focus
will be on descriptive statistics to show the findings from our census data. We will run
univariate tests for frequency and central tendencies of the data, including mean, median, and
mode. These tests will highlight on average how students are achieving goals, as well as show
the variability among participant responses. We create tables with this information that make the
findings easily accessible for our stakeholders. We want to clearly demonstrate the strengths and
challenges regarding their desired outcomes within our statistical analyses. In addition, we will
include histograms to highlight variability for items that showed dramatic differences in
participant responses.
Although we will also perform tests for measures of central tendency on our demographic
information, the primary purpose of including that information is to examine differences across
certain groups. In order to carry out this goal, we will rely on ANOVA tests. Using ANOVA
tests, we will compare how folks of different ethnicities and genders differed in their response to
the Likert items included in the survey. Our stakeholders prioritized the inclusion of such tests
to ensure that they were aware of any potential weaknesses in the program specific to the
experience of students from marginalized groups. When presenting this information to our
stakeholders, we will highlight significant findings from these analyses, particularly what areas
require improvements.
17
Qualitative Approach
Focus Group
While the quantitative survey allows us to gather data around what the strengths and
weaknesses are, the group discussion will help us think about what to strengthen or improve by
providing context. We will utilize focus groups instead of interviews as a way to get a more indepth understanding of various aspects of the program. Similar to interviews, focus groups allow
for face to face interactions, but have the added advantage of allowing participants to process
with other participants about their experience (Fitzpatrick et al., 2011). Discussing Ramble
experiences with others who have participated in the same program will allow participants to
compare experiences and impacts of the program.
We chose a focus group because it will allow folks who participated in different years to
highlight commonalities and differences between the various trips that made them more or less
successful. By focusing specifically on students who had a less positive experience, we can
begin to recognize what has prevented students from reaching the stated program outcomes.
Moreover, the focus group should shed light on how the trip coordinators can work to improve
the experience for future Ramble participants. Therefore we have included predominantly
descriptive and action questions in our instrument (Appendix A).
Focus Group Participants
The sample will be made up of disconfirming cases. For the purpose of this focus group,
we will be reaching out to students whose surveys indicated that the Ramble had little to no
significance for them. On the initial survey email, we mentioned the possibility of a follow up
focus group invitation. We will determine these students based on who rated experiences as
neutral more often than agree or strongly agree, and those who selected disagree or
18
strongly disagree for any of the questions. We will contact these participants via email
explaining that we appreciated their honest feedback and would like their help determining how
to improve the Quinlan Ramble (Appendix I). We will only hold one focus group, as opposed to
multiple for participants to choose from, since we anticipate a small group of participants for the
focus group given the small population.
Protocol Description
The facilitators will hold a semi-structured focus group, where they will have a few
questions to start the conversation and keep the conversation active. However, they will let the
conversation progress naturally whenever possible. Our protocol attempts to leave room for this
emergent design, as the questions may shift as the facilitators conduct the group (Creswell,
2009). This focus group will cover questions centered on the desired outcomes of the program,
and will address the experiences and concerns of those participants who indicated that the
experience was not beneficial for them. For example, our protocol includes probes like What if
anything do you think would have made it easier to build community with the other students and
alumni? (Appendix J). The focus group also included a consent and demographic form to
present the potential risks and benefits of participating in the focus group, as well as to gather
identity information on those who participate (Appendix K).
We will pilot test the focus group questions with the same students we pilot tested the
survey with: the Quinlan ambassadors. Although the ambassadors have not participated in the
Quinlan Ramble, we will ask them to answer the questions in a mock focus group session. They
will be able to relay to us if the questions are understandable and answerable. Additionally, this
mock focus group session will serve as a practice session for the facilitators.
19
20
21
made, the feedback could also be included in the report in order to highlight consensus or
disagreement among participants regarding the point. Subsequently, this version will also
include direct quotes to serve as evidence of the conclusions drawn.
Although the results will triangulate using the member checking feedback, the survey
data will not be heavily relied on. Since we are relying on a disconfirming approach, the
experience of these students will most likely not align with many of the other results of the data.
However, it could help us to explain some of the variance in the data within our final report. A
table of descriptive will also be included to note how often students discussed certain codes and
themes throughout the focus group. The table will be a convenient way for stakeholders to note
how prevalent the themes were in the focus group session. Finally, the report will also include a
statement recognizing the potential biases of the stakeholders involved in the collection and
coding of data.
Validity
Since the coding will be divided up among the graduate assistants, before beginning, a
section of the transcript will be given to each graduate assistant. It will be the same section, and
they will each be asked to code it individually. Afterwards, they will compare their findings to
determine inter-rater reliability. Through this process, the graduate assistants can explain
rationales and address discrepancies in their interpretations of the codes. The goal is make sure
that there is consistency in the way the data is coded (Wholey et al., 2010).
Since generalized conclusions will be drawn from the data, we feel that is essential to
give participants a chance to challenge the interpretations being made. For that reason, we will
employ member checking. Member checking is a way to keep researchers accountable by
ensuring that the participants agree with their interpretations of the data (Wholey, Hatry, &
22
Newcomer, 2010). We will create a brief report, without quotes or identifiers, to send via email
to the participants that details what we believe the major themes were from the data. The
participants will be encouraged to contact us with their opinions, especially if they feel that the
data has been misinterpreted.
Limitations
There are a number of limitations associated with both our quantitative and qualitative
approaches. Our approach could suffer from attrition, where those we identify from the survey
decide not to participate in the focus group (Schuh & Associates, 2009). Since our initial
population is already small, losing participants could be detrimental to the data collection.
Another limitation could be bias, in that we are suggesting that the program planners, who are
primary stakeholders in the Ramble, facilitate the evaluation. Due to their stake in the programs
success, it would be easy for them to interpret responses during data coding in the programs
favor. In addition, it is possible that participants will not be comfortable sharing critiques of the
program with those who put so much time and effort into it. While we have employed methods
to reduce this bias, it is necessary to name and recognize the potential effects. Another limitation
focuses specifically on our qualitative focus group approach. We are using disconfirming cases
from the survey to identify areas of improvement. While we know this type of approach is
important to the goal of the evaluation, we realize that by using disconfirming cases, we are
limiting the feedback available on the ways that the program may be doing really well.
Next Steps
The first round of this evaluation will encompass participants from three Quinlan Ramble
trips, which is a total population of 44 people. After this initial run of the evaluation, we suggest
that the survey and focus group continue each year and be modified to fit each trip. We also
23
suggest that the facilitators of the evaluation contact us regarding any questions or changes that
may come arise. We are happy to serve as a resource and work together with the staff to
understand the results of the evaluation. Ultimately, the finding of the evaluation should
improve trip planning in order to better meet the intended outcomes. In the longer term, using
the data from each of the evaluations will be helpful in defending the Rambles budget in the
face of any potential funding cuts.
24
Applicant Information
Applicants Full Name (as listed on a state ID or a passport):
__________________________________________________________________________________
Loyola ID#: _________________________________
Gender: _____________________________________________________________
Email Address:
_________________________________________________________________________________
Local Address:
__________________________________________________________________________________
Permanent Address:
_______________________________________________________________________________
Telephone number: (_____) __________-________________
Cell phone number:
(______) ___________-__________________
Major/s:
__________________________________________________________________________________
Year in school, expected graduation (e.g., Junior, May 2017):
_____________________________________________
Are you willing to drive a van to transport students during the trip? (must be 21 and have a valid
U.S. DL) ___ yes ___ no
_________________________________________________________________________________________
Essay Question
Please attach an additional statement addressing the following questions in your application essay:
25
1. Why are you applying to participate in the Quinlan Ramble Program again?
2. Based on your previous Spring Ramble experience, how do you think that you will contribute to the
Spring Break trip to California? How will you utilize your leadership skills during this Ramble?
3. What do you hope to gain from this experience?
4. What other commitments will you have in the fall and spring semesters that will impact your ability to
participate in planning meetings and individual work time for the trip?
__________________________________________________________________________________________
Resume
Please submit a copy of your resume upon turning in this application.
__________________________________________________________________________________________
Trip Details (tentative)
Dates: Saturday, March 5 Saturday, March 12
Cost: $450, includes the following:
o Round trip airfare (Chicago Bay Area, CA)
*potential cities include: San Jose, Oakland, and San Francisco
o 7 nights at hotel (2 students per room)
o 3 lunches
o 3 dinners
o Corporate site visits including transportation to and from each company
For more information, please talk to Alex See, Student Services Coordinator, asee@luc.edu.
Additional Information
Important Application Requirements:
The following is a checklist of documents that need to be included in your application packet.
o Application Form
o OSCCR Student Educational Records Disclosure Form (Sign in Schreiber Center Suite 320,
witnessed by staff member)
Important Application Process Information:
After application packets have been reviewed, selected students will be notified and informed of the next steps.
Students will be required to submit the following
26
27
Are you interested in taking and submitting photos for the Quinlan Ramble to California blog and Flickr site?
Circle one: YES/ NO
Roommate:
All participants will automatically be placed in a double room with the same gender.
_________________________________________________________________________
Applicants signature
Date
28
Applicant Information
Applicants Full Name (as listed on a state ID or a passport):
__________________________________________________________________________________
Loyola ID#: _________________________________
Gender: _____________________________________________________________
Email Address:
__________________________________________________________________________________
Local Address:
__________________________________________________________________________________
Permanent Address:
_______________________________________________________________________________
Telephone number: (_____) __________-________________
Cell phone number:
(______) ___________-__________________
Major/s:
__________________________________________________________________________________
Year in school, expected graduation (e.g., Junior, May 2017):
_____________________________________________
Are you willing to drive a van to transport students during the trip? (must be 21 and have a valid
U.S. DL) ___ yes ___ no
__________________________________________________________________________________________
Short Essay Question
5.
Why are you applying to participate in the Quinlan Ramble to California Program?
29
30
31
Signature
The information that I have provided is true and complete. I also understand that this trip is
subject to change or cancelations at the discretion of the Quinlan School of Business. If the trip
is canceled by the Quinlan School of Business, any trip fees paid to the Quinlan School of
Business will be refunded.
_________________________________________________________________________
Applicants signature
Date
32
Appendix C: Ramble Survey
Ramble To Texas
Q1 Thank you for being a part of the Quinlan Ramble to Texas!We value your feedback and
look forward to incorporating your suggestions into future trips. Please answer questions as
precisely as possible. All responses will be kept confidential and private. You do not need to
include your name but may choose to do so.If you have any feedback that you would like to
share in person or through a different means of communication contact: Alex See, Quinlan
School of Business Student Services (asee@luc.edu / 312.915.6572)
Q2 What type of student are you?
m Freshman (1)
m Sophomore (2)
m Junior (3)
m Senior (4)
Q3 COMPANY VISITS
How satisfied were you with our company visits?
Highly
Dissatisfied
(1)
Dissatisfied
(2)
Neutral (3)
Satisfied
(4)
Highly
Satisfied
(5)
ATI [UT
Austin
Incubator]
(Monday) (1)
AMD
(Monday) (2)
Dimensional
Funds
(Tuesday) (3)
Dell Social
Media
Listening and
Command
Center
(Tuesday) (4)
33
Andy Smith
(Wednesday)
(10)
Capital
Factory
(Tuesday) (5)
The Richard's
Group
(Thursday) (6)
Gatorade
(Friday) (7)
Hunt
Consolidated
(Friday) (8)
American
Airlines
(Friday) (9)
Q4 Please list a takeaway from each of the following tours and presentations.
ATI (Monday) (1)
AMD (Monday) (2)
Dimensional Funds (Tuesday) (3)
Dell Social Media Listening Command Center (Tuesday) (4)
Capital Factory (Tuesday) (5)
Andy Smith (Wednesday) (10)
The Richard's Group (Thursday) (6)
Gatorade (Friday) (7)
Hunt Consolidated (Friday) (8)
American Airlines (Friday) (9)
Q5 Overall, did you find the company visits educational?
m Yes (1)
m No (2)
m Yes, except for: (3) ____________________
Q6 Overall, did you find the presentations and tours helpful in developing your understanding of
business?
m Yes (1)
34
m No (2)
m Yes, except for: (3) ____________________
Q7 Overall, did you find the presentations and tours helpful in developing your understanding of
business strategies?
m Yes (1)
m No (2)
m Yes, except for: (3) ____________________
Q8 Overall, did you find the presentations and tours helpful in developing your understanding of
hiring practices?
m Yes (1)
m No (2)
m Yes, except for: (3) ____________________
Q9 Please rate how helpful you found the tours and presentations in connecting your major to
other business majors.
Very
Helpful (1)
Helpful (2)
Neutral (3)
Not Helpful
(4)
Not Very
Helpful (5)
Tours (1)
Presentations
(2)
2 Agree (2)
3 Neutral
(3)
4Disagree
(10)
5 Strongly
Disagree (5)
A greater
sense of
belonging (1)
I expanded
my
knowledge
(2)
35
I built my
skills (3)
I found my
calling (4)
I invested in
the creation
of my future
(5)
Q12 Please answer the following statements with the first word that comes to mind.
I am better able to (1)
I am more willing to (2)
I have enhanced my understanding of (3)
I have enhanced my ability to (4)
My involvement in the Ramble to Texas program has positively affected my (5)
Q13 ALUMNI RECEPTIONS
How satisfied were you with our Alumni Receptions?
Highly
Dissatisfied
(1)
Dissatisfied
(2)
Neutral (3)
Satisfied
(4)
Highly
Satisfied
(5)
Freddie's
Place
(Austin) (1)
Maggiano's
(Dallas) (2)
Q14 What did you like or not like about both of our receptions?
Q15 What would you have liked to learn more about?
Q16 Do you feel anything was missing?
Q17 Did you feel the group had an adequate amount of free time?
Q18 What was your favorite memory?
36
37
Appendix D: Logic Model
Inputs
Outputs
Activities
What we do
Participation
Who is reached
What we invest
1 full-time staff
member
Visit different
business
Student
participants
1 full-time
faculty/staff
member from
another office
Interact with
company
leaders and
employees
Peer leaders
2 graduate
assistants
Research
companies and
staff members
prior to visits
Graduate
Assistants
2 peer leaders
(starting in 2016)
14-16 student
participants
$34,000 budget
Support of
Quinlan
Undergraduate
Programs Office
Tourist
attractions and
activities
Meet with
alumni
Pre-trip
preparation
meeting
Create a group
messaging
account
Marketing
materials
Post-trip
survey
Faculty/staff
partner
Company
partners
Alumni
Community
Partners
Undergraduate
Programs staff
Outcomes
Short-term
Short-term
changes we
expect
Build
community
among
participants
Keep in contact
with others
after trip
Increased
consideration
for future plans
Medium-term
Medium-term
changes we expect
Build community
among participants
Long-term
Long-term
changes we
expect
Build
community
among Quinlan
network
Participate in other
career
development/Quinlan Alter or
social events
solidify career
plans
Retain participants to
apply to serve as trip Influence job
leaders (starting in
search
2016)
Consider
company
culture and
values when
applying for
positions
Apply
experiences
from trip to
academic
setting and
office setting
Assumptions: Alumni/Companies in the area are willing and able to speak to students, students will do necessary
research before visits, students looking for meaningful experience | External Factors: New Dean, new participant
pool, destination meteorological and political climate, company interest, financial factors
38
Appendix E: Quinlan Ramble Survey
Quinlan Ramble Survey
Thank you for participating the Quinlan Spring Ramble. The purpose of this survey is to help us
understand the impact of the trip on its participants. We want to make sure that we are planning
and implementing the trip in ways that best fit the students who attend. We will be asking
questions about building community on the Ramble, the Rambles influence or non-influence on
career exploration, and the connection of Ramble experiences to the classroom or office setting.
Instructions: Please respond carefully to each question in the survey. You have the option to
abstain from answering any question. This survey should take about 15 minutes of your time.
The following questions provide us with demographic information about our respondents.
Please indicate the answers that best describe you.
39
Q5 Are you currently enrolled/were you enrolled in a Dual Degree program at Quinlan?
m Yes, BBA/MBA program (1)
m Yes, BBA/MSA program (2)
m I am not currently/was not enrolled in a Dual Degree program at Quinlan. (3)
Please indicate to what extent you agree or disagree with the following statements
regarding building community on the Quinlan Ramble.
Q6 The Ramble was helpful in connecting with other Quinlan students and alumni.
m Strongly agree (1)
m Agree (2)
m Neither Agree nor Disagree (3)
m Disagree (4)
m Strongly Disagree (5)
Q7 I was able to build relationships with other Loyola students on the trip.
m Strongly agree (1)
m Agree (2)
40
Q8 I was able to build relationships with leaders on the trip (this can include Loyola staff
and faculty).
m Strongly agree (1)
m Agree (2)
m Neither Agree nor Disagree (3)
m Disagree (4)
m Strongly Disagree (5)
Q9 Please indicate to what extent you agree or disagree with the following statements
regarding career exploration and the Quinlan Ramble activities.
41
Q12 I am more certain of the career I want to enter after the Quinlan Ramble than I was
before it.
m Strongly agree (1)
m Agree (2)
m Neither Agree nor Disagree (3)
m Disagree (4)
m Strongly Disagree (5)
Q14 The company visits helped me to consider company culture in my job search.
m Strongly agree (1)
42
43
m Agree (2)
m Neither Agree nor Disagree (3)
m Disagree (4)
m Strongly Disagree (5)
Q Please indicate to what extent you agree or disagree with the following statements
regarding connecting the Quinlan Ramble activities to the classroom or work environment.
Q15 I can apply/did apply the knowledge I gained in the Quinlan Ramble to what I am
learning/was learning in my classes.
m Strongly agree (1)
m Agree (2)
m Neither Agree nor Disagree (3)
m Disagree (4)
m Strongly Disagree (5)
Q18 Thank you for participating in the Quinlan Ramble Survey. Your insights are greatly
appreciated. Is there any other information you would like to share with us regarding your
experience?
44
45
Appendix F: Initial Survey Email
Hello Ramblers,
You are being contacted because you participated in the Quinlan School of Business Quinlan
Ramble program either during the Spring Break week of 2014, 2015, or 2016. To better
understand the ongoing impact and learning outcomes of the Ramble, we invite you to
participate in our new survey to share what you learned or gained from the experience.
Please note, some participants will be contacted for a voluntary follow up focus group
discussion. If you have any questions regarding this project, please contact Alex See, Student
Services Coordinator, in the Quinlan School of Business.
Sincerely,
Quinlan Ramble Team
46
Appendix G: Informed Consent
47
There will be no cost to you if you participate in this survey. There is an incentive for
those who do complete the survey.
Confidentiality
By participating in the study, you understand and agree that the Quinlan School of
Business of Loyola University Chicago may be required to disclose your consent form, data and
other personally identifiable information as required by law, regulation, subpoena or court order.
Otherwise, your confidentiality will be maintained in the following manner: Your data and
consent form will be kept separate. All data beyond the consent will be kept anonymous. By
participating, you understand and agree that the data and information gathered during this survey
may be used by the Quinlan School of Business and published and/or disclosed to others outside
of the university. However, your name, contact information and other personal identifiers in
your consent form will not be mentioned in any such publication or dissemination of the research
data and/or results.
The research team will take the following steps to protect participants identities during
this study: (1) Each participant will be assigned a number; (2) The researchers will record any
data collected during the study by number, not by name; (3) Any original recordings or data files
will be stored on a secured computer accessed only by authorized researchers.
Rights
Your participation is voluntary. You are free to stop your participation at any point.
Refusal to participate or withdrawal of your consent or discontinued participation in the study
will not result in any penalty or loss of benefits or rights to which you might otherwise be
entitled. The Principal Investigator may at their discretion remove you from the study for any of
48
a number of reasons. In such an event, you will not suffer any penalty or loss of benefits or
rights to which you might otherwise be entitled.
Voluntary Consent
By clicking Yes, you agree that the above information has been explained to you and
all your current questions have been answered. You understand that you may ask questions
about any aspect of this research study during the course of the study and in the future. By
clicking Yes below, you agree to participate in this research study.
I have read and understood the above consent form, am currently 18 years or older, and desire of
my own free will to participate in this study.
m Yes (1)
m No (2)
49
Appendix H: Survey Construct Map
Survey Matrix
Goal #
Question #
Question
Type of
measure
Radio button
Likert scale.
Strongly
disagree (5) to
strongly agree
(1).
5
4
3
2
1
Radio button
Likert scale.
Strongly
disagree (5) to
strongly agree
(1).
5
4
3
2
1
Radio button
Likert scale.
Strongly
disagree (5) to
strongly agree
(1).
5
4
3
2
1
Radio button
Likert scale.
Strongly
disagree (5) to
strongly agree
(1).
5
4
3
2
1
10
Radio button
Likert scale.
Strongly
disagree (5) to
strongly agree
(1).
5
4
3
2
1
11
Radio button
Likert scale.
Strongly
disagree (5) to
5
4
3
2
50
strongly agree
(1).
12
Radio button
Likert scale.
Strongly
disagree (5) to
strongly agree
(1).
5
4
3
2
1
13
Radio button
Likert scale.
Strongly
disagree (5) to
strongly agree
(1).
5
4
3
2
1
14
Radio button
Likert scale.
Strongly
disagree (5) to
strongly agree
(1).
5
4
3
2
1
Radio button
Likert scale.
Strongly
disagree (5) to
strongly agree
(1).
5
4
3
2
1
Radio button
Likert scale.
Strongly
disagree (5) to
strongly agree
(1).
5
4
3
2
1
#3 Quinlan Ramble
participants can apply their
15
experiences to the
classroom or work
environment.
16
51
Hello,
Thank you for participating in our survey. We valued the feedback you provided, and
would like to invite you to participate in a focus group. This focus group will be held by (insert
names here). We are very interested in hearing your opinions and thoughts on your Quinlan
Ramble experience. Below is a review of the information needed for the focus group.
When:
*date to be decided on by Ramble staff members; we suggest the fall term, once the
Ramblers from the previous spring have had the summer to process their experience.
Where:
We will be conducting the focus group in the Schreiber Center, WTC. Light
refreshments will be served.
Purpose:
The purpose of this study is for facilitators to learn more about the perspectives and
experiences students have had regarding their participation the Quinlan Ramble program. By
hearing personal experiences and thoughts from former Ramblers, researchers hope to positive
changes in the program based on the facilitators observations of the focus group.
In order to fulfill the consent process, please read the information below and follow the link to
sign consent for the focus group:
https://loyolaquinlan.qualtrics.com/SE/?SID=SV_5jciwrOxgrHMBqR
52
If you have any questions after you review the above information, please feel free to contact one
of the focus group team members.
Thank You,
Focus Group Team
53
Appendix J: Focus Group Protocol
Focus Group Purpose: To understand the perspectives of past Ramble participants who
indicated that the experience was not influential for them
Focus Group Topics: Community Building
Career Exploration
Applicability
Improvement
Script
Introduction
Hello and welcome! Please allow us to introduce ourselves. We are (insert names here).
We will be the moderators for the focus group today. If you havent already, please read and sign
the consent form for the study. We have asked you to participate because we feel that you are
best suited to answer the questions for our topic. This focus group will last for one hour and
thirty minutes. The purpose of this study is to learn from the participants perspectives on their
Quinlan Ramble experience. The information given during the focus group by participants will
be handled in a sensitive manner. No participant in the focus group will be named in the data
collected throughout. We will audio record the focus group so that we are able to fully capture
the voices and opinions of the participants in the focus group. As a participant of the focus
group, you can decide to abstain from answering a question. We ask that you please be respectful
of other members of the focus group and their opinions and views. Please remember that any
54
information or stories shared during our focus group is not to be shared outside of the group after
the focus group is over.
We ask that you please place all electronic devices that will not be used for the study on silent
for the duration of the focus group session.
Teamwork
We will work as a team to complete our focus study group. Our teamwork will be
achieved by appointing a role for each member. The role that each member holds will be utilized
to have a seamless focus group session. As a team, we also recognize our availabilities so we
will work with each other to complete deadlines and keep open lines of communication. Two of
the facilitators will serve as moderators, asking questions and probing for more information. The
third facilitator will be controlling the audio recorder and taking notes.
Questions
Question 1: Introductory Conversations (spend 10 minutes on this section)
a.
b.
c.
55
Probe: To what extent if any were you able to build relationships with other students on the
c.
What was your experience like at the alumni receptions that were held?
d.
What do you think would have made it easier to build community with the other students
and alumni?
Probe: What types of experiences, if any, did you have that involved career exploration?
b.
Probe: What would have better prepared you for our company visits?
c.
Which visits did you find more or less helpful than others in terms of career exploration?
d.
e.
What about those visits made them stand out positively or negatively for you?
Were there other aspects of the trip that you felt would have been more influential in terms
In what ways, if at all, have you been able to apply any of the experiences from the Ramble
56
i. If not, why do you think you have not been able to apply what you learned to other
contexts?
b.
Do you reference your Ramble experience in conversations with others in your classes or
What would have made the trip more applicable to your everyday life and experiences?
Probe: How did the Ramble compare to your expectations of the trip?
What could the trip facilitators have done to make the trip a more positive experience for
you?
c.
d.
Probe: If you could change anything about the Ramble, what would it be? Why?
What weaknesses did you see in the planning and implementation of the trip? How would
you address those weaknesses if you were planning a trip like the Ramble?
e.
Is there anything else any of you would like to add to the discussion? Is there anything
youd like to share that you havent gotten a chance to share yet?
Thank you so much for your time today. We really appreciate and value your honest feedback
about your Quinlan Ramble experience.
After the focus group: Take 15 minutes to write a memo; any initial reactions or observations
from the focus group. Take a little time to process immediately.
57
2014
2015
2016
Other:
Black
Latin@/Latinx
Asian/Pacific Islander
Multiracial
White
Other:
Are you currently enrolled/were you enrolled in a Dual Degree program at Quinlan?
58
Yes
No
Please sign.
Name:
Email:
Phone Number:
59
60
Appendix L
Coding Rubric: A Priori Codes
Coding
Meaning
Definition
AP
Applicability
COM
Community
NEW
New Ideas
IMP
Improvements
CD
Career Development
RMV
Remove
61
Appendix M: Evaluation Timeline
Tasks
Aug
Meet with stakeholders
to discuss evaluation
plan
x
Pilot test survey
(quantitative)
x
Revise survey based on
feedback from pilot
testers
Prepare survey through
Qualtrics for distribution
Distribute survey to
student
Send email reminder to
complete survey
Compile and analyze
quantitative results
(after survey closes)
Contact possible focus
group participants to set
up focus group
Pilot test focus group
questions
Schedule focus group
Revise focus group
questions based on
feedback
Distribute consent and
demographics form
Conduct focus group
Transcription and
coding
Analyze qualitative data
from focus group
Member checking with
initial findings
Presentation of final
report and
recommendations
Sept
Oct
Nov
Dec
Jan
Feb
March
x
x
x
x
x
x
x
x
x
x
x
x
x
62
Item
Gift Cards
Food for Focus Group
Schreiber Room
Transcription
Facilitators for focus group
Data Storage
Emails
Item Type
Incentive
Incentive
Materials
Personnel
Personnel
Materials
Materials
Cost
$50
$150
$0
$0
$0
$0
$0
$200
Rationale: Most of our money is being used on incentives to increase participation. Since we
are working with such a small population, we want to ensure that we are attracting as many
folks as possible. The other costs will mainly be in terms of human resources. We will be
relying on the stakeholders for the collection, transcription, and analysis of various data.
63
Presentation
64
65
66
References
Creswell, J.W. (2014). Research design: Qualitative, quantitative, and mixed method approaches
(4th Edition). Thousand Oaks, CA: Sage Publications.
Engberg, M.E. (2015). Sampling approaches (class handout). Higher Education, Loyola
University Chicago.
Fitzpatrick, J., Sanders, J., & Worthen, B. (2011). Program evaluation: Alternative approaches
and practical guidelines (4th Ed.) New York: Longman.
Jones, S., Rowan-Kenyon, H., Ireland, S., Niehaus, E., & Skendall, K. (2012). The meaning
students make as participants in short term immersion programs. Journal of College
Student Development, 53(2), 201-220. doi: 10.1353/csd2012.0026
Mission and Vision: Loyola University Chicago. (n.d.). Retrieved September 27, 2015, from
http://www.luc.edu/mission/mission_vision.shtml
Niehaus, E., & Inkelas, K. (2015). Exploring the role of alternative break programs in students
career development. Journal of Student Affairs Research and Practice, 52(2), 134-148.
doi: 10.1080/19496591.2015.1020247
Road Scholars Program: Kogod School of Business. (n.d.). Retrieved September 29, 2015, from
http://www.american.edu/kogod/klab/road_scholars.cfm
Schuh, J.H. & Associates (2009). Assessment methods for student affairs. San Francisco, CA:
Jossey-Bas Schuh, J.H. & Associates (2009). Assessment methods for student affairs. San
Francisco, CA: Jossey-Bass
[Untitled photograph on The Schreiber Center]. Retrieved November 30, 2015 from
http://www.luc.edu/quinlan/stories/archive/
Wholey, J.S., Hatry, H.P., & Newcomer, K.E. (Eds.) (2010). Handbook of practical program
evaluation (Third Edition). Jossey-Bass: San Francisco.
67