Employer Program Assessments: Validating Your Mission

This session will consider a new creative approach for gathering employer feedback about business and management programs. It capitalizes on technology and, in a recent pilot study, generated outstanding results. This new approach solves many of the problems with historic employer mail surveys by delivering the survey to the person within the company in the best position to evaluate the quality of your program. Less costly and more effective than a mail methodology, this approach shows real promise.

Session Objectives
Employer Program Assessments: Validating mission and improving the quality of academic programs
Why employer studies have historically been difficult and compromised.

Exploring a new methodology for conquering past shortcomings and capturing valuable stakeholder feedback.

Fulfilling Your Mission

Management education programs

commonly include within their mission providing an education experience that prepares candidate to succeed in a business career.

Fulfilling Your Mission

Which stakeholders are in the best position to ascertain the degree to which a program has fulfilled its mission of preparing candidates to succeed?

Underlying Premise
In principle the degree to which the academic program has prepared candidates for a successful business career is best determined by: Current Students Faculty Alumni Employers

Underlying Premise
Focus on the program, not individuals: Protect alumni and employer anonymity Focus on performance as it relates to preparedness

Underlying Premise
The Alumni: Having been in the work place they are now prepared to evaluate the degree to which their education prepared to succeed

Underlying Premise
The Employer: Who is in the best position to evaluate preparedness and performance?
Recruiter HR representative Supervisor

Underlying Premise
The Supervisor: Know strengths Knows weaknesses Know performance Can compare with others educated at different institutions

Difficulties of Traditional Employer Assessment Methodologies

Who to Survey: Identifying employer representatives to survey Contact Info: Gathering contact information for employer representatives (physical or email address) Cost: Contact information, incentives, distribution, follow-up reminders, analysis, distribution of results Confidentiality: Protecting anonymity of participants Response Rates: Reaching an acceptable minimum Assessment Durations Time: Time to implement

Web versus Mail

Web Mail

Who to Survey:
Alumni: Employer: Contact Info: Cost: Challenging Easy email lower Difficult mail higher

Confidentiality:
Response Rates: Assessment Durations Time:

high
higher short

low
lower long

Contact:

Difficult to identify who should be surveyed Supervisors change quickly over time

Difficult to track contact information

Mailing Address: Graduates move from firm to firm Email Address: Change less than mailing addresses

Stops most employer assessments in their tracks

Contact:
Cost:

Paper Administration:
Printing & Materials Costs: Surveys, letters, envelopes Postage: Send initial request, follow-up request (possible return postage) Handling: Stuffing initial mailing, opening responses, (potential for loss of damage) Processing: Cost to scan surveys

Contact:
Cost:

Web Administration:
Dependant on acquiring valid alumni email

Web Survey Software to administer surveys

Web software to capture employer representation and launch linked survey Web Analysis Reporting

Web Report Distribution

Contact:
Cost:
Confidenti ality:

Mail Administration

Follow-Up Reminders: Surveys must be tagged with identifier to determine upon receipt who has responded
Surveys received must identified and be logged, compromising confidentiality Survey returned to academic program staff who can identify participants

Contact:
Cost:
Confidenti ality:

Web Administration

Follow-Up Reminders: While identifies exist, reminders can be sent without access to responses (if third party involved)
While employer representative evaluates individual, responses used only to evaluate program, not individuals Alumni evaluation can remain confidential (if third party involved)

Contact Info:
Confidenti ality: Response Rate:

Mail Administration: Mail surveys traditionally have a very low response rate driven by: Bad addresses
Potential to lose survey

Effort to complete and return

Limited reminder capabilities

Contact Info:
Confidenti ality: Response Rate:

Web Administration:
Easy to complete Email addresses change less often Survey always available Easy to send multiple reminders

Contact Info:
Confidenti ality: Response Rate: Results TurnAround Time:

Mail Administration:
Long time from initial mailing to review of results Web Administration

Fast turnaround on completion and viewing of results

Combining Assessments
New Methodology for Combining Alumni and Employer Assessments:
Send academic program assessment to alumni that captures their perceptions of their preparation and their supervisors name and email address Assure alumni of their anonymity, and the purpose of sending their supervisor a survey is to capture their perceptions of the academic program (vale of third party) Once the alumni completes the survey an employer survey is automatically sent to their supervisor

Combining Assessments
Benefits:
More Information: Gather feedback from two stakeholder populations simultaneously Accuracy Identifying Employer: Solve the problem of identifying the employer representative by having alumni provide the name of their supervisor

Combining Assessments
Benefits:
Higher Response Rates: (our experience is response rates are higher for email surveys than mail surveys) Results Timing: Immediate real-time results reporting Stakeholder Comparisons: Dual comparative analysis of both surveys Results Distribution: Fast, easy distribution of results

Principles
Personal

Assure Confidentiality
Secure Link

Link Surveys
Immediacy Accuracy

Linking the Assessments

State of Washington
54%: Alumni response rate

92%: Alumni providing employer info

49% Employer response rate

Examples
Alumni Invitation Example

Examples
Alumni Survey Example

Examples
Employer Invitation Example

Examples
Employer Survey Example

Pros & Cons: Pros

Identification of employer representative is extremely accurate
Simultaneous capturing of alumni and employer feedback Respectable response rates

Short turn-around time

Cost effective

Pros & Cons: Cons

Need accurate alumni email addresses (valuable investment for many other purposes) Need software to administer surveys (or contract out)