Setting Institutional Standards for Student Achievement:

Meeting the Call for Academic Quality

Fred Trapp, Cambridge West Partners Carolyn Arnold, Chabot College Daylene Meuschke, College of the Canyons Elaine Kuo and Andrew LaManque, Foothill DeAnza Community
College District

Robert Pacheco, MiraCosta College

Some RP Resources
Learning Assessment Listserv (www.cccnext.net ) RP Group Listserv (www.cccnext.net)

Perspectives (www.rpgroup.org)

Cloud Housing Session Docs http://tinyurl.com/mjdu9ha

Outcomes for the Session

Discover some best practices in setting collegewide standards for achievement. Brainstorm ways to integrate the standards into planning and decision-making processes at your institution. Evaluate the reasonableness of your colleges institution-set standards.

How Well Roll

Trends and a Literature Review (10-15 minutes each) Three College/District Stories (10-15 minutes each)

Interactive Portion (25 minutes)

Reflect on your own institution, its culture, history, mission to consider your own institution-set standards. Compare, borrow, validate

Institution Set Standards: Trends & Literature

Frederick Trapp Cambridge West Partnership, LLC Long Beach City College (retired) October 2013

Professional Literature/National Trends

The three Es of public agencies 1. Economy 2. Efficiency 3. Effectiveness
How high or low is the effectiveness bar is set to define quality service?

ACCJC Annual Report, 2013

Student achievement performance metrics 14b Successful student course completion rate 15b Student retention percentage 16b Student degree completion 17b Student transfer to 4-year colleges 18b Student certificate completion A standard is the level of performance set by the institution to meet educational quality and institutional effectiveness expectations. This number may differ from a performance improvement goal which an institution may aspire to meet.

Standard I.B.2
The institution sets goals to improve its effectiveness consistent with its stated purposes. The institution articulates its goals and states the objectives derived from them in measurable terms so that the degree to which they are achieved can be determined and widely discussed.

Standard I.B.3
The institution assesses progress toward achieving its stated goals and makes decisions regarding the improvement of institutional effectiveness in an ongoing and systematic cycle of evaluation, integrated planning, resource allocation, implementation, and re-evaluation.

How bad does performance have to get before we decide it is unacceptable and we need to do something?

US Dept. of Ed., New Regulations

34 Code of Federal Regulations (CFR) & USDE Guidelines for 602.16(a)(1)(i).
Standards effectively address success with respect to student achievement in relation to the institutions mission, which may include different standards for different institutions or programs, as established by the institution, including, as appropriate, consideration of course completion, State licensing examinations, and job placement rates. Whether institutionally-developed standards to demonstrate student success are being used by the accreditor in the accreditation assessment, and the institutions performance with respect to student achievement is assessed.

ACCJC Direction to Teams

Evaluation teams will examine student achievement data at the programmatic and institutional levels. The institution must set standards of satisfactory performance for student achievement and evaluate itself against those standards at both levels. 1. Examine the institutional set-standards for student achievement and assess their appropriateness (reasonableness) 2. Consider the set-standards in the context of satisfactory performance, goals for improvement (achievement and learning). 3. Cite this information as evidence in describing its evaluation of how well the institution accomplishing its mission.

ACCJC Direction to Teams

Describe in the team report Not the basis of a recommendation or a sanction

Health Care- Setting Standards

National Quality Forum (1999)
700 best in class quality measures, rooted in evidence-based practice

Medical Loss Ratio (80-85% of premium goes to care) Centers for Medicare & Medicaid Services (CMS)- Accountable Care Organizations (ACOs) initiative performance domains
Patient/caregiver experience Care coordination/patient safety Preventive care At-risk populations

Moving from fee-for-service to pay-for-performance using national performance benchmark data and a sliding scale.

Performance Scorecards and Dashboards

Strategic means visually to monitor achievement of objectives
Rests on some cause and effect beliefs

Association for Institutional Research (AIR) study of 66 higher education dashboards

Category Financial Indicators Admissions Enrollment Faculty Student Outcomes Student Engagement Academic information Physical Plant Satisfaction Research External Ratings % Using 80% 79% 77% 77% 72% 58% 55% 38% 35% 35% 21%

Setting the Performance Standard

Levels set by
1. 2. 3. Benchmarking (www.NCCBP.org) Industry standards Recent performance history
5-year average 3-year rolling average

Color codes for performance (stop light)

Red- poor; yellow- below target; green- on target

Performance expressed as a range

=>5% above- very good =<5% below- needs attention + or 5%- on target

Performance Funding for Higher Education

No Action 30%

In place 30%

In transitio n 8% 50 States Considering 32%

National Council of State Legislatures

Measuring Institutional Effectiveness through Institutional Standards One Colleges Approach

Daylene M. Meuschke, Ed.D. Director, Institutional Research College of the Canyons Strengthening Student Success Conference October 2013

The Plan / Timeline

March 2013
*Required to establish standards *Convened College Planning Team sub-group *Established indicators and standards

April-June 2013
Source: www.photo-dictionary.com

*Presented Indicators-Senate, College Planning Team, Board of Trustees *Obtained feedback on Performance Indicators

April 2013-present *Held monthly meetings *Using RP Groups Principles of Redesign to guide planning activities

Source: www.2.acs.ncsu.edu

Existing Data Sources

Scorecard ACCJC Annual Survey
Student Learning / Institutional Effectiveness

Institutional Data

COCs Performance Indicators and Institutional Standards

Coming soonMetric for Career Technical Education

Planning Guided by RP Groups Principles of Redesign

1. Accelerate entry into coherent programs of study 2. Minimize time required to get college-ready 3. Ensure students know the requirements to succeed 4. Customize and contextualize instruction 5. Integrate student support with instruction 6. Continually monitor student progress and proactively provide feedback 7. Reward behaviors that contribute to completion 8. Leverage technology to improve learning and program delivery

Infusing the Standards

College-wide College Planning Team Performance Indicators Sub-Committee Skills for Success Committee Roadmap Project (AAC&U) Grants (e.g. Title III) Departments Academic and Administrative Program Reviews Documented in Research Brief #57 More to Come!

Just scratching the surface

Source: www.burkemuseum.org

Plan

Next Steps: Revise Indicators Add CTE Revise Standards Completion (success) Continue Disaggregating Connect to departments Some are clean, clear Others (e.g., transfer)more problematic

Evaluate

Implement

Thank you!

Any questions?

Measuring up to the Standards

Creating Meaningful Student Groups to measure progress towards Institutional Standards

Carolyn Arnold, Ph.D.

Coordinator, Office of Institutional Research Planning, Review, and Budget Council (PRBC)

Chabot College

Chabot Institutional Standards

Setting the standards

Analyzed long term trends for the 5 standards Chose five year rolling averages

Reflect recent trends Smooth out extreme highs or lows Sometimes above standard; sometimes below

Assumption no matter where we set them

How can we improve these outcomes?

Chabot Institutional Standards

Challenge 1 to improving the outcomes

Degrees, Transfers, and Certificates are longterm outcomes

Take semesters or years to complete Completion numbers fluctuate year to year

Most college initiatives focus on short term objectives:

More students taking basic skills courses Increasing success in College English, Math

How can short and long term be connected?

Chabot Institutional Standards

Challenge 2 to improving the outcomes

Not all students aiming for these outcomes

Have different goals Going towards goals at different speeds Starting from different places

How can we take into account student differences in goals, speed, and starting places when we attempt to improve outcomes?

We do want to improve!

Chabot Strategic Plan Goal

Increase the number of students who achieve their educational goal in a reasonable time (outcomes)

Institutional Research supposed to measure the number

by clarifying pathways and providing more information and support (methods)

College initiatives supposed to improve the number

Not One Number!

Why we created distinct student groups

To take into account student differences

in goals, speed, levels

To monitor progress and outcomes of student cohorts with different educational goals To determine reasonable time to completion for each group To target college initiatives and support services needed by each group

Creating educational goal groups for new students

Groups based on: Their educational goal Self-reported educational goal How long they will take How many units they take (FT/PT) Where they start (assessment in English)

Creating educational goal groups

Ed Goal
Transfer or Degree (GE) Undecided Certificate or Job training
Cert/Job/Und/ Pers Devel

Units
Fulltime Parttime

English Assessment College Basic Skills Not Assessed College Basic Skills Not Assessed

Student Ed Goal Groups

6+ units Full-time Part-time 6-11 units Under 6 units

Laser (FT) College Laser (FT) Basic Skills Laser (FT) Not Assessed Seeker (PT) College Seeker (PT) Basic Skills Seeker (PT) Not Assessed Explorer Career-builder FT Career-builder PT Skills-builder

10 student educational goal groups

Laser (FT) College Laser (FT) Basic Skills Laser (FT) Not Assessed Seeker (PT) College Seeker (PT) Basic Skills Seeker (PT) Not Assessed Explorer Career-builder FT Career-builder PT Skills-builder

Chabot Fall 2012 New Students

Percentage in each Educational Goal Group
12% 3% 2% 22% 10%

8%
Laser FT College Laser FT BasSkls Laser FT NoAssess Seeker PT College Seeker PT BasSkls Seeker PT NoAssess Explorer Career Builder FT Career Builder PT Skills Builder

10%

3% 5%

25%

Student ed goal groups:

Look and behave differently by

Age
Gender Race-ethnicity Day vs. evening Early Decision Financial Aid Pct assessed in Math Math assessment levels Types of major

Student ed goal groups:

Taking differences into account

How do we know what is the reasonable time it will take each group to achieve their educational goal?

First, answer challenge #1 to meeting the standards outcomes Connect long term outcomes with short term objectives by setting milestones

Student ed goal groups

progress at different rates

Early Engagement

Enrolled in basic skills Completed basic skills Persistence Completed College Math or English Earned Transferable units Earned CTE units Earned certificates or degrees Transfer ready / transferred Employed

Momentum

Progress

Completion

Student ed goal groups

allow us to:

Get beyond one number for each outcome.

10 student groups Progress milestones

Measure progress along the way.

Determine reasonable time to milestones and completion for each group based on past cohorts Tailor college initiatives for each group and see effect of initiatives.

Setting ACCJC Institutional Standards at Foothill College

Strengthening Student Success Conference 2013 Elaine Kuo, Ph.D. Andrew LaManque, Ph.D.

Legal Foundation Summary

The requirement for Institutional Standards flow from Federal Regulations The Regulations are not new (2009) but the Commission has stated that the Federal interpretation changed last year. The new interpretation was included in evaluation team training in Spring 2013.

40

The Accreditation Standards Process

Recommendations to meet the standards or make them reasonable?

Colleges set Institutional Standards

Visiting teams evaluate whether Standards are met and reasonable

ACCJC collects annual data on Institutional Standards

41

Foothill Approach
The Process Accreditation team Shared governance groups
Planning and Resource Council (PaRC) Academic Senate Classified Senate

The Data Most recent term or year

Fall 2012 or 2011-12

Longitudinal
2007-08 to 2011-12

Disaggregation
Ethnicity Program

Imbed in college planning, program review

Sources
CCCCO data mart FHDA IR&P
42

The Data: Setting the Standards

Table 1. Student Course Completion Rate Fall 2012 Five-Year Average Five-Year Average by Ethnicity-Low Five-Year Average by Program-Low Institutional Standard Recommendation
Source: CCCCO Data Mart, FHDA IR&P

75% 76% 61% 60% 55%

Fall rates:
2011: 74% 2010: 76% 2009: 77% 2008: 79%

Completion based on passing course with A, B, C, or P grade; includes all courses offered in Fall term

43

The Process: Shared Governance Review

Institutional-set standards scheduled for annual review Methodology and recommendations accepted by main shared governance body and college president

Presentation and discussion at main shared governance body Discuss methodology with Accreditation Team

Key players include: ALO, VPI, IR, Academic Senate President, Classified Senate President
44

The Process: Program Review

Shared Governance Integrated Planning & Budget Taskforce

Program Review Templates Student equity and institutional standards section

Program Review Committee

Data sources: Program review data sheets, CCCCO data mart

Prompts include: Comment on your programs course success data, including any differences in completion rates by student demographics as well as efforts to address these differences.

45

Institution and Program Standards

For discussion: How should a department course success / completion rate reported during program review relate to the institution standard?
If the department is below the institutional standard should there be a plan for getting it up to the institutional standard?
46

Shaping the Dialogue

Three Roles, One Product Three Roles:

Scrivener Time Keeper Facilitator

Product:

Summary Sheet
Notes Three takeaways from the discussion One Muddiest Point

Shaping the Dialogue

How were the institutional set standards developed at your institution? How might these standards be integrated with your colleges decision-making and planning processes? How would you justify the reasonableness of the institutional set standards in the context of your mission?

Wrap Up:
What Have We Learned?

Discover some best practices in setting college-wide standards for achievement. Brainstorm ways to integrate the standards into planning and decision-making processes at your institution. Evaluate the reasonableness of your colleges institutionset standards.