The Competency Assessment Toolkit for Professional Psychology

Catherine L. Grus, Ph.D., Associate Executive Director for Professional Education and Training Education Directorate American Psychological Association

THANK YOU
Many thanks to Nadya Fouad, University of Wisconsin Nadine Kaslow, Emory University Benchmarks Workgroups Assessment Toolkit Workgroup

Overview
Considerations in the assessment of
competence Development of the Toolkit Putting the toolkit into practice

Competence
the habitual and judicious use of
communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and community being served.
From Epstein and Hundert (2002)

Competence
is developmental depending on stage of
professional functioning context dependent executed according to ethical principles, guidelines, standards, and values of the profession and requires public verification

Competencies

are elements of competence that are . Observable Measurable Containable Practical Derived by experts And flexible

Assessment of Competence Considerations

Use of both formative and summative
evaluation methods Authentic evaluations Real world tasks, meaningful activities [High] Fidelity: Degree to which
the assessment represents actual performance

 Employ a range of perspectives (multiinformant) and methods Challenge of evaluating attitudes and values Psychometric issues Feasibility/costs Lifelong self-assessment

What methods are commonly used to assess students/ trainees?

In class examinations and course grades Faculty/supervisor completes rating scale Letters of recommendation EPPP, state jurisprudence exams

Developing Faculty/Supervisor Competencies

Training in difficult conversations Develop awareness of potential dual roles in
How to critique constructively

evaluation and how that could influence results

Benefits of having other faculty evaluate students performance Explicit obligations to make evaluations accurate Grade inflation Letters of recommendation to market vs. evaluate

Knowledge of a range of assessment tools,

access to resources toolkits

Competency Assessment for Toolkit for Professional Psychology

Charge from APA Board of Educational Affairs: Develop a Toolkit for professional psychology Purpose: Promote broader implementation of competence assessment and provide information about application of assessment methods to the assessment of competence Coordinated with Benchmarks Work Group

Benchmarks Core Competencies

Foundational Competencies: Professionalism Reflective practice Scientific knowledge and methods Relationships Individual and cultural diversity Ethical and legal standards and policy Interdisciplinary systems Functional Competencies: Assessment Intervention Consultation Research and evaluation Supervision Teaching Administration Advocacy

How are the Benchmarks Organized?

Core Foundational and Functional
Competencies Essential Component: what are the critical elements of/what knowledge/skills/attitudes that make up this competency? Behavioral Anchor: what would it look like if you saw it (essential component)?

Components of Toolkit
Background and Introduction Assessment Method Fact Sheets Grid of Assessment Methods and

Competencies/Essential Components Best used for

Fact Sheet: Portfolio Review

Description A portfolio is a collection of products, gathered by the person being assessed Application Competency Applies to1 Predominant Development Use of2 al Level3

Professionalism

F,S

2,3,4

Reflective Practice E F,S 1,2,3,4 Implementation Deciding on form (e.g., web-based or hard copy) Psychometric Properties Reliability has not been well established due to the variable content included in a portfolio Strengths Provides a low cost assessment strategy Challenges Requires intense commitment of time and labor intensive for the assessors and the person being assessed Future directions Create and implement training modules for assessors and people being assessed in the portfolio assessment strategy

360 Evaluations 1 1 Written exams

Grid of Possible Assessment Methods (Toolkit)

Stand. oral exams

2 1 2 2 1 2 1 1 1 1 1 Implementation 1 Progress evaluation 3 1 2 2 2 3 2 2 1 2

Self-assessment 2

Record reviews

Ratings Performance

Portfolio reviews

OSCE

Consumer surveys 2 2 1

Rating Forms 3 2

Process/outcome data 3

Case reviews

Intervention Knowledge 2 interventions Intervention 2 planning Skills 2

Annual review

Stand. patient interview Simulations/role plays

Assessment Methods in Toolkit

360 evaluation Portfolio OSCE Structured Written & Oral Exams Case Presentation Simulation/Role Play Competence Evaluation Rating Form Self-Assessment

Ratings of live or

recorded performance Standardized Client Interview Client/Patient Process/Outcome Measure Consumer Satisfaction Survey End of Rotation Performance Review

Putting it all Together

Benchmarks

Toolkit

Using the Toolkit and Benchmarks to Guide the Assessment of Student Learning Outcomes

Identify expected student learning outcomes using competency model (e.g., Benchmarks)

Select assessment method(s)

Expected level of competence attainment and training plan developed with trainee

Student demonstrates problems with competencies acquisition

Remediation plan developed

Formative assessment

Student demonstrates emerging competencies

Training plan continues

Summative evaluation to assess readiness for next level of training

Identify expected student learning outcomes using competency model (e.g., Benchmarks)

Program Outcome: Students will demonstrate the following intervention (functional) competencies prior to going on internship.
1. Knowledge of scientific, theoretical, empirical and contextual bases of intervention, including theory, research, and practice 2. Formulates and conceptualizes cases and plan interventions utilizing at least one consistent theoretical orientation 3. Clinical skills 4. Implements evidence-based interventions that take into account empirical support, clinical judgment, and client diversity (e.g., client characteristics, values, and context) 5. Evaluate treatment progress and modify treatment planning as indicated, utilizing established outcome measures

Identify expected student learning outcomes using competency model (e.g., Benchmarks)

Select assessment method(s)

1 1

Grid of Possible Assessment Methods (Toolkit)

2 1 2 2 1 2 1 1 1 1 1 1 Implementation Progress evaluation 3 1 2 2 2 3 2 2 1 2

Stand. patient interview

Knowledge interventions

Intervention planning

Intervention

Skills

Annual review

Case reviews

Process/outcome data

Rating Forms

Consumer surveys

OSCE

Portfolio reviews

Ratings Performance

Record reviews

Self-assessment

Simulations/role plays

Stand. oral exams

Written exams

360 Evaluations

Identify expected student learning outcomes using competency model (e.g., Benchmarks)

Select assessment method(s)

Expected level of competence attainment and training plan developed with trainee

Developmental Level A. Knowledge of Interventions Readiness for Practicum Readiness for Internship
Essential Component: Basic knowledge of scientific, theoretical, and contextual bases of intervention and basic knowledge of the value of evidencebased practice and its role in scientific psychology Behavioral anchor: Articulates the relationship of EBP to the science of psychology Identifies basic strengths and weaknesses of intervention approaches for different problems and populations Essential Component: Knowledge of scientific, theoretical, empirical and contextual bases of intervention, including theory, research, and practice

Behavioral Anchor: Demonstrates knowledge of interventions and explanations for their use based on EBP Demonstrates the ability to select interventions for different problems and populations related to the practice setting Investigates existing literature related to problems and client issues Writes a statement of ones own theoretical perspective regarding intervention strategies

Identify expected student learning outcomes using competency model (e.g., Benchmarks)

Select assessment method(s)

Expected level of competence attainment and training plan developed with trainee

Student demonstrates problems with competencies acquisition

Remediation plan developed

Formative assessment

Student demonstrates emerging competencies

Training plan continues

Remediation Plan

Competenc y Domain/ Essential Component s

Problem Behaviors

Expectations for Acceptable Performance

Trainees Responsibilitie s/ Actions

Supervisors/ Faculty Responsibilities / Actions

Timeframe for Acceptable Performance

Assessmen t Methods

Dates of Evaluatio n

Consequence s for Unsuccessful Remediation

Identify expected student learning outcomes using competency model (e.g., Benchmarks)

Select assessment method(s)

Expected level of competence attainment and training plan developed with trainee

Student demonstrates problems with competencies acquisition

Remediation plan developed

Formative assessment

Student demonstrates emerging competencies

Training plan continues

Summative evaluation to assess readiness for next level of training

For more information:

http://www.apa.org/ed/graduate/competency.html

Coming soon:

Kaslow, N. J., Grus, C. L., Campbell, L. C., Fouad, N. A., Hatcher, R. L., & Rodolfa, E. R. (in press) Competency Assessment Toolkit for Professional Psychology. Training and Education in Professional Psychology.