32nd SGIM Annual Meeting

May 13, 2009

The Six-Step Approach to Curriculum Development

David E. Kern, MD, MPH Patricia A. Thomas, MD Mark T. Hughes, MD, MA L. Randol Barker, MD Eric B. Bass, MD, MPH Joseph A. Carrese, MD, MPH Leah Wolfe, MD

LEARNING OBJECTIVES By the end of this precourse, participants will be able to: A. Describe the six-steps of curriculum development: 1. Problem identification and general needs assessment 2. Targeted needs assessment 3. Goals and objectives 4. Educational Strategies 5. Implementation 6. Evaluation and Feedback B. Apply the steps in curricular planning exercises C. Identify additional resources that are available to help them improve their curriculum development skills. AGENDA 1:30-2:15 Lecture: Step 1. Problem Identification and General Needs Assessment Step 2. Targeted Needs Assessment Step 3. Goals & Objectives Facilitated Small Group Exercise #1: Applying Steps 1 to 3 to a curricular project.

2:15-3:00

3:00-3:15 Break 3:15-4:00 Lecture: Step 4. Educational Strategies Step 5. Implementation Step 6. Evaluation Facilitated Small Group Exercise #2: Applying Steps 4 to 6 to a curricular project. Summary Dissemination Additional Resources Evaluation

4:00-4:45 4:45-4:55

4:55-5:00

32nd SGIM Annual Meeting

May 13, 2009

SMALL GROUP EXERCISE #1

Step 1: Problem Identification and General Needs Assessment Step 2: Targeted Needs Assessment Step 3: Goals and Objectives

1. Based on your knowledge of the literature and your targeted learners, build a short logical

argument for the curriculum you are developing, moving from problem identification to the gap between the current versus ideal approaches of addressing the problem, to specific information you have about your targeted learners and targeted learning environment. The logical argument should lead to the objective you propose in the second question. It could also serve as the introduction to a paper on your curriculum. (See Tables 2.1, 2.2, and 3.1.)

2. Please write one cognitive (knowledge), affective (attitude), or psychomotor (skill or

performance) objective for your curriculum, whichever is most relevant. Remember: Who will do how much of what by when? (See Tables 4.1-4.3.) (Preview: This objective will determine your educational and evaluation methods, which you will detail in the next small group exercise.)

32nd SGIM Annual Meeting

May 13, 2009

Table 2.1 Identification and Characterization of the Health Care Problem Whom does it affect? Patients Health care professionals Medical educators Society What does it affect? Clinical outcomes Quality of life Quality of health care Use of health care and other resources Medical and non-medical costs Patient and provider satisfaction Work and productivity Societal function What is the quantitative and qualitative importance of the effects?

Table 2.2 The General Needs Assessment What is currently being done by the following? Patients Health care professionals Medical educators Society What personal and environmental factors impact upon the problem? Predisposing Enabling Reinforcing What ideally should be done by the following? Patients Health care professionals Medical educators Society What are the key differences between the current and ideal approaches?

Tables from: Kern DE, Thomas PA, Hughes MT, eds. Curriculum Development for Medical Education: A Six-Step Approach. 2nd ed. Baltimore (MD): Johns Hopkins University Press; 2009. In Press.

32nd SGIM Annual Meeting

May 13, 2009

Table 3.1: Content Potentially Relevant to a Needs Assessment of Targeted Learners Content about Targeted Learners Previous training and experiences relevant to the curriculum Already planned training and experiences relevant to the curriculum Existing characteristics / proficiencies / practices: Cognitive: knowledge, problem-solving abilities Affective: attitudes, values, beliefs, role expectations Psychomotor: skills / capabilities (e.g. history, physical examination, procedures, counseling); current behaviors / performance / practices Perceived deficiencies and learning needs Preferences and experiences regarding different learning strategies Synchronous (educator sets time, such as with noon lecture) Asynchronous (learner decides on learning time, such as with computer learning) Duration (amount of time learner thinks is needed to learn or that they can devote to learning) Methods (e.g. lectures, readings, web-based computer interactive modules, case-based discussions, group learning, role plays / simulations, supervised experience) Content about Learning Environment Related existing curricula Needs of stakeholders other than the learners (course directors, clerkship directors, residency program directors, accrediting bodies, others) Barriers, enabling, and reinforcing factors that affect learning by the targeted learners The informal and hidden curriculum Incentives Resources Patients and Clinical Experiences Faculty Information Resources Computers Audio-visual Equipment Role Models, Teachers, Mentors Other Table from: Kern DE, Thomas PA, Hughes MT, eds. Curriculum Development for Medical Education: A Six-Step Approach. 2nd ed. Baltimore (MD): Johns Hopkins University Press; 2009. In Press.

32nd SGIM Annual Meeting Table 4.1. Verbs Open to More and Fewer Interpretations Verbs Open to More Interpretations Verbs that frequently apply to Cognitive Objectives: Taxonomy of cognitive objectives (2,3) know Remember (recall of facts) Verb Verbs Open to Fewer Interpretations

May 13, 2009

identify list recite define recognize retrieve define contrast interpret classify describe sort explain illustrate Implement execute use (a model, method) complete differentiate distinguish organize deconstruct discriminate detect judge critique test design hypothesize construct produce

understand

Understand

be able know how appreciate

Apply

Analyze

Evaluate

know how

Create

Verbs that frequently apply to Affective Objectives: appreciate grasp the significance of believe enjoy internalize Verbs that frequently apply to Psychomotor Objectives: rate as valuable, rank as important identify, rate, or rank as a belief or opinion rate or rank as enjoyable use one of above terms

32nd SGIM Annual Meeting Skill / Competence: be able know how Behavior / Performance Internalize

May 13, 2009

demonstrate show use or incorporate into performance (as measured by)

Other Verbs: learn teach (use one of the above terms) (use one of the above terms; do not confuse the teacher and the learner in writing learner objectives)

Table from: Kern DE, Thomas PA, Hughes MT, eds. Curriculum Development for Medical Education: A Six-Step Approach. 2nd ed. Baltimore (MD): Johns Hopkins University Press; 2009. In Press.

32nd SGIM Annual Meeting

May 13, 2009

Table 4.2. Examples of Less Well Written and Better Written Objectives Less Well Written Objectives Residents will learn the techniques of joint injections. [The types of injections to be learned are not specified. The types of residents are not specified. It is unclear whether cognitive understanding of the technique is sufficient, or whether skills must be acquired. It is unclear by when the learning must have occurred, and how proficiency could be assessed. The objective on the right addresses each of these concerns.] Better Written Objectives By the end of their residency, each family practice resident will have demonstrated at least once (according to the attached protocol) the proper techniques of: - subacromial, bicipital, and intra-articular shoulder injection; - intra-articular knee aspiration and/or injection; - injections for lateral and medial epicondylitis; - injections for deQuervain's tenosynovitis; and - aspiration and/or injection of at least one new bursa, joint, or tendinous area, using appropriate references and supervision. By the end of their internal medicine ambulatory medicine clerkship, each 3rd year medical student will have achieved cognitive proficiency in the diagnosis and management of hypertension, diabetes, angina, chronic obstructive pulmonary disease, hyperlipidemia, alcohol and drug abuse, smoking, and asymptomatic HIV infection, as measured by acceptable scores on interim tests and the final examination. By the end of their internal medicine clerkship, each 3rd year medical student will have seen and discussed with their preceptor, or discussed in a case conference with colleagues, at least one patient with each of the above disorders. Physician practices, whose staff complete the 3-session communications skills workshops, will have more satisfied patients. [This objective does not specify the comparison group or what is meant by "satisfied". The objective on the right specifies more precisely which practices will have more satisfied patients, what the comparison group will be, and how satisfaction will be measured. It specifies one aspect of performance as well as satisfaction. One could look at the satisfaction questionnaire and telephone management monitoring instrument for a more precise description of the outcomes being measured.] Physician practices, which have 50% of their staff complete the 3 session communications skills workshops, will have lower complaint rates, higher patient satisfaction scores on the yearly questionnaire, and better telephone management as measured by random simulated calls, than practices that have lower completion rates.

By the end of their internal medicine clerkship, each 3rd year medical student will be able to diagnose and manage common ambulatory medical disorders. [This objective specifies "who" and "by when", but is vague about what it is the medical students are to achieve. The two objectives on the right add specificity to the latter.]

Table from: Kern DE, Thomas PA, Hughes MT, eds. Curriculum Development for Medical Education: A Six-Step Approach. 2nd ed. Baltimore (MD): Johns Hopkins University Press; 2009. In Press.

32nd SGIM Annual Meeting

May 13, 2009

Table 4.3. Types of Objectives. Examples from a Smoking Cessation Curriculum for Residents

INDIVIDUAL LEARNER LEARNER Cognitive (Knowledge)

AGGREGATE OR PROGRAM

Affective (Attitudinal)

Psychomotor (Skill or Competence)

Psychomotor (Behavioral or Performance)

By the end of the curriculum, each By the end of the curriculum, 80% resident will be able to list the 5of residents will be able to list the step approach to effective smoking 5-step approach to effective cessation counseling. smoking cessation counseling, and 90% will be able to list the 4 critical (asterisked) steps. --------------------------------------------By the end of the curriculum, each By the end of the curriculum there primary care resident will rank will have been a statistically smoking cessation counseling as significant increase in how primary an important and effective care residents rate the importance intervention by primary care and effectiveness of smoking physicians (3 on a 4-point scale). cessation counseling by primary care physicians. --------------------------------------------During the curriculum, each primary During the curriculum, 80% of care resident will demonstrate in residents will have demonstrated role play a smoking cessation in role play a smoking cessation counseling technique that counseling technique that incorporates the attached 5-steps. incorporates the attached 5-steps. ---------------------------------------------By 6 months after completion of the By 6 months after completion of the curriculum, each primary care curriculum, there will have been a resident will have negotiated a statistically significant increase in plan for smoking cessation with the % of GIM residents who have 60% of his/her smoking patients, negotiated a plan for smoking or have increased the percentage cessation with their patients. of such patients by 20% from baseline. Each primary resident will have attended both sessions of the smoking cessation workshop. By 12 months after completion of the curriculum, the smoking cessation rate (for 6 months) for the patients of each primary care resident will increase 2-fold from baseline or be 10%. 80% of primary care residents will have attended both sessions of the smoking cessation workshop. By 12 months after completion of the curriculum, there will have been a statistically significant increase in the % of primary care residents' patients who have quit smoking (for 6 months).

PROCESS

PATIENT OUTCOME

Table from: Kern DE, Thomas PA, Hughes MT, eds. Curriculum Development for Medical Education: A Six-Step Approach. 2nd ed. Baltimore (MD): Johns Hopkins University Press; 2009. In Press.

32nd SGIM Annual Meeting

May 13, 2009

SMALL GROUP EXERCISE #2

Step 4: Educational Strategies Step 5: Implementation Step 6: Evaluation

1. Choose 2 educational strategies that you could use in your curriculum that would help learners achieve the learner objective that you identified. Are the strategies congruent with your objective? (See Tables 5.2-5.3.)

2. Identify one or two evaluation methods that you could use for assessing learner achievement of your learner. (See Table 7.3.) Is (are) your evaluation method(s) congruent with your educational objective and methods?

3. Are your proposed educational strategies and evaluation plans feasible in terms of available resources (personnel, equipment, space, time, funding)? (See Table 6.1)

32nd SGIM Annual Meeting

May 13, 2009

Table 5.2 Matching Educational Methods to Objectives* Type of Objective Cognitive: Knowledge +++ +++ +++ ++ Cognitive: ProblemSolving + + ++ ++ +++ +++ + ++ ++ +++ +++ ++ ++ +++ +++ +++ + + + + + + + + + + ++ ++ ++ ++ ++ + ++ + ++ + ++ ++ ++ ++ + + + ++ +++ +++ +++ +++ +++ + Affective: Attitudinal + + Psychomotor: Skills or Competence + + + + +++ +++ + + +++ +++ + + + + ++ ++ + + + +++ + +++ Psychomotor: Behavioral or Performance

Educational Method Readings Lectures Programmed learning Discussion Reflection on experience Feedback on performance Small-group learning Problem-based learning Team-based learning Learning projects Role models Demonstration Role plays Artificial models and simulation Standardized patients Real life experiences Audio or video review of learner Behavioral / environmental interventions**

*blank = not recommended; + = Appropriate in some cases, usually as an adjunct to other methods; ++ = good match; +++ = excellent match (consensus ratings by author and editors). ** = Removal of barriers to performance; provision of resources that promote performance; reinforcements that promote performance. Table from: Kern DE, Thomas PA, Hughes MT, eds. Curriculum Development for Medical Education: A Six-Step Approach. 2nd ed. Baltimore (MD): Johns Hopkins University Press; 2009. In Press.

32nd SGIM Annual Meeting

May 13, 2009

TABLE 5.3 Summary of Advantages & Limitations of Different Educational Methods Educational Method Readings Advantages Low cost Cover fund of knowledge Little preparation time Low cost Accommodate large numbers of learners Structured presentation of complicated topics Active learning Don't need clinical material at hand Safe simulations for learners Immediate feedback on knowledge, sequencing, efficiency, clinical decision-making Learner applies new knowledge Active learning Permits assessment of learner needs Allows learner to apply newly acquired knowledge Suitable for higher order cognitive objectives: problem-solving and clinical decision-making; can address affective objectives Exposes students to different perspectives Promotes learning from experience Promotes self-awareness / mindfulness Can be built into discussion / group learning activities Can be done individually through assigned writings / portfolios Disadvantages Passive learning Learners must be motivated to complete Passive learning Teacher-centered Quality depends on speaker/audiovisual material Developmental costs if not commercially available

Lectures

Programmed Learning

Discussion

More faculty intensive than readings or lectures Cognitive/experience base required of learners Group dependent Usually facilitator dependent

Reflection on experience

Requires protected time Usually requires scheduled interaction time with another / others. Often facilitator dependent

Feedback on performance Promotes learning from experience Can be used with role play, artificial models / simulation, standardized patients, clinical experience, and audio/video review Small group learning Active learning Resources usually available Allows multidisciplinary approaches Suitable for team-based and problembased learning, clinical decisionmaking, community-based projects Encourages cooperation, team-work among learners. Incorporates discussion.

Requires observer who is a skilled provider of feedback

Group should have some training in group process skills, conflict management, etc. May require faculty facilitators with training in above Time required for successful functioning

32nd SGIM Annual Meeting Problem-based leaning (PBL) Active learning Facilitates higher cognitive objectives: problem-solving and clinical decision-making; can incorporate objectives which cross domains: ethics, humanism, cost-efficiency Active learning Facilitates higher cognitive objectives; constructive knowledge Students take responsibility for learning Collaborative Uses less faculty than PBL / other small group learning methods Active learning Promote, teach self-directed learning Learners sets individual learning objectives Suitable for higher-order cognitive objectives Faculty often available Impact often seems profound

May 13, 2009 Developmental costs Requires faculty facilitators and small groups Less efficient for transferring factual information Developmental costs for Readiness Assurance Test (RATs) and application exercises Students need to be self-directed Requires orientation of students to process of team work and peer evaluation Learners need motivation Learners need basic skills to access and optimally use learning resources Requires effective faculty mentor Require valid evaluation process to identify effective role models Specific interventions usually unclear Impact depends on interaction between specific faculty member and learner Outcomes multifactorial and difficult to assess Passive learning Teacher-centered Quality depends on teacher/audiovisual material Require trained faculty facilitators Learners need some basic knowledge or skills Can be resource intensive if large numbers of learners

Team-based learning (TBL)

Learning projects

Role models

Demonstration

Efficient method for demonstrating skills/procedures

Role plays

Suitable for objectives which cross domains: knowledge, attitudes, and skill Efficient Low cost Can be structured to be learnercentered Safe environment for skills practice Safe environments to practice skills Learners can use at own pace; less faculty supervision required Ensure appropriate clinical material Approximate "real life" more closely than role plays Safe environment for skills practice Can give feedback to learners on performance Can be reused for ongoing curricula

Artificial models and simulation Standardized patients

May not be available for specific curriculum Can be expensive Cost Expertise required to develop and train standardized patients

32nd SGIM Annual Meeting Clinical experience "Real life" Promotes learner motivation and responsibility Promotes higher level cognitive, attitudinal, skill, and performance learning

May 13, 2009 Requires clinical material when learner is ready Requires faculty to supervise and to provide feedback Learner needs basic knowledge or skill Needs to be monitored for case mix, appropriateness Requires reflection, follow-up Requires trained faculty/facilitators Recording can be awkward or intrusive, and pose logistic problems Requires patient permission Assume competence Require control over learners real-life environment

Audio or video reviews of learner

Provides accurate feedback on performance Provides opportunity for selfobservation Influence performance

Behavioral / environmental interventions*

* Removal of barriers to performance; provision of resources that promote performance; reinforcements that promote performance. Table from: Kern DE, Thomas PA, Hughes MT, eds. Curriculum Development for Medical Education: A Six-Step Approach. 2nd ed. Baltimore (MD): Johns Hopkins University Press; 2009. In Press.

32nd SGIM Annual Meeting Table 7.3. Uses, Strengths, and Limitations of Commonly Used Evaluation Methods

May 13, 2009

METHOD Global Rating Forms (separated in time from observation)

USES Cognitive, affective, or psychomotor attributes; real life performance

STRENGTHS Economical Can evaluate anything Open ended questions can provide information for formative purposes

LIMITATIONS Subjective Rater biases Inter and intra-rater reliability Raters frequently have insufficient data upon which to base ratings Subjective Rater biases Agreement with objective measurements often low Limited acceptance as method of summative evaluation Subjective Rater biases Requires qualitative evaluation methods to analyze Focus varies from respondent to respondent

Self-assessment forms

Cognitive, affective, psychomotor attributes; real life performance

Economical Can evaluate anything Promotes self-assessment Useful for formative evaluation

Essays on Respondent's Experience

Attitudes, feelings, description of respondent experiences, perceived impact

Rich in texture Provides unanticipated as well as anticipated information Respondent-centered

Written or computerinteractive tests

Knowledge; higher level cognitive ability

Often economical Constructing tests of higher Objective level cognitive ability, or Multiple choice exams can computer-interactive achieve high internal tests, can be resource consistency reliability, intensive broad sampling Reliability and validity vary Good psychometric with quality of test (e.g. properties, low cost, low questions that are not faculty time, easy to carefully constructed can score be interpreted differently Widely Accepted by different respondents, Essay type questions or there may be an computer-interactive tests insufficient number of can assess higher level questions to validly test a cognitive ability, domain) encourage students to integrate knowledge, reflect problem solving

32nd SGIM Annual Meeting Oral Examinations Knowledge; higher level cognitive ability; indirect measure of affective attributes

May 13, 2009 Flexible, can follow-up and Subjective scoring explore understanding Inter and intra-rater Learner-centered reliability Can be integrated into case Reliability and validity vary discussions with quality of test (e.g. questions that are not carefully constructed can be interpreted differently by different respondents, there may be an insufficient number of questions to validly test a domain) Faculty intensive Can be costly Economical Subjective Constructing reliable and valid measures of attitudes requires time and skill Subjective Rater biases Constructing reliable and valid measures of attitudes requires time and skill Requires interviewers Subjective Requires skilled interviewer or facilitator to control group interaction and minimize facilitator influence on responses Does not yield quantitative information Information may not be representative of all participants

Questionnaires

Attitudes; perceptions; suggestions for improvement

Individual Interviews

Attitudes; perceptions; suggestions for improvement

Flexible, can follow-up and clarify responses Respondent-centered

Group Interviews / Discussions

Attitudes; perceptions; suggestions for improvement

Flexible, can follow-up and develop / explore responses Respondent-centered Efficient means of interviewing several at once Group interaction can enrich or deepen information Can be integrated into teaching sessions

32nd SGIM Annual Meeting Direct Observation Skills; performance First hand data Can provide immediate feedback to observed Development of standards, use of observation check lists, and training of observers can increase reliability and validity. The Objective Structured Clinical Examination (OSCE) (62,63) and Objective Structured Assessment of Technical Skills (OSATS) (64-66) combine direct observation with structured checklists to increase reliability and validity. Objective Reliability and accuracy can be measured and enhanced by the use of standards and the training of raters Unobtrusive

May 13, 2009 Rater biases Inter and intra-rater reliability Personnel intensive Unless observation covert, assesses capability rather than real-life performance

Performance Audits

Record keeping; provision of recorded care (e.g. tests ordered, provision of preventive care measures, prescribed treatments)

Dependent on what is reliably recorded; much care is not documented Dependent on available, organized records or data sources

Table from: Kern DE, Thomas PA, Hughes MT, eds. Curriculum Development for Medical Education: A Six-Step Approach. 2nd ed. Baltimore (MD): Johns Hopkins University Press; 2009. In Press.

32nd SGIM Annual Meeting Table 6.1: Checklist For Implementation ____

May 13, 2009

Identify resources ____ Personnel: faculty, AV, computing, secretarial and other support staff, patients ____ Time: faculty, support staff, learners ____ Facilities: space, equipment, clinical sites, virtual space (servers, content management software) ____ Funding/costs: direct financial costs, hidden or opportunity costs Obtain support ____ Internal from: those with administrative authority (dean's office, hospital administration, department chair, program director, division director, etc.), faculty, learners, other stakeholders for: personnel, resources, political support ____ External from: government, professional societies, philanthropic organizations or foundations, accreditation bodies, other entities (e.g. managed care organizations), individual donors for: funding, political support, external requirements, curricular or faculty development resources Develop administrative mechanisms to support the curriculum ____ Administrative structure: to delineate responsibilities and decision-making ____ Communication content: rationale; goals and objectives; information about the curriculum, learners, faculty, facilities & equipment, scheduling; changes in the curriculum; evaluation results; etc. mechanisms: websites, memos, meetings, syllabus materials, site visits, reports, etc. ____ Operations: preparation and distribution of schedules and curricular materials; collection, collation and distribution of evaluation data; curricular revisions and changes, etc. ____ Scholarship: plans for presenting and publishing about curriculum; human subjects protection considerations; IRB approval, if necessary Anticipate and address barriers ____ Financial & Other Resources ____ Competing Demands ____ People: attitudes, job/role security, power & authority, etc. Plan to introduce the curriculum ____ Pilot ____ Phase-in ____ Full implementation

____

____

____

____

Table from: Kern DE, Thomas PA, Hughes MT, eds. Curriculum Development for Medical Education: A Six-Step Approach. 2nd ed. Baltimore (MD): Johns Hopkins University Press; 2009. In Press.

32nd SGIM Annual Meeting

THE 6-STEP APPROACH TO CURRICULUM DEVELOPMENT

David E. Kern, MD, MPH Patricia A. Thomas, MD Mark T. Hughes, MD L. Randol Barker MD, ScM Eric B. Bass, MD, MPH Joseph A. Carrese, MD, MPH May 13, 2009

No conflicts of interest to disclose.

Goals of Workshop
By the end of the workshop, participants will demonstrate their ability to a Describe the principles of curriculum development a Apply these principles to enhance their work in educational program development a Apply these principles to benefit learners, society, and their own career advancement as educational scholars

CURRICULUM: DEFINITION

aA planned educational experience

CURRICULUM DEVELOPMENT: UNDERLYING ASSUMPTIONS

a Educational programs have goals or aims, whether articulated or not a Medical educators have professional and ethical obligations to meet the needs of their learners, patients and society a Medical educators should be held accountable for the outcomes of their interventions a A logical, systematic approach to CD will help achieve these goals

Rudolf Virchow
Medical instruction does not exist to provide individuals with an opportunity of learning how to make a living, but in order to make possible the protection of the health of the public.

CURRICULUM DEVELOPMENT PRECOURSE: OVERVIEW

a6 Steps

RRC REQUIREMENTS FOR A Written Curriculum (U.S.)

For each rotation or major learning experience, must include: a Educational purpose (Goals & Objectives) a Teaching methods (Educational strategies) a Method of evaluation of resident performance

LCME (U.S.)

aSimilar requirements

(Evaluation)

ACGME OUTCOME PROJECT (U.S.) (http://www.acgme.org/Outcome)

a6 Competencies
1. 2. 3. 4. 5. 6. Patient Care Medical Knowledge Practice-Based Learning & Improvement Interpersonal and Communication Skills Professionalism Systems-Based Practice

Glassick* Criteria for Scholarship

1. 2. 3. 4. 5. 6. Clear goals and aims Adequate preparation Appropriate methods Significant results Effective dissemination Reflective critique

aFocus on:
objectives evaluation continuous improvement

*Scholarship Assessed:Evaluation of the Professoriate,1997

Glassick Criteria STEP 1: PROBLEM IDENTIFICATION AND GENERAL NEEDS ASSESSMENT building the foundation for meaningful objectives
1. Clear goals and aims 2. Adequate preparation 3. Appropriate Methods 4. Significant Results 5. Effective Dissemination 6. Reflective Critique

PROBLEM IDENTIFICATION & GENERAL NEEDS ASSESSMENT: WHY? aBuilds a rationale for your curriculum aFocuses a curriculums goals and objectives aWhich in turn focus the educational and evaluation strategies aMakes you an expert and a scholar

STEP 1: PROBLEM IDENTIFICATION

aIdentify and Characterize the Health Care Problem That Will Be Addressed by the Curriculum

Whom Does the Problem Affect?

aPatients aSociety aHealth Care Professionals aTrainees

What Does the Problem Affect?

aClinical Outcomes aQuality of Life aQuality of Health Care aUse of Health Care and Other Resources aMedical and Non-medical Costs aPatient and Provider Satisfaction aWork and Productivity aSocietal Function

Example: Problem Identification

We need a curriculum in communication skills for our residents becomes: Why is it important for residents to be effective communicators?
`What is the impact `What is the impact `What is the impact `What is the impact `What is the impact satisfaction? on on on on on the process of care? clinical outcomes? malpractice? utilization and costs? patient and physician

Example: Problem Identification

We need a curriculum in professionalism for our residents becomes: What is professionalism, and why is it important for our residents to behave professionally?
` What are the critical, evidenced-based or agreed upon components of professionalism? ` How do physicians professional behavior impact on patients (satisfaction, clinical care outcomes), functioning of the health care team, and society?

GENERAL NEEDS ASSESSMENT

GENERAL NEEDS ASSESSMENT

aWhat is the ideal approach to the problem?
`By `By `By `By patients? practitioners? medical educators? society at large

aWhat is currently being done about the problem?

`By `By `By `By patients? practitioners? medical educators? society at large

GENERAL NEEDS ASSESSMENT

Example: Communication Skills

(Kern DE et.al. Residency training in interviewing skills and the psychosocial domain of medical practice. J Gen Intern Med 1989; 4:421-431.)

General Needs Assessment = Ideal Approach - Current Approach

a CS critical to diagnosis, patient education, trust, patient satisfaction, clinical decision-making a CS related to patient outcomes: satisfaction, compliance, diabetes control, malpractice a Physicians are hypo-competent a Physician education often ignored or deficient at medical student and resident level a Examples of effective education exist a Effective education uses: effective educational methodologies which includes 2 experiential methods, same specialty role models, and reinforcement

Example: Professionalism
a Generally agreed upon components of professionalism include: altruism, respect, cross-cultural sensitivity, accountability, confidentiality, communication and shared-decision making, integrity, compassion / empathy, duty, competence, recognizing and managing conflicts of interest, self-awareness, and commitment to excellence and ongoing professional development a Frequent lapses in professionalism have been documented in residents, faculty, and practicing physicians. There is some evidence of erosion of professionalism during training. a Explicit teaching and assessment of professionalism is uncommon in medical schools and residency programs, although exceptions exist. a Patient and societal trust of physicians is related to their perceptions and experiences of physicians professionalism.

PI AND GNA: OBTAINING THE INFORMATION

aReview of Available Information
`Published Literature: PubMed, ERIC `Accreditation Bodies: AAMC, LCME, ACGME `Professional Societies: IOM `BEME `Government Databases and Reports

aUse of Consultants / Experts aCollection of New Information

Example: Musculoskeletal Medicine

a MS disorders common and major cause of disability a Patients desire quick access a Training increases PMD confidence and referrals a FCIM, ACGME, COGME recommend training in physical examination, diagnosis, and management of common MS disorders (including joint aspiration and injection, when appropriate) a Preferred training modality is supervised clinical practice in settings similar to those in which the trainees will eventually practice, i.e. in primary care settings with trained preceptors possessing the desired expertise.

Example: Resident Teaching Skills

(Houston TK et. al. A primary care musculoskeletal clinic for residents: success and sustainability. J Gen Intern Med 2004;19:524-529.

(Morrison EH et.al. The effect of a 13-hour curriculum to improve residents teaching skills: a randomized trial. Ann Intern Med 2004;141:257-263.

a Recognizing the crucial roles that resident physician teachers fulfill in medical education, the LCME (1) and other professional bodies (2) are calling upon residency training programs to ensure residents competence as clinical teachers. a More residency programs today are offering teaching skills training to their housestaff (3,4), but the evidence for how this training should be accomplished is limited. . ..

Example: Smoking Cessation (1)

(Cornuz J et.al. Efficacy of resident training in smoking cessation: a randomized control trial.. Ann Intern Med. 2002;136:429437.

Example: Smoking Cessation (2)

(Cornuz J et.al. Efficacy of resident training in smoking cessation: a randomized control trial.. Ann Intern Med. 2002;136:429437.

a Medical advice and pharmacotherpay are effective interventions in clinical practice to help patients stop smoking (1,2). a Although primary care physicians can play a key role in promoting smoking cessation to their patients who smoke (3), they miss many opportunities to advise smokers (4-7), mainly because they lack skills in counseling about smoking cessation (8). a Residency training in ambulatory care is an ideal setting in which to learn the attitudes and skills of preventive medicine, including smoking cessation (9).

aTraining programs in smoking cessation improve the

frequency and quality of smoking cessation interventions administered by physicians (10-17). . .

a Most training programs mainly use didactic teaching rather than such potentially effective methods as active learning of practical skills (20,21). . a A few training programs based on active learning methods effectively improve counseling skills, selfefficacy, and attitudes (22-25), but their effect on rates of smoking cessation remains unknown. . .

TARGETED NEEDS ASSESSMENT:

DEFINITION

STEP 2: TARGETED NEEDS ASSESSMENT

aA needs assessment of ones `Targeted learners `Targeted learning environment

whose needs may differ from the needs of learners and learning environments in general.

refining the foundation

IMPORTANCE
a Identifies the specific needs and preferences of targeted learners and other stakeholders, which may be different from learners and stakeholders in general. a Assesses the environment (including the hidden and informal curriculum) which will likely influence behavioral / performance outcomes. a Permits tailoring the educational intervention to specific needs. a Increases efficiency, prevents duplication. a Builds relationship with stakeholders. a Aligns resources with strategy

TARGETED NEEDS ASSESSMENT: THREE STEPS

1. Choose your targeted learners. 2. Find out information about the targeted learners. 3. Determine characteristics of the learning environment.

INFORMATION ABOUT TARGETED LEARNERS

a Previous training & experience a Already planned training & experience a Existing proficiencies: knowledge / attitudes / skills a Current performance / behaviors a Perceived deficiencies and learning needs a Preferences

INFORMATION ABOUT TARGETED ENVIRONMENT

a Related existing curricula a Hidden / informal curriculum a Specific enabling and reinforcing factors / barriers a Resources a Stakeholders a Politics / factors related to institutional administration, policy and procedure

CONSIDERATIONS IN COLLECTING INFORMATION

aAre data sources already available? aHow many resources should be devoted to this process? aWhat are long term plans for using the information that is gathered? (Need for IRB Approval?)

METHODS FOR COLLECTING INFORMATION

a Inventory of existing training a Informal discussion a Formal interviews a Focus groups a Questionnaires a Observation a Tests a Audits a Strategic Planning Sessions

EXAMPLE: ETHICS CURRICULUM

FOR IM PGY-2 AND 3 RESIDENTS
Methods for Collecting Information: a Inventory of previous curricula to which the residents had been exposed in the residency a Informal interviews with several residents a Survey of all targeted residents about their previous training, perceived competencies, and perceived needs.

EXAMPLE: ETHICS CURRICULUM

FOR IM PGY-2 AND 3 RESIDENTS
Findings: a Residents had considerable training related to autonomy, beneficence, substituted judgment, advance directives, and end-of-life decisions a Residents had no training related to clinical decision making in the context of competing interests such as patient vs. family vs. societal needs, or to reimbursement structures for ambulatory healthcare. a All of the residents' training had centered around inpatient cases.

EXAMPLE: ETHICS CURRICULUM

FOR IM PGY-2 AND 3 RESIDENTS
Response: a The curriculum developers decided to focus their curriculum on clinical decision making, with an emphasis on the ambulatory setting.

EXAMPLE:

MUSCULOSKELETAL CURRICULUM
Methods for Collecting Information: a Review of existing training a Senior resident exit interview a Focus group of residents at noon conference a Survey of current residents

EXAMPLE:
Findings:

MUSCULOSKELETAL CURRICULUM

EXAMPLE:

MUSCULOSKELETAL CURRICULUM
Response: a Development of workshops and syllabus materials on diagnosis and management, including injection therapy, for the musculoskeletal disorders commonly presenting to primary care practices. a Institution of a new primary care musculoskeletal clinic supervised by Internal Medicine preceptors with a special interest in musculoskeletal medicine, to which other primary care practitioners referred patients for diagnosis and injection. The clinic enabled residents to gain supervised experience in diagnosing and managing common musculoskeletal disorders, such as back, shoulder, elbow, hip, knee and foot pain, to perform procedures, and to see role models in action.
Houston TK, Connors RL, Cutler N, Nidiry MA.A primary care musculoskeletal clinic for residents: success and sustainability. J Gen Intern Med. 2004 May;19(5 Pt 2):524-9.

a Wrong case mix in Rheumatology and Orthopedics a Low self-rated proficiency a Low levels of training and clinical experience a Strong desire for training a Preferred educational method was direct supervision of patient care by primary care practitioners with expertise in MS medicine.

EXAMPLE:

PALLIATIVE CARE CURRICULUM

Methods for Collecting Information: a Focus group of trainees (residents) a Focus group of nurses a Informal interviews with selected faculty.

EXAMPLE:
Findings:

PALLIATIVE CARE CURRICULUM

a While the trainees conceptually understood the selection and dosing of opioid analgesics, there was resistance in prescribing them, especially in high doses or to those patients with a history of substance abuse disorders. a The resistance was prevalent among nursing and attending staff as well.

EXAMPLE:
Response:

PALLIATIVE CARE CURRICULUM

At the conclusion of Steps 1 & 2:

a You have a strong argument for the need for your curriculum. a Set the stage for generalizability and dissemination of your curriculum. a Understand the particular needs of your targeted learners and institution(s) a Identified potential resources and support. a Have the introduction and elements of a discussion for a manuscript a You are now the expert!

a One of the objectives of the curriculum became addressing attitudinal barriers in prescribing opiates. a Nurses and attendings became targeted learners as well.

QUESTIONS?

STEP 3: GOALS & OBJECTIVES focusing the curriculum the reason for teaching

Glassick Criteria
1. Clear goals and aims 2. Adequate preparation 3. Appropriate Methods 4. Significant Results 5. Effective Dissemination 6. Reflective Critique

GOALS
aGoals are broad educational objectives, the general ends toward which an effort is directed. They are usually not measurable as written. aExample: The purpose of the musculoskeletal curriculum is to prepare residents to evaluate and manage musculoskeletal conditions commonly seen in General Internal Medicine practice.

OBJECTIVES
a Objectives are specific & measurable. a Examples: By the end of the curriculum, residents will demonstrate their ability to:
`Correctly label shoulder anatomy on a diagram. `List the 4 most common causes of shoulder pain. `Perform an appropriate physical examination of the shoulder. `Diagnose the 4 most common causes of shoulder pain, based on history and physical examination. `Appropriately manage these 4 conditions. `Appropriately perform subacromial and intra-articular injections.

GOALS VS. OBJECTIVES

IMPORTANCE OF OBJECTIVES
aHelp prioritize aDirect content aIdentify learning methods (congruity) aEnable and direct evaluation aPermit clear communication to learners, faculty, and other stakeholders aRequired by ACGME/ LCME

aGoals: visionary, lofty, expansive aObjectives: precise, measurable

TYPES OF OBJECTIVES
a Learner Objectives
cognitive affective psychomotor (skill/competence vs behavior/performance)

LEVELS OF OBJECTIVES aIndividual Learner aAggregate or Program

a Process Objectives
curriculum implementation measures

a Patient / Healthcare Outcome Objectives

effects beyond those delineated in learner and process objectives, e.g. patient outcomes, career choice

HOW TO WRITE OBJECTIVES

1. 2. 3. 4. 5. Who will do how much / how well of what by when?

HOW TO WRITE OBJECTIVES

By the end of the gynecology curriculum (BY WHEN), each IM resident (WHO) will have demonstrated (WILL DO), at least once (HOW MUCH), the appropriate technique (HOW WELL), as defined on a check sheet, for obtaining a Pap smear and cervical cultures (OF WHAT).

10

HOW TO WRITE OBJECTIVES (Table 4.1)

Words Open to More Interpretations
a know or understand

HOW TO WRITE OBJECTIVES

aAchieve balance between specificity and readability aHave someone else read them and explain them to you aHave a manageable number of objectives

Words Open to Fewer Interpretations

a list, recite, present, define, describe, give an example of a demonstrate (as measured by) a be able a use, incorporate a know how, internalize a rate as valuable a appreciate a grasp the significance of a rank as important a identify or rate as an opinion a believe a learn

a (use one of the above terms)

EXAMPLE: PROFESSIONALISM
a Poor: Residents will be able to obtain informed consent. a Better: By the end of PGY-2, residents will routinely obtain informed consent that includes the following critical elements:
`natural course without treatment `alternative treatments `risks and benefits of the alternative treatments `assessment of patient understanding `sensitivity to patient needs and preferences `answering of patient questions

EXAMPLE: Communication Skills

a Cognitive Objective: By the end of the rotation, residents
will be able to list the critical components of effective patient education: assessing patients knowledge, beliefs, needs; tailoring education to needs; giving information clearly and effectively; checking patients comprehension and agreement.

a Affective Objective: By the end of the rotation, residents

will believe that it is a physicians role to effectively educate patients.

a Psychomotor Objective: By the end of the rotation,

residents will have demonstrated their proficiency in the above patient education skills. By the end of residency, patient surveys will reveal the implementation of of these skills in practice.

EXAMPLE: Communication Skills

a Process Objective: By the end of the rotation, each
resident will have reviewed 3 videotapes of their actual patient interactions with their colleagues and a facilitator.

REMEMBER
aGoals provide overall direction aA manageable number of objectives should
`interpret the goals `focus and prioritize curricular components

a Patient / Healthcare Outcome Objective: Two months

after the end of the rotation, patients of trained residents will be more satisfied with their physicians and be more compliant with their prescribed medication regimen than patients of untrained residents.

aMost curricula encompass more than the sum of their written objectives aObjective can be written to encourage creativity, flexibility, and nonspecified learning relevant to curricular goals

11

Example: Goal & Objective for a Ward Rotation

aGoal: Students and residents will become self-directed learners. aObjective that encourages creativity,flexibility, and

nonspecified learning:
`Each week during a ward rotation, each student and each resident will identify a question relevant to the care of one of their patients, and briefly report during morning rounds the sources used, the search time required, and the answer to their question.

QUESTIONS?

SMALL GROUP EXERCISE #1

a Step 1: Problem Identification & General Needs Assessment a Step 2: Targeted Needs Assessment

STEP 4: EDUCATIONAL STRATEGIES accomplishing educational objectives

aStep 3: Goals & Objectives

Glassick Criteria
1. Clear goals and aims 2. Adequate preparation 3. Appropriate Methods 4. Significant Results 5. Effective Dissemination 6. Reflective Critique

12

STEP 4: EDUCATIONAL STRATEGIES

EDUCATIONAL METHODS Education is not the filling of a pail, but the lighting of a fire.
William Butler Yeats

aContent of the Curriculum aEducational Methods

EDUCATIONAL METHODS: GENERAL GUIDELINES

aMaintain congruence between objectives and methods aUsed multiple educational methods aChoose educational methods that are feasible aRemember that assessment can drive learning (internalization of assessment criteria)

Congruence: Educational Methods for Achieving Cognitive Objectives aReading aLecture aAudio-visual Materials aDiscussion aInteractive Programmed Learning

Congruence: Educational Methods for Achieving Affective Objectives aExposure (readings, discussions, experiences) aFacilitation of openness, introspection, & reflection aRole models

Congruence: Educational Methods for Achieving Psychomotor Objectives

aSkill or Competency Objectives
`Supervised clinical experience `Simulations `Audio or visual review of skills

aBehavioral or Performance Objectives

`Removal of barriers to performance `Provision of resources that facilitate performance `Provision of reinforcements for performance

13

NEW CHALLENGES
What are the educational methods that will foster the attainment of the ACGME competencies? `Professionalism `Practice-based learning and improvement `Systems-based practice

Educational Methods for Achieving PBLI

a Training in skills related to self-directed learning
`Self assessment, including audits of own care `Information searching `Critical appraisal `Clinical decision-making

a Independent learning projects a Personal learning plans or contracts a Use of learning portfolios a Encouraging learners to formulate and answer their own questions a Role modeling

Educational Methods for SystemsBased Practice / Teamwork

a Providing feedback on costs of care a Case conferences focused on cost effectiveness and quality of care a Training in team skills a Collaborative learning experiences a Opportunities to work in disease management programs a Participation in quality improvement and safety teams a Redesigning teaching services towards multidisciplinary integration / work environments that model effective teamwork a Focused curricula on team functioning and related skills a Reflection on / assessment of team function

Educational Methods for Promoting Professionalism

a Faculty role modeling a Facilitated reflection on experience a Participation in writing professionalism goals a Peer evaluations a Participation in patient advocacy groups a Service learning and volunteerism a Attention of institutional and program leaders to the policies and culture of the training institution.

EXAMPLE: Musculoskeletal

EXAMPLE: Primary Care GYN

a By the end of the gynecology curriculum, each resident will have demonstrated, at least once, the appropriate technique, as defined on a check sheet, for obtaining a Pap smear and cervical cultures. ` Lecture / demonstration on proper communication / procedural skills ` Practice with genitourinary teaching associate (GTA) ` Observed pelvic exams in resident clinic a By the end of residency, patient surveys will reveal the implementation of these procedures in their continuity clinic. ` Pap exam trays in clinic ` Provision of trained MAs to assist in clinic `Provision of audit feedback to residents on # of the patients they have seen eligible for Pap / Culture, # of these who have received the tests, and location of testing.

(Houston TK et.al. A primary care musculoskeletal clinic for residents: success and sustainability. J Gen Intern Med. 2004 May;19(5 Pt 2): 524-9.)

Curriculum

a By the end of the curriculum, residents will be able to perform subacromial and intra-articular corticosteroid injections of the shoulder, using proper technique. `Didactic discussion, with demonstration `Supervised practice with simulated models `Supervised practice with real patients in a specially designed musculoskeletal clinic

14

EXAMPLE: Informed Consent

a By the end of PGY-2, residents will routinely obtain informed consent that includes the previously listed critical elements:
`Didactic / handout on components of informed consent, and relevant information on 10 most common procedures. `Demonstration by role model physicians. `Supervised practice, with explicit reflection and feedback. `Feedback of patient survey information and nurse evaluations. `Faculty and nurse development that addresses the informal and hidden curriculum. `General and procedure specific forms that enable informed consent.

True teaching is not an accumulation of knowledge; it is an awakening of consciousness which goes through successive stages.

from a temple wall inside an Egyptian pyramid

Education is what survives when what has been learned has been forgotten.

QUESTIONS?

-B.F. Skinner

QUESTIONS?

STEP 6: EVALUATION AND FEEDBACK

assessing the achievement of objectives and stimulating continuous improvement.

15

Glassick Criteria
1. Clear goals and aims 2. Adequate preparation 3. Appropriate Methods 4. Significant Results 5. Effective Dissemination 6. Reflective Critique

EVALUATION AND FEEDBACK: WHY?

a To determine if goals and objectives met a To provide information for improvement a To assess individual achievement a To satisfy external requirements (e.g., ACGME) a To document accomplishments of curriculum developers a To maintain and garner support a To serve as a basis for presentations/publications

ACGME REQUIREMENTS (U.S.)

Resident Evaluation a Evaluate the 6 competencies
patient care, medical knowledge, practice-based learning, interpersonal & communication skills, professionalism, systems based practice

a Include observation a Include humanistic qualities in each observation a Include chart audit a Evaluate in writing a Maintain permanent records

ACGME REQUIREMENTS (U.S.)

Resident Evaluation (continued) a Provide regular and timely feedback a Have a process in place to achieve progressive improvements in residents competence & performance

ACGME REQUIREMENTS (U.S.)

Program Evaluation a Program and faculty members should be evaluated by residents a There should be a process in place to review and improve the program that includes review of: evaluations by residents; aggregate evaluations of residents; patient mix; quality of supervision

16

LCME: FUNCTIONS AND STRUCTURE OF A MEDICAL SCHOOL (U.S.)

The medical school faculty must establish principles and methods for the evaluation of student achievement, and make decisions regarding promotion and graduation. The evaluation of student achievement must employ a variety of measures of knowledge, competence and performance, systematically and sequentially applied throughout medical school. Each accredited program must utilize methods for determining the quality of its program and the level of achievement of its students compared to national norms.

THE 10 TASKS OF EVALUATION

I. Identify Users II. Identify Uses III. Identify Resources IV. Identify Evaluation Questions* V. Choose Evaluation Designs* VI. Choose Measurement Methods* and Construct Instruments VII. Address Ethical Concerns VIII. Collect Data IX. Analyze Data X. Report Results

IV. IDENTIFY EVALUATION QUESTIONS

a Ensure that some evaluation questions are congruent with learner objectives. a Include some evaluation questions that do not relate to specific learner objectives (program evaluation). a Include some that are open-ended in nature. a Prioritize and select key evaluation questions, based upon user needs and feasibility.

EXAMPLE: MED-PSYCH
a Do residents communication skills improve following training? Are they superior to those of untrained residents? a How do residents rate the curriculum and its various components? a What are its strengths? a How can it be improved?

V. CHOOSE EVALUATION DESIGNS

a Choose an evaluation design congruent with the evaluation question. a Choose an evaluation design that is feasible in terms of resources. a Consider internal validity : history, maturation, testing, instrumentation, selection bias, dropout bias, statistical regression. a Consider external validity or generalizabilty to other populations and settings.

V. COMMON EVALUATION DESIGNS

a Posttest Only a Pretest Posttest a Control Group X ---- O O1---- X ---- O2 E (O1---) X ---- O2 C (O1---)------- O2

(R)

17

CONGRUITY EXAMPLE: MED-PSYCH

a Do residents communication skills improve following training? O1---- X ---- O2 a Are they superior to those of untrained residents? E X ---- O2 R C ------- O2

CONGRUITY EXAMPLE: MED-PSYCH

a How do residents rate the curriculum and its various components? a What are its strengths? a How can it be improved? X ---- O

VI. CHOOSE MEASUREMENT METHODS AND CONSTRUCT INSTRUMENTS

a Global rating forms a Self-assessment forms a Direct observation a Written or computerized tests a Oral Examinations a Performance audits a Patient Questionnaires a Learner Questionnaires a Individual interviews a Group interviews / discussions a Essays
(see Table 7.3)

VI. CHOOSE MEASUREMENT METHODS AND CONSTRUCT INSTRUMENTS

aChoose a measurement method that is congruent with the evaluation question.

`i.e. written tests for knowledge objectives; observation session for skill objectives

aChoose a measurement method that is feasible in terms of available resources.

VI: Choose Measurement Methods: Congruence

Knowledge Learner Oral exam Written exam/Qaire Case discussion Global rating scales Attitude Skill/Performance

EXAMPLE: MED-PSYCH
a Do residents communication skills improve following training?
`Pre-post self-assessments `Faculty observation during experiential learning exercises `consensus global rating form

Learner interview Direct observation Questionnaire Audio/video Self-evaluation observation Global rating scales Record audit Outcomes of care Self-evaluation Patient interview Global rating scales Aggregated scores Aggregated scores from above methods from above methods

a Are they superior to those of untrained residents?

`Randomized controlled trial using a rigorously evaluated videotape of a resident-standardized patient interaction.

Program

Aggregated scores from above methods

a How do residents rate the curriculum and its various components? / What are its strengths? / How can it be improved?
`End-of-rotation questionnaire

18

EXAMPLE:

MUSCULOSKELETAL
a Can residents perform an appropriate physical examination of the shoulder? perform subacromial and intra-articular injections using proper technique? `Observed physical examinations, preceptor checklists. `Observed shoulder injections, preceptor checklists. `OSCEs a Can residents appropriately diagnose and manage the 4 most common causes of shoulder pain? `Closed and open-ended case-based test. `OSCEs a How do residents rate the curriculum and its various components? / What are its strengths? / How can it be improved? `End-of-rotation questionnaire

EXAMPLE:

INFORMED CONSENT
a Do residents know the essential components of informed consent? `Written exam. `Demonstration of inclusion of components in observed patient interactions. a Are residents capable of obtaining informed consent that includes the essential components for the 5 most common procedures and for an unfamiliar procedure? `Supervised observation and documentation, with rater using checklist; or `Rater evaluation of audio or videotaped patient interactions. a Do residents routinely include the essential component of informed consent in practice? `Nurse survey form. `Patient survey form.

TAKE HOME MESSAGE: CONGRUENCY

OBJECTIVES

QUESTIONS?
EVALUATION METHODS

EDUCATIONAL METHODS

STEP 5: IMPLEMENTATION
aIdentify Resources

STEP 5: IMPLEMENTATION
making the curriculum a reality converting a good plan into an accomplishment.

aObtain Support (Institutional, External) aDevelop Administrative Mechanisms to Support the Curriculum aAnticipate and Address Barriers aHave a Plan for Introducing the Curriculum

19

IDENTIFY RESOURCES
a Personnel: faculty, secretarial, and other support staff; patients; other a Time: faculty, support staff, learners a Facilities: space, equipment, clinical sites a Funding/Costs: direct financial costs, release time, hidden or opportunity costs

OBTAIN SUPPORT
a Internal
`From: administrators (dean, department chair, program director, division director, hospital administrator), faculty, learners, other stakeholders `For: personnel, resources, political support

a External
`From: government, donors and philanthropic organizations, accrediting bodies, professional societies `For: funding, requirements, political support, curricular or faculty development resources

DEVELOP ADMINISTRATIVE MECHANISMS

a Administrative Structure: delineate responsibilities
and decision making

ANTICIPATE AND ADDRESS BARRIERS

aFinancial and Other Resources aCompeting Demands aPeople: attitudes, job/role security, power and authority

a Communication:
`Content: rationale; goals & objectives; information about the curriculum, schedules, learners, faculty, facilities and equipment; changes; evaluation results; etc. `Mechanisms: memos, meetings, syllabus, website, site visits, reports, news articles, etc.

a Operations: preparation and distribution of materials;

collection, collation, and distribution of evaluation data, curricular revisions and changes; etc.

PLAN FOR INTRODUCING THE CURRICULUM

aPilot aPhase-In aFull Implementation

EXAMPLE: MUSCULOSKELETAL
a Establishment of a musculoskeletal clinic that provided a concentration of cases that represented the epidemiology of MS problems in primary care practice.
`3 of 4 curriculum developers worked in CBP `Financial analysis and pilot demonstrated feasibility `Administrative support was obtained `PI, GNA, TNA provided evidence for need `Advance communication with Rheumatology and Orthopedics `Evaluation that demonstrated success & popularity

20

EXAMPLE: MED-PSYCH
a External funding for GIM Residency that required CS and PS training a Started with external experts a Faculty development / co-teaching a Phase-in with interested residents a Documented resident satisfaction a Presented evaluations and examples to department chair. a Expanded to entire residency program

QUESTIONS?

SMALL GROUP EXERCISE #2

aStep 4: Educational Strategies aStep 6: Evaluation aStep 5: Implementation

CURRICULUM DEVELOPMENT : OVERVIEW

a 1. a 2. a 3. a 4. a 5. a 6. Problem ID & Genl Needs Assessment Needs Assessment of Targeted Learners Goals & Objectives Educational Strategies Implementation Evaluation & Feedback

a 7. Curriculum Maintenance & Enhancement a 8. Dissemination

LONGITUDINAL PROGRAM IN CURRICULUM DEVELOPMENT

a 9 1/2 Months a Workshops on Each Curricular Step a Sessions on Literature Searching, IRB, Searching for Funding, Simulation Center, Dissemination a Mentored Project a Individual Meetings with Facilitators, Written Feedback on Each Step a Work-in-Progress Sessions a Written Paper / Curriculum and Oral Presentation

REFERENCES
a Kern DE, Thomas PA, Howard DM, Bass EB. Curriculum Development for Medical Education: A Six-Step Approach. Baltimore (MD): Johns Hopkins University Press; 1998. / Kern DE, Thomas PA, Hughes MT, eds. Curriculum Development for Medical Education: A Six-Step Approach. 2nd ed. Baltimore (MD): Johns Hopkins University Press; 2009. In Press. a Thomas PA, Kern DE. Internet resources for curriculum development in medical education: an annotated bibliography. J Gen Intern Med. 2004; 19(5): 598-604. a ACGME Outcome Project: http://www.acgme.org/Outcome/ a ACGME: http://www.acgme.org/ a AAMC/LCME: http://www.lcme.org/ a Kern DE, Branch WT, Green ML, et.al. Making it count twice: how to get curricular work published. SGIM Workshop, 2005. Available at http://www.sgim.org/userfiles/file/AMHandouts/AM04/Workshops/WB0 9.pdf . a David E. Kern, MD, MPH dkern1@jhmi.edu

21

NEXT STEPS:
aReflect on the main things you have learned from the CD Workshop. aThink concretely about how you can apply them to the work you envision. aArticulate next steps that will ensure that you apply what you have learned to something that is meaningful to you.

SUMMARY

TAKE HOME MESSAGE

OBJECTIVES EDUCATIONAL METHODS EVALUATION METHODS

CURRICULUM DEVELOPMENT : OVERVIEW

a 1. a 2. a 3. a 4. a 5. a 6. Problem ID & Genl Needs Assessment Needs Assessment of Targeted Learners Goals & Objectives Educational Strategies Implementation Evaluation & Feedback

LONGITUDINAL PROGRAM IN CURRICULUM DEVELOPMENT

a 9 1/2 Months a Mentored Project a Workshops on Each Curricular Step a Individual Meetings with Facilitators, Written Feedback on Each Step a Work-in-Progress Sessions a Written Paper / Curriculum and Oral Presentation a Sessions on Literature Searching, Survey Design, IRB, Searching for Funding, Dissemination

a 7. Curriculum Maintenance & Enhancement a 8. Dissemination

22

Glassick* Criteria for Scholarship

1. 2. 3. 4. 5. 6. Clear goals and aims Adequate preparation Appropriate methods Significant results Effective dissemination Reflective critique

DISSEMINATION / PUBLICATION
making it count twice

*Scholarship Assessed:Evaluation of the Professoriate,1997

DISSEMINATION
aPublication in Peer Reviewed Journals aElectronic Publication aPresentation
`Local `Regional Professional Meetings `National and International Professional Meetings

Needs Assessments
a Important information a New information or systematic review of existing data a Methodologically sound a Generalizable information
e.g. Ratanawongsa N, Bolen S, Howell EE, et.al. Residents perceptions of professionalism in training and practice: barriers, promoters, and duty hour requirements. J Gen Intern Med. 2006;21(7):758-763. e.g. Clark JM, Houston TK, Kolodner K, et.al. Teaching the teachers: a national survey of faculty development in departments of medicine of U.S. teaching hospitals. J Gen Intern Med 2004;19:205-214.

Goals and Objectives

a Topic of demonstrated importance a Well described, methodical, sound process of developing the goals and objectives a A broad consensus supports the goals and objectives. a Professional organizations, authoritative bodies support the goals and objectives. a E.g., CDIM/SGIM Core Medicine Clerkship, FCIM

Educational Strategies
a Topic of demonstrated importance a Educational strategies innovative and add to the existing literature a Educational strategies are adaptable to other settings. a Evaluation provides evidence regarding efficacy (If very innovative and topical, less rigorous evaluation may be acceptable.)
e.g. Houston TK, Connors RL, Cutler N, Nidiry MA. A primary care musculoskeletal clinic for residents: success and sustainability. J Gen Intern Med. 2004 May;19(5 Pt 2): 524-9. e.g. Branch WT Jr., Kern DE, Gracey K, Haidet P, Weissmann P, Mitchell G, Inui T, Novak TL. Teaching the human dimensions of care in clinical settings. JAMA. 2001;286:1067-1074

23

Evaluations of Educational Interventions (1)

a Important topic a Generalizable sample: multi-institutional a Strong evaluation design: sufficient numbers / power, control group, long-term f/u, blinded raters a Important outcomes: clinical outcomes > behaviors/performance > skills,attitudes > knowledge > satisfaction a Strong assessment methods: objective, reliability, content and other measures of validity a Appropriate data analysis: when appropriate, account for confounding variables with multivariate analysis a Evaluation strategy innovative a Contribution to the existing literature

Evaluations of Educational Interventions (2)

e.g. Roter DL, Hall JA, Kern DE, Barker LR, Cole KA, Roca RP. Improving physicians interviewing skills and reducing patients emotional distress: a randomized clinical trial. Arch Intern Med 1995; 155: 1877-1884. e.g. Smith RC, Lyeles JS, Mettler J, et al. The effectiveness of intensive training for residents in interviewing: a randomized, controlled trial. Ann Intern Med. 1998;128:118-126. e.g.Watkins RS, Moran WP. Competency-based learning: the impact of targeted resident education and feedback on Pap smear adequacy rates. J Gen Intern Med 2004;19:545-548. e.g. Morrison EH, Rucker L, Boker JR, et. al. The effect of a 13-hour curriculum to improve residents teaching skills: a randomized trial Ann Intern med 2004;141:257263. e.g. Cornuz J, Humair JP, Seematter L, et. Al. Efficacy of resident training in smoking cessation: a randomized control trial of a program based on application of behavioral theory and practice with standardized patients. Ann Intern Med 2002:429-437. e.g. Sisson SD, Hughes MT, Levine D, Brancati FL. Effect of an internet-based curriculum on postgraduate education: a multicenter intervention. J Gen Intern Med 2004;19:505-509. e.g. . Windish DM, Gozu A, Bass EB, et.al. A ten-month program in curriculum development for medical educators: 16 years of experience. J Gen Intern Med. 2007;22:655-61. / Gozu A, Windish DM, Knight AM, et.al. Long-term follow-up of a ten-month programme in curriculum development: a cohort study. Med Educ. 2008;42:684-692.

NEXT STEPS:
aReflect on the main things you have learned from the CD Workshop. aThink concretely about how you can apply them to the work you envision. aArticulate next steps that will ensure that you apply what you have learned to something that is meaningful to you.

REFERENCES
a Kern DE, Thomas PA, Howard DM, Bass EB. Curriculum Development for Medical Education: A Six-Step Approach. Baltimore (MD): Johns Hopkins University Press; 1998. / Kern DE, Thomas PA, Hughes MT, eds. Curriculum Development for Medical Education: A Six-Step Approach. 2nd ed. Baltimore (MD): Johns Hopkins University Press; 2009. In Press. a Thomas PA, Kern DE. Internet resources for curriculum development in medical education: an annotated bibliography. J Gen Intern Med. 2004; 19(5): 598-604. a ACGME Outcome Project: http://www.acgme.org/Outcome/ a ACGME: http://www.acgme.org/ a AAMC/LCME: http://www.lcme.org/ a Kern DE, Branch WT, Green ML, et.al. Making it count twice: how to get curricular work published. SGIM Workshop, 2005. Available at http://www.sgim.org/userfiles/file/AMHandouts/AM04/Workshops/WB0 9.pdf . a David E. Kern, MD, MPH dkern1@jhmi.edu

THE END

THANK YOU

24