Tackling the Challenges to Professional Education Project: Reforming Professional Education for a Knowledge-based Society

Using Criterion-Referenced Assessment to promote student learning Dr. Charles C. Chan, APSS, HKPU
seminar01102003/using CRA to promote student learning01102003.ppt

Assessing student learning outcomes

Who: teachers, instructors, peers and even students themselves can administer the assessment of learning outcomes. For whom: students (useful feedback for further improvement: formative evaluation), instructors (formative and summative evaluation), and school or university administration (summative and quantitative evaluation). What: subject and content specific, and/or generic transferable skills; performance at practicum. How: subjective grading, GPA, CRA: taxonomy of educational objectives (cognitive) and behavioral competency.
seminar01102003/using CRA to promote student learning01102003.ppt

Criterion-referenced assessment

Judges performance against a set of pre-determined criteria; Is independent of any other students results; Increase students intrinsic motivation and their sense of being responsible for their learning outcomes; Encourages the cooperation between students; Improves the effectiveness of teachers feedback to students.
seminar01102003/using CRA to promote student learning01102003.ppt

Comparison between NRA and CRA

Dimensions Assessment method Feedback Distribution Benefits NRA Compare with peers Summative Normal Easy to administer; Save time. CRA Mark with pre-set criteria Formative Skewed Clear feedback; Provide guidelines for further improvements; Increase students intrinsic motivation.

Limitations

Discourage cooperation Time-consuming; among students; Not widely used. Limits the effectiveness of feedback; Increase students anxiety.
seminar01102003/using CRA to promote student learning01102003.ppt

CRA: The SOLO taxonomy (Biggs & Collis, 1982)

SOLO level
Prestructural Unistructural Multistructural Relational

Characteristics of responses
Irrelevant information is included or the task is not tackled incorrectly One relevant information is included More than one independent information are presented Various relevant elements are integrated that an organized picture is presented The original coherent picture is generalized to a new and more abstract domain, reflecting a higher mode of processing
seminar01102003/using CRA to promote student learning01102003.ppt

Extended Abstract

CRA: Behavioral Competence Taxonomy

Professionalism Acquisition Assimilation
Show awareness towards major societal problems Self-management; personal growth Manage workload in an orderly manner

Knowledge
Recognize social welfare policies Understand social, cultural factors to specific client system Constructive use of available community resources Apply theories learnt with flexibility

Practice Competence
Observe and listen skillfully Specify goals of interventions Monitor and coordinate different work of the action plans Handle emergencies effectively

Adaptation

Performance Aspiration

Accept advice and criticism

Incorporate new approach to practice

Initiate to expand Recommend future ones knowledge base improvements

seminar01102003/using CRA to promote student learning01102003.ppt

Project I

Subject: Social Psychology (level 3) [01-02] Participants: 35 undergraduate students (yr1-3) CRA used: the SOLO taxonomy Details: Students are required to write three short papers with subject-specific scenarios and questions provided (one baseline, one mid-term and one final). The SOLO taxonomy is used to analyze students assignments and students are informed of their SOLO levels as feedback of their learning outcomes after each assessment.

seminar01102003/using CRA to promote student learning01102003.ppt

Project I Assignment sample

Short Paper I (500 words). You and one of your secondary school friends were successful to get into a degree program in the HK PolyU and the CityU / HKU respectively after one failure attempt. Right after both of you received the rejection letters, you met each other and found that the two of you had very different emotions. Q.1 Right after that failure attempt, recall what you and your friend came up with separately to account for that rejection. Q.2 Compare and contrast those reasons that were from you and from your friend and whether these reasons explained the different emotional states you were in. Q.3 By using theories-in-use, demonstrate to your friend that you understand why s/he thought the way s/he did. Q.4 Critically analyze why your friend agreed / disagreed with your perception of why s/he felt the way s/he did.
seminar01102003/using CRA to promote student learning01102003.ppt

Project I Record sheet of SOLO level

Name of student Paper 1 Paper 2 Paper 3

seminar01102003/using CRA to promote student learning01102003.ppt

Our Project I - Results

Student learning outcome:

2.54 (SOLO 1); 3.00 (SOLO 2); 3.25 (SOLO 3) Correlates with instructors judgment/subjective ratings. Undergraduates generally produce multistructural level papers. Students gradually improve during the semester.

seminar01102003/using CRA to promote student learning01102003.ppt

Our Project I

Instructors comments:

seminar01102003/using CRA to promote student learning01102003.ppt

Our Project II

Subject: Social Work Fieldwork I, II Participants: 103 Social work undergraduates (including both bachelor and diploma students); 40 fieldwork supervisors; CRA used: Behavioral Competence Taxonomy (selfdeveloped, based on Hauenstein model and APSS fieldwork manual); Details: fieldwork supervisors rated students performance in practicum with BCT (baseline, midterm and final).
seminar01102003/using CRA to promote student learning01102003.ppt

Our Project II Record Sheet

Please

identify the level of behavioral competence:

Student: _________________________ Supervisor: _______________________ Prof Knowledge PC Period under supervision: ___________________________________ Name of the agency: Level _______________________________________ Nature of the setting: Illustration _______________________________________ Nature of the major pre-selected piece of professional practice*: ______________________________________________ 1= acquisition; 2= assimilation; _______________________ Please check the following box to identify the nature of 3= adaptation; 4= performance; the major practice that requires the student to perform 5= aspiration during the time period under review (Choose either one): 1. Require execution of relatively tangible, routine and well-defined practice procedures 2. Require re-negotiation, re-definition and problem-solving skills in the execution of practice procedures Remark: __________________________ * may include planning and execution of a service program for adolescents, supportive counseling for pregnant teens etc.
seminar01102003/using CRA to promote student learning01102003.ppt

Our Project II - Results

Students improve over the semester

Baseline
Professionalism

Mid-term 2.79 2.55 2.84

Final 3.42 3.16 3.50

1.79 1.39 1.43

Knowledge

Practice Competence

seminar01102003/using CRA to promote student learning01102003.ppt

Our Project II - Results

Baseline:
Student is trying to acquire the knowledge of service and the organization.

Mid-term:
Shows potentiality in integrating her experience and practice. Asks more sensible questions and more responsive in supervision.

Final:
Student tried to apply social work values and principles into practice e.g. confidentiality of cases. Student was getting more confident in working with different client group.
seminar01102003/using CRA to promote student learning01102003.ppt

Our Project II - Supervisors comments

It takes quite some time to be familiar with the instruments. BCT offers more objective assessment than the traditional impression grading system. Its good to see students improve gradually over the semester. Sometimes it is difficult to decide which level students attained. Some placement setting (the nature) may limit the level of attainment a student can achieve.
seminar01102003/using CRA to promote student learning01102003.ppt

Summary and Conclusion

CRA benefits both students and teachers by providing valuable and objective feedback for continuous improvement in learning outcomes.

seminar01102003/using CRA to promote student learning01102003.ppt

References

Biggs, J. B. & Collis, K. F. (1982). Evaluating the quality of learning: the SOLO taxonomy (structure of the observed learning outcome). New York: Academic. Chalmers, D. & Fuller, R. (1996). Teaching for Learning at University: Theory and Practice. London, UK: Kogan Page Ltd. Chan, C.C., Tsui, M.S., Chan, M., & Hong, J. (2002). Applying the SOLO Taxonomy on studetns learning outcomesL an empirical study. Assessment & Evaluation in Higher Education, 27,6,511-528 Chan, C. C. & Cheng, G. H.-L. (2002). Using the SOLO Taxonomy to assess students written work. Submitted to Teaching of Psychology. Chan, C. C., Cheng, G. H.-L. and Chan, M. Y.-C. (2002, June). Assessing APSS Students Learning Outcomes and Learning Process. Teachers Manual version 2.1. (Available from the Network for Health and Welfare Studies, Department of Applied Social Sciences, the Hong Kong Polytechnic University)
seminar01102003/using CRA to promote student learning01102003.ppt