Innovative methods of assessment and their impact on learning

ASSESSMENT AND LEARNING

Assessment


Formative
Assessment used as a learning tool: students realize what they know or dont know through answering questions

Summative
Assessment used as a student learning evaluation tool and is typically used to assign students a course grade

Assessment has an effect on how teachers teach and how students learn (Challis, Houston, and Stirling 2006).  It also implies that assessment can be considered as a learning activity in its own right (Fullan, Hill, and Crvola 2006).


Summative assessment


Assessment must be matched to the desirable learning outcomes

Student learning needs

Suggested approach to course planning

Aims/ desired learning outcomes

alignment

Learning activities

Feedback

Content/ fundamental concepts

Assessment

Actual learning outcomes

The new learning objectives



New: Understanding of concepts, PLUS

application of knowledge, preparation for future work, acquisition of learning skills that enable lifelong learning, better learning attitudes

Learning Processes Products

Knowledge
Critical thinking Reflection Group work

True understanding Application of concepts

Skills Attitudes

Decisionmaking/ diagnosis

Confidence Motivation

COMMON CHALLENGES IN ASSESSMENT

Multiple-choice questions
Assessment Multiplechoice questions. Assessed Memory for facts. Understanding of ideas, theories. Application of principles. Analytic thinking. Advantages Fast marking. Reliable marking. Broad coverage of syllabus. More feedback to students. Disadvantages Difficult to prepare without faults. Cannot assess skills of organising, originality or ability to solve illdefined problems. Do nothing to develop writing skills or change conceptions of learning.

Short-answer written questions

Assessment Short-answer written questions. Assessed Memory for facts. Understanding of ideas, theories. Advantages Broad coverage of syllabus. Fast marking. More reliable marking. More feedback to students. Disadvantages Limited opportunity to show argument or originality.

Traditional essay questions in exam

Assessment Traditional essay questions in examinations. Assessed Memory of facts. Understanding of ideas. Ability to organise material. Ability to develop an argument. Original thinking. Advantages Easy to set. Disadvantages Time consuming to mark. Marking unreliable. Poor coverage of syllabus. Favours fast and fluent writers. Limited feedback to students.

Learning Processes Products

Knowledge
Critical thinking Reflection Group work

Assessment strategies?

Skills Attitudes

Decisionmaking/ diagnosis

Confidence Motivation

So, what can be done?

Refining traditional assessment strategies  Innovative/ alternative assessment methods


INNOVATIVE METHODS (4 CUHK stories)

How?
A case is a story, often told as a sequence of events in a particular place. Students put themselves in the role of an actor in the situation

    

Teaching of main concepts Cases Group discussion Presentation & class teaching

Courses involved
Traditional w/o enhanced website Number of courses
UG undergraduate PG postgraduate

Pilot Refined case-based case-based

6 4 UG 2 PG

3 2 UG 1 PG

3 1 UG 2 PG

Why?
Processes

Learning Products

Knowledge
Problem-solving Searching for information Group work Presentation

Understanding of key concepts Application of concepts

Skills
Ready for real-life problems

Attitudes

Year 1 Undergraduate Course in Materials Science

A case example
CASE: Peter Lam's Puzzle in Organizing Data for his First Job Interview

Peter Lam received a phone call from David Chan, HR manager of Compass Technology Ltd. at 9:45am this morning, who requested him to attend a one-hour job interview at 2pm But he is also very nervous, particularly because David specifically asked him to prepare a ten-minute presentation on his knowledge on materials science and engineering

Marking scheme (presentation)

Needs Improveme nt Satisfactory Excellent Good

Understanding the issue Evidence and data Analysis Solution Presentation

Understanding the case clearly and identifying key issues Searching and quoting evidence and data that are appropriate and relevant Clear reasoning and critical evaluation of arguments for and against Well reasoned solution which is, by and large, practicable

Structure Logical presentation structure Clarity Clarity of voice. Visual aids Use of visual aids Reaction Responses to questions and feedback Punctuality Team work Timely completion of task Cooperation between group members

Marking scheme (report)

Needs Improvement Satisfactory Excellent Good

Focus of study

A clearly identifiable focus Clear research question(s) or hypothesis

20%

Evidence and data

Appropriate methodology Searching and quoting evidence and data that are appropriate Clear reasoning and critical evaluation of arguments Discussion enriching our understanding to the research question(s) or hypothesis 20%

Analysis

20%

Conclusion

Clear summary of points made Well reasoned solutions/ implications which are, by and large, practicable 15%

Language

Logical presentation structure Proficiency

15% 5% 5% Total 100

Reference Punctuality

Full and accurate references and referencing according to APA format Timely completion of task

Form for team mate participation

A. Overall Group Performance 1 this item is never or only rarely true of me 2 this item is sometimes true of me 3 this item is true of me about half the time 4 this item is frequently true of me 5 this item is always or almost always true of me 1. My team members in general prepare well before the          small group discussion. 2. I find that my team member does good presentation in          the small group discussion. 3. I find that my team member communicates effectively in          the small group discussion. 4. I find that my team member contributes in the small          group discussion. 5. Please grade the overall performance of your classmate          in the small group performance.

Learning (refined case-based, 2004-05) I learned more by going through the cases than what I could have learnt if the course had been conducted in a traditional lecture format
2004-2005 Student feedback on usefulness of case-based approach in learning
50 40 30 20 10 0 Strongly agree Agree No comment Students' answer Disagree Strongly Disagree

n UG = 28 n PG = 41

Pe rce ntage

Undergraduates

Postgraduates

Acquisition of skills The cases have helped me acquire the following skills which are important for my future career:
2003-2005 Students' feedback on sk ills acquisition

60 50 40 30 20 10 0

No of responses

Problem solving skill Skills to work efficiently in groups

A gr ee

co m m en t D is S ag tr on re gl e y D is ag re e

ag re e

Self learning skill Presentation skill

S tr on gl y

n (UG&PG) = 129

N o

Students' answers

Strengths
 

Case-based learning helps students gain better understanding of theories Students observed improvement in their skills including problem-solving skills, group-working skills, and self-learning skills and presentation skills Available data of exam scores suggests that students perform well in answering questions that require knowledge of fact and perform even better in questions that require higher analytical skill. Students had become more able to apply relevant material when answering analysis based questions.

Weaknesses


For some part-time postgraduates, they found case-based approach which demands group work time consuming Students require stronger learning support in terms of guidance, teaching support and communications with tutors Students required strong guidance it is important for the lecturer to disseminate more concrete instructions and basic knowledge before the start of the case project (maybe in the form of a briefing session), give tips and direction to students during the process, and give more comments to students after their presentations.

Extension: Case-based in School Science

Flow
Manufacturing process Form groups to learn different specialties: collective learning Peer-teaching Chemical properties Microstructure Mechanical properties

Form new groups (4 students) with different specialties Present a case problem

Tackle the problem

Why?
Processes

Learning Products

Knowledge
Self-learning Searching for information Presentation Group work

True understanding of concepts

Skills

Motivated to learn

Attitudes

The Case


Related to HK industry (e.g. electronic packaging)

If the PCB board fails, what are the causes? Failure analysis.

Failure of a chip

Students comments
      

Active, improve English skills. It can improve my presentation skill and cooperation. I could learn from others. It is very useful for future job. It improves my communication ability. Presentation skill practice, problem solving Its a happy learning method, knowledge can be built by students themselves.

Students comments
Students expressed their difficulties in selflearning:  Dont know where to begin  Dont know where to stop  Do not understand  Do not know how to study

3. Student grading


More information: Prof W.Y. Poon (Statistics) (wypoon@cuhk.edu.hk)

Overall course structure

     

Setting of marking criteria Performance Teacher evaluation Self-evaluation Peer-evaluation Calculation of final mark: judged by how close the self-evaluation is similar to the teacher and peer evaluations

Strategy 2: Self- and peer-assessments

Students must pay attention to peers work in order to assess themselves properly
More critical More serious

Teacher (T-scores) Peer (P-scores) Self (S-scores)

40 out of a total of 100 (40%) 30 out of a total of 100 (30%) 30 out of a total of 100 (30%)

Example: S-P scores comparison

Principle: higher S-P difference = lower S-P scores Criteria Peerassessment Selfassessm ent G4 1 5 3 8 10 (2+3+4)/3=3 (6+7+8)/3=7 (2+3+4)/3=3 (7+8+9)/3=8 (8+9+10)/3=9 Raw P-score = (3+7+3+8+9)/5= 6 |1-3| = 2 |5-7|=2 |3-3|=0 |8-8|=0 |10-9|=1 Raw Peer score The S-P difference

S-scores = S-P C1 and S-T scores C2

C3 C4 C5 Average

G1 2 6 2 7 8

G2 3 7 3 8 9

G3 4 8 4 9 10

S-P difference = (2+2+0+0+1)/ 5=1

Why?
Processes

Learning Products

Knowledge
Critical thinking Presentation and writing Ability to judge Reflection

Understanding of key concepts Application of concepts

Skills

Motivated to learn

Attitudes

Work done
on assessment strategies  About 20 cases completed, more ongoing  Website launched
 Seminars

The Case Studies



In courses offered by different programmes at different levels (year 1 to postgraduate)

Biochemistry Biology Chemistry Environment Science Food and Nutritional Science Mathematics Physics Statistics University General Education College General Education

 The Case Studies



Have been used to assess a variety of learning activities

Group presentation Individual presentation Group project written report Individual written assignment Laboratory experiment report Poster presentation In-class activities in student-oriented courses  Debate  Role-play  Portfolio of activities designed to explore a topic

Student feedback

Evaluation Examples of students positive comments

Quite interesting and innovative. Innovative, can allow students to identify and think about what really matters. This can force students to pay attention to the presentations of the others instead of daydreaming or chatting to each other. It allows students to be more aware of their performance towards their peers, such that the presentation is more interesting and interactive. It helps us to focus on the aspects other than the content itself, like the presentation skills and how the questions from audience are answered. Through this process, we could learn much more from each other.

Evaluation Examples of students suggestions or negative comments

As an undergraduate student, it is a bit difficult for us to assess others work. Students may not have enough knowledge to comment others group topic. A weighting can be assigned to each marking criterion to better reflect the importance of each criterion. There may be strong disagreement among group members in deciding the scores I think the assessment will be much fairer if it is carried out individually.

Website

How


Using blogs (instead of the purely paperbased format used before)

Reflections  Lesson plans  Taped teaching sessions  Review and comment on each other


Why?
Processes

Learning Products

Knowledge
Teaching skills Critical thinking Ability to judge Reflection

Application of concepts

Skills Attitudes

Motivated to learn

Students comments
Students found the new format useful  The new approach will be continued


ASSESSMENT STRATEGIES REVISITED

The strategies talked about

Case-based assessment  Peer-teaching  Student-made marking scheme  Peer-assessment  Self-assessment  Portfolio


Conclusion
Innovative assessment methods have a great potential in enhancing learning outcomes  Challenges were met but continuous refinement of the strategies should solve many of the problems


Your comments
Any comments about the abovementioned strategies?  Any other assessment strategies?


Thank you