Some perspectives on

MCQ assessment

Stuart Palmer
Mary Dracup

Source: http://www.dreamstime.com/item.php?imageid=7299038 1
Assessment generally

“Examinations are formidable

even to the best prepared, for the
greatest fool may ask more than
the wisest man can answer.”

Charles Caleb Colton (1780 - 1832)

2
General thoughts

MCQs can be a useful element of an overall

assessment design
Can’t be abstracted from the context
No marks / diagnostic / formative / interactive
Few marks / semi-formative / assignment
High stakes / summative / examination
Design and evaluation approach should be
appropriate for the context

3
No need to look too far from home

Considine, J; Botti, M and Thomas, S. Design, format, validity and

reliability of multiple choice questions for use in nursing research and
education. Collegian, 2005 Jan; 12 (1): 19-24.

4
A proposed MCQ process

From: Considine, J; Botti, M and Thomas, S. Design, format, validity and reliability of multiple choice questions
for use in nursing research and education. Collegian, 2005 Jan; 12 (1): 19-24.

5
An alternative/pragmatic process
Design

Content/Face validity

Use

Construct validity

Reliability

Keep / Replace

6
Design – unit assessment plan

Assignment No. Assignment details Marks

1 Introduction to DSO 3%
2 Information literacy & the Library 5%
3 Referencing 10 %
4 Professional ethics report 15 %
5 Multi-choice test 1 6%
6 Multi-choice test 2 6%
7 Major report and presentation 25 %
Exam Must pass! 30 %

7
Design – MCQ test plan

Content Processes
Areas 1. Knowledge 2. Comprehension 3. Application %

A 3.75% 5.25% 6% 15
B 6.25% 8.75% 10% 25
C 3.75% 5.25% 6% 15
D 7.5% 10.5% 12% 30
E 3.75% 5.25% 6% 15
% 25 35 40 100

8
 Design – NCLEX-RN MCQ test plan

Source: 2010 NCLEX-RN® Detailed Test Plan

9
Face validity

A sub-set of content validity

Use an ‘expert panel’ to :

assess item clarity
assess item readability
assess consistency of style
check for errors

10
Construct validity

Key check – do ‘experts’ agree on the correct

answer?
Item difficulty – the proportion of examinees
that get the correct answer
Distracter evaluation
Item discrimination – the ability of an item to
distinguish between knowledgeable students
and others

11
Correlation with final unit mark

r5 = 0.584 r6 = 0.569
(p < 5.3x10-10) (p < 1.8x10-9)

12
Correlation with final unit mark
0.9
Exam
0.8
Ass4 Ass 7
t 0.7
n
e
ic
if Ass2
f 0.6
e Ass5 Ass3 y=0.1854ln(x) +0.2209
o
c Ass6
n 0.5 R² =0.8698
o
it
a
l
e
rr 0.4
o
c Ass1
n 0.3
o
s
ra
e 0.2
P
0.1

0
0 5 10 15 20 25 30 35
Percentageofunitmarks

13
Correlation with final unit mark
0.9
Exam
0.8
Ass4 Ass 7


t 0.7
n
e
ic
if Ass2
f 0.6
e Ass5 Ass3 y=0.0139x +0.4626
o
c Ass6
n 0.5 R² =0.7434
o
it
a
l
e
rr 0.4
o
c Ass1
n 0.3
o
s
ra
e 0.2
P
0.1

0
0 5 10 15 20 25 30 35
Percentageofunitmarks

14
Reliability

Really ‘internal consistency’

Typically, Cronbach’s alpha or KR-20

Provides a measure of the average correlation

among items within a test scale

If a test examines a range of ‘themes’, may

need to divide the test data into ‘scales’

15
Review of items

Where items have been developed based on a

systematic test plan and good content validity,
they should only be discarded on the basis of
clearly poor construct validity and reliability

16
Negative marking / formula scoring

“Be the reasons for test-takers’ reluctance to answer under

formula scoring what they may be, this reluctance is a
psychometric Pyrrhic victory. Neither test-takers’ risk attitudes
nor their strategic reasoning are what multiple-choice tests
purport to measure. From the perspective of educational
measurement, individual differences in people’s risk attitudes are
nothing but unwanted noise: they add another source of
variance, which is likely to reduce the test’s reliability and
validity. Furthermore, the testmakers’ persisting inability to
distinguish between a guessed correct answer and a known
correct answer is compounded by their inability to distinguish
between an omission deriving from ignorance and one deriving
from risk aversion.” (Bar-Hillel, Budescu & Attali, 2005, p. 7)

17
Negative marking – discontinued by:

The Royal College of General Practitioners – from

1992
 discriminated against female candidates
The Royal College of Obstetricians and
Gynaecologists – prior to 2004
 benefited bold and test-wise candidates; inhibited reasoning
Examination Committee of the European Society of
Anesthesiology – from 2008
 does not reflect real practice where decisions must be made on
incomplete knowledge; no effect on rank ordering of results

18
 A useful
resource?

Source: https://www.ncsbn.org/1287.htm

19
Development of the NCLEX-RN Test Plan

“A total of 12,000 newly licensed RNs are asked

about the frequency and importance of performing
155 nursing care activities. Nursing care activities
are then analyzed in relation to the frequency of
performance, impact on maintaining client safety
and client care settings where the activities are
performed. This analysis guides the development of
a framework for entry-level nursing practice that
incorporates specific client needs, as well as
processes fundamental to the practice of nursing.”

20
NCLEX-RN test plan structure

Safe and Effective Care Environment

Management of Care
Safety and Infection Control

Health Promotion and Maintenance

Psychosocial Integrity
Physiological Integrity
Basic Care and Comfort
Pharmacological and Parenteral Therapies
Reduction of Risk Potential
Physiological Adaptation

21
Design – NCLEX-RN MCQ test plan

22
Detailed
test plan

23
Detailed
test plan

24
Sample items – management of care

For each of the eight content areas a sample

item is provided

25
Steps in item writing

1. Select an area of the test plan for the focus of the item
2. Select a subcategory from the chosen area of the test
plan
3. Select an important concept within that subcategory
4. Use the concept selected and write the stem
5. Write a key to represent important information the entry-
level nurse should know
6. Identify common errors, misconceptions or irrelevant
information
7. Use the previous information and write the distracters
8. Complete the item using the stem, key and distracters

26
Steps in item writing - example

Step 1. Select an area of the test plan for the focus of the item
Safety and Infection Control

Step 2. Select a subcategory from the chosen area of the test plan
Standard Precautions/Transmission-Based
Precautions/Surgical Asepsis

Step 3. Select an important concept within that subcategory

Evaluate infection control precautions implemented by staff
members

27
Steps in item writing - example

Step 4. Use the concept selected and write the stem

The nurse and nursing assistant are caring for a client with
vancomycin-resistant enterococci (VRE). Which of the
following activities by the nursing assistant would require
immediate follow-up?

Step 5. Write a key to represent important information the entry-

level nurse should know
Contact Isolation: Assisting the client to ambulate in the
hallway

28
Steps in item writing - example

Step 6. Identify common errors, misconceptions or irrelevant

information
Lack of understanding of isolation precautions
Uncertainty related to specific diagnosis

Step 7. Use the previous information and write the distracters

Leaving a blood pressure cuff in the client’s room to be
used by the client only
Putting on a protective gown to assist the client to sit in
a chair
Taking the gloves off before leaving the client’s room

29
Steps in item writing - example

Step 8. Complete the item using the stem, key and distracters

The nurse and nursing assistant are caring for a client with
vancomycin-resistant enterococci (VRE). Which of the
following actions performed by the nursing assistant would
require immediate follow-up from the nurse?

a.Leaving a blood pressure cuff in the client’s room to be used

by that client only
b.Putting on a protective gown to assist the client to sit in a
chair
c.Taking the gloves off before leaving the client’s room
d.Assisting the client to ambulate in the hallway (key)

30
Case scenario exercise

Reduction of Risk Potential

The nurse is caring for client who had a

procedure three hours ago. Write an item that
includes assessment data the nurse would
observe in this client and which data should the
nurse respond to first.

31
Reduction of Risk Potential – model

The nurse is caring for a client who had a cardiac

catherization 3 hours ago. Which of the following
findings would be essential for the nurse to follow-up?

a.blood pressure increase from 103/68 to 110/70 over

the past one hour
b.blood urea nitrogen (BUN), 22 mg/dL (key)
c.pulse, 101
d.decrease in respiratory rate from 18 to 16 over the
past one hour

32
DSO assessment tools - resources

 Online assessment strategies:

 http://www.deakin.edu.au/itl/dso/strategies-teaching/assessm
 Blackboard assessment tools guides:
 http://www.deakin.edu.au/itl/dso/guides/guide-list-assessme
 Respondus guides:
 http://www.deakin.edu.au/itl/dso/guides/guide-list-respondus

33
Case scenario exercise

Psychosocial Integrity

A nurse on an inpatient psychiatric unit

observes a client pacing the hallway, mumbling
and occasionally yelling aloud “Stop it!” Write an
item describing the action the nurse should take
in this situation.

34
Psychosocial Integrity – model MCQ

The nurse is caring for a client who is mumbling, pacing

in the hallway, and occasionally yelling “Stop it!” Which
of the following actions should the nurse take?

a.Remove other clients from the area.

b.Escort the client back to the client’s room.
c.Request that the client be quiet and not disrupt others.
d.Use distraction to re-focus the client to reality. (key)

35
Thank you for your time

36