Topic 7: Factors Affecting Test

Scores & Simple Test Evaluation

for Class Teacher
(Bachman, 1990, Chapters 6&7)

Nguyen Nhu Hoa

1
Factors Affecting Test Scores
Communicative
and
Language
ability

Test Method Random

Test scores
Facets Factors

Personal
attributes

2
 Part 1: Factors affecting scores (1)
• Test method facets: such as test format/ response format, input format,
length of test
- Same testing skills & sub-skills but different test formats or response
formats (may be in favour of one group but not the other)

• Test content
- Culture background
Culture features embedded in test content may bias one group but
not the other.
- Background knowledge
 e.g.. an IELTS test listening section 3 about process of doing
an assignment in Western education or a mini-lecture about birds
in Tasmania, a TOEFL iBT lecturer or a lecture about Pluto.
 In ESP testing, it is necessary to distinguish between language
proficiency and background knowledge and the test designed should
define specific language ability as part of language ability to be tested.
Score from an ESP reading test should not be interpreted as general
reading ability.

3
 Part 1: Factors affecting scores (2)
• Cognitive characteristics
- Field independence
Field independence is “the extent to which a
person perceives part of a field as discrete from
the surrounding field as a whole, rather
embedded, or … the extent to which a person
perceives analytically” (p.275).
Test takers who are field independent are
likely to perform better than those field
dependence especially in discrete point tests.

4
 Part 1: Factors affecting scores (3)
• Ambiguity tolerance
Ambiguity tolerance is “a person’s ability to function
rationally and calmly in a situation in which interpretation
of all stimuli is not clear” (p.277)
Individuals with high ambiguity tolerance might
perform better than those with low ambiguity tolerance.
- E.g.. Cloze test (clearer till the end of test,
dictation: some words could not be recognized until the
second or last item of reading)
 This evidence is not so clear with multiple-choice
response format though research show a significant (but
low) correlation between scores on a measure of
ambiguity and multiple choice measures of English
proficiency
5
 Part 1: Factors affecting scores (4)
• Random factor: testing environment
• Native language background, ethnicity, sex
and age
- These characteristics of test takers are not
facets of test methods thus they cannot be
considered as possible sources of measurement
error.
- All these can only provide information about
how language learning varies with age, ethnicity,
sex, and other individual characteristics.

6
 Part 2: Practical Approach of Test
Evaluation of Class Teacher (1)
• Preparing an item analysis
- score all the test takers
- rank them from the highest to the lowest
and divide them equally into three groups
high - middle – low? Why? Making a gap
between high and low group
-record students’ responses: from high
group to low group; if it is a multiple
choice, circle the correct option
7
 Part 2: Practical Approach of Test
Evaluation of Class Teacher (1)
Item 1 High Group (H) Low Group (L)

A/ ///

(B) ///// //

C // /

D / //

X / //
8
 Part 2: Practical Approach of Test
Evaluation of Class Teacher (2)
1. Difficulty level
High correct + Low correct or Hc + Lc
Total number in sample (H+L) N

e.g.. In this case (5+2): 20 = 7/20 – 35%

Note: easy ≥ 90%, difficult: ≤ 30%

2. Discrimination level
High correct - Low correct or Hc - Lc
Total number in sample (H+L) N

E.g.. In this case, (5-2):20 = 3/20 = 15%, ≥ 15% is acceptable; from

10% to 15% is questionable
Note: ≤ 10% discrimination is not acceptable
9
Part 2: Practical Approach of Test Evaluation of
Class Teacher (3)
• Distracter evaluation
• Weak distracters: poor discrimination or not chosen by
examiners
• Only one or two distracters attract attention.
Reasons:
- the item has been revised well in the class and
everyone master it: the answer is obvious
- no other choices seem likely to be the answers
- Obviously impossible distracter
• If many items are left blank near the end of the test, the
test should be shortened or more time should be given to
it.
10
1. Practice exercises
Item 1(D) Item 2 (A) Item 3 (B) Item 4 (C)

High Low High Low High Low High Low

A // // // //// //

B / // / ///// / //

C ///// // / // ///// ///// //

////
D //// / / /// //
///// /
X / / //
(blank) 11
Questions
1. Calculate the level of difficulty for each of the four items
Item 1= Item 2= Item 3= Item 4=
Which of these items are too difficult and which are easy?
2. Calculate the discrimination of each item.
a. Which item has the poorest discrimination? ……………………
b. Which item has unsatisfactory discrimination? …………………
c. Which item(s) has borderline? ……………………
3. Look at the distracters on the four items.
In which are they the most effective? ……………………
In which are they the least effective? ……………………
4. Is there any item with negative discrimination? If so, which one?
……………………………………………………………………………
5. Which item did the fewest students leave blank? …………………..

12