IELTS Writing Test: Reliability and Raters

Robert Kirkpatrick
Shinawatra International University, Thailand

Editor@languagetestingasia.com

What this presentation covers:

Validity and reliability of writing tests Difficulties in scoring writing tasks

IELTS writing module

How IELTS organization attempts

objective and reliable rating

What raters (examiners) are looking for

University academic life

Requires the writing of essays, reports and theses
1

University administrators may require writing samples

For native speakers

personal narratives, (also used as indicators of character and motivation)

For non-native speakers

a score from a large scale proficiency test (IELTS, TOEFL)

1Carson,

2001; Hale et al., 1996; Eblen, 1983; Moore & Morton, 1999; Horowitz, 1986; Cansecon & Byrd, 1989

Validity of a test
A test is valid if it reflects the psychological reality of behavior in the area being tested.
2

Great interest among researchers of ways to improve

validation methods and concerns about the tests validity and fairness.
3

2 Hamp-Lyons,

1990

3Ewell,

2004

Validity of Writing Tests

Multiple choice grammar/vocabulary tests [Indirect] May have
Predictive validity Concurrent validity (compares the results with a validated test)

Writing essays [Direct] (Extra costs and problematic rating issues)

Face validity Content validity Construct validity

Direct tests and validity

Even direct writing tests are not perfect
the time restrictions (2 essays in one hour) snapshot approach
4

Students writing reports or theses

may take days or weeks

students have access to books and references

4 Hamp-Lyons

and Kroll, 1996

Kirkpatrick, 1997

Direct tests and validity

Researcher noted the problems of independent tasks Can be problematic for students
Have little background knowledge on a set topic Some luck out with a previously researched topic.
6

Areas where IELTS is known to ask

environment, internet and so on.

4 Hamp-Lyons

5 Kirkpatrick, 1997 6 Gebril & Plakans, 2009; Plakans, 2007; Gebril, 2006; Weigle, 2002, 2004; Cho, 2003; and Kroll, 1996 Cumming, Kantor, Powers, Santos, & Taylor, 2000; Leki & Carson, 1997

Reliability and validity

IELTS Task 1
Typically a graph or process IELTS prompt supplies all necessary information

Task 2 is independent

Many factors cause score variance in writing assessment

tasks, topics, genre, raters, test conditions, time limits,

writing mode (e.g. online or handwritten) or rating scales. Predictable tasks lead to formulaic writing
7 Schoonen,

2005

Reliability and validity

Rater reliability is a major issue Tension between validity and reliability is unresolved

For direct writing tests, raters are central to the entire

assessment. If raters cannot supply scores which are appropriate and consistent, then the whole enterprise is defunct.

8 Shaw

and Weir, 2007

Trained raters
No matter how high the construct validity if reliability of rating is questionable then one is losing. The process of training
9

Familiarization activities, discussion, and practice rating

In comparisons between untrained raters' ratings,

trained raters were more reliable than the untrained raters

11

Lane & Stone (2006)

10

Shohamy, Gordon, and Kraemer, 1992

11

Weigle, 1998

Raters training experience

Full training about 5 years ago Four-day intensive course, Chiang Mai
for speaking and writing

Discussions about marking some raters

not fully understanding getting lost in the details of each section of the band scores.

Raters training experience

Given full registration but not confident Often referred to the self access material
Samples of writing for benchmark

The first day raters scripts were taken and checked. Plus regular evaluations where random samples are checked Every two years have to endure the training again Jagged profile where if a candidate has especially high or low values on the writing section (compared with other sections) another examiner checks the script.

Raters training experience

Now I have rated hundreds and hundreds of scripts
My confidence has grown Rarely see more than a 0.5 band score difference (between my score and a senior rater)

0.5 variation is considered acceptable by IELTS There were differences in rater abilities
Some raters get more experienced but develop bad habits
11

Professional, educational backgrounds are important Only degreed, experienced EFL teachers can apply to be IELTS raters.
9

Wolfe, Kao, and Ranne, 1998; Huot, 1993; Barkaoui, 2010

The band scores

This last section explains the IELTS band scores
Looks at what good academic writing is And what students should be looking to do

Band scores for writing

From late 2005 IELTS changed its policy (a little) Previously the rating scales were hidden but now

information is available to the general public

how band scores are reached what raters are looking at.

Some aspects remain confidential

Band scores for writing

The astute student and IELTS tutor now has enough material to understand what IELTS (and hence what raters) expect at different band levels.

They have four sections

1. Task Response 2. Coherence and Cohesion 3. Lexical Resource 4. Grammar

What I look for

Focused on topic Evidence of English beyond scripted sentences e.g. this is a topic that is hotly debated at this time and I will talk

about both sides.

Complexity of sentence structure. Clear overall structure.

Thank You