Grammar Assessment

Advanced Assessment and Teaching Profession

Compiled by
Fajar Darmawan 201710560211010
Ary RW 201710560211019

Magister of English Education

University of Muhammadiyah Malang

2017
 Grammar was once regarded as a set of rules that prescribed the so-called
correct ways of structuring and using language.What was taught as grammar was often
related to understanding and learning patterns of other languages such as Latin.This
traditional view of grammar was seen by many linguists as limiting, as it could not
always be applied to the dynamic and evolving nature of modern English language. As a
consequence, grammar also evolved into a dynamic concept that could be applied to
current language usage. As such, grammar is now seen as a way of describing regular
language patterns and the way these patterns function to make meaning in particular
contexts or cultures.
Grammar can be defined as systems for ordering the resources of language in a
text to achieve particular meanings in particular contexts. In learning about grammar,
students learn to understand how these systems work and to make grammatical choices
appropriate to particular contexts or situations. Thus grammar as it is related to the
writing tasks in ELLA and BST Writing needs to be understood at two levels:
1. Textual Grammar where whole text processes and features, such as structure,
organisation and the requirements of a particular text type, are considered in
relation to purpose, audience and theme, and
2. Syntactical or Sentence Grammar which considers how smaller elements such
as paragraphs, sentences and word choice contribute to the meaning and
effectiveness of the text.

Grammaring
Larsen-Freeman (2001, 2003) offers “grammaring” – the ability to use grammar
structures accurately, meaningfully, and appropriately as the proper goal of grammar
instruction. The addition of “-ing” to grammar is meant to suggest a dynamic process of
grammar using. In order to realize this goal, it is not sufficient for students to notice or
comprehend grammatical structures. Students must also practice meaningful use of
grammar in a way that takes into account “transfer appropriate” processing (Roediger &
Guynn, 1996). This means that in order for students to overcome the inert knowledge
problem and transfer what they can do in communicative practice to real.
communication outside of the classroom, there must be a psychological similarity
between the conditions of learning and the conditions of use (Segalowitz, 2003).
Other benefits of grammar instruction have been proposed (R. Ellis 1993, 1998,
2006). One is to help students “notice the gap” between new features in a structure and
how they differ from the learners’ interlanguages (Schmidt & Frota, 1986). Grammar
instruction can also help students generalize their knowledge to new structures (Gass,
1982). Another contribution of grammar teaching may be to fill in the gaps in the input
(Spada & Lightbown, 1993), since classroom language will not necessarily represent all
grammatical structures that students need to acquire.

Grammatical Assessment
In the traditional approach to assessing grammar, grammatical knowledge is
defined in terms of accurate production and comprehension, and then assessed through
the four skills. Testing is typically done by means of decontextualized, discrete-point
items such as sentence unscrambling, fill-in-the-blanks, error correction, sentence
completion, sentence combining, picture description, elicited imitation, judging
grammatical correctness, and modified cloze passages. Such formats test grammar
knowledge, but they do not assess whether test takers can use grammar correctly in real-
life speaking or writing.
A significant contribution of the communicative or proficiency-based approach
in the 1970s and 1980s was a shift from seeing language proficiency in terms of
knowledge of structures, which could best be assessed using discrete-point items, to the
ability to integrate and use the knowledge in performance, which could best be assessed
through the production and comprehension of written texts and through face-to-face
interaction under real-time processing conditions (McNamara & Roever, 2006, pp. 43–
4). In the latter, more integrative, approach to grammar assessment, grammatical
performance is typically assessed by raters using scales that gauge grammatical
accuracy, complexity, and the range of grammatical structures used. The judgments are
subjective, and because the assessment formats are more open-ended, they are subject to
possible inconsistencies.
 For this reason, certain factors, such as rater severity and prompt difficulty, must
be examined, usually accomplished by means of generalizability theory or item-
response theory (Purpura, 2006). Because of the preference in recent years for
measuring the use of grammar mholistically through speaking and writing, some
standardized examinations, e.g., the TOEFL, no longer have a separate section of the
test that deals with structure explicitly. The decision to eliminate the explicit testing of
grammar was made in at least two cases based on research showing that a separate
subtest of grammatical knowledge could not be adequately differentiated from other
sections of a test (Cushing Weigle & Lynch, 1995 and Alderson, 1993, cited in Purpura,
2004).
A consequence of such decisions, however, is that it is difficult to separate out
what in the ability to read or write the texts is due to the lack of knowledge concerning
grammatical structures and what might be due to other factors. We also have no way of
diagnosing grammatical difficulties learners may be experiencing or in providing them
with feedback (Purpura, 2004). In sum, discretepoint and integrative tests represent
different approaches to grammar assessment, each of which have a contribution to
make.

Innovations in Grammar Assessment

There are a number of innovations underway, or at least proposed, in the way grammar
is being assessed.
1. Redefining the construct
The first involves a definition of the grammar construct itself. Expanding beyond form
to include grammatical meaning is one such move. For instance, Purpura (2004, p. 89)
defines grammatical ability for assessment purposes as involving “the capacity to
realize grammatical knowledge accurately and meaningfully in test-taking or other
language-use contexts.” Grammatical ability may (also) interact with pragmatic ability,
which Purpura considers a different ability area.
2. Partial scoring
Discrete-point tests usually rely on dichotomous scoring of grammatical accuracy.
Recently, it has been proposed that scoring grammatical items polytomously would
yield information about learners who have an intermediary knowledge of grammar,
rather than their being treated as if they have no knowledge at all (Purpura, 2006). To
examine the extent to which answers on multiple-choice grammar items can be ordered
along a path of progressive attainment, Purpura (2006) examined the grammar section
of the University of Michigan ECPE, and found that many of the items did show what
seemed to be a progressive attainmentpattern in the response patterns of 1,000
candidates. If these items are indeed measuring developmental levels, dichotomous
scoring raises several concerns. First, a considerable amount of developmental
information is lost with students who have partial knowledge. More seriously, scoring
dichotomously underestimates some students’ true ability, and it makes it impossible for
some students to receive feedback appropriate to their developmental level. While
partial scoring is not a complete solution, it is one step in the long-hoped-for
development of an interlanguage-sensitive approach to assessment (Larsen- Freeman &
Long, 1991).
3. The social dimension
Language tests have ignored the social use dimension of language and have followed
traditional psychometric methods in measuring isolated pieces of grammar and
vocabulary knowledge; as a consequence, measuring test takers’ ability to use language
in social contexts has been overlooked (McNamara & Roever, 2006). Importantly, this
awareness goes beyond extending the construct being measured. A social view of
performance is incompatible with the traditional view of performance as a simple
projection or display of individual competence. Increasingly, therefore, language testers
are questioning whether it is possible to isolate the contributions of test takers from
those of the test takers’ interlocutors, say in oral proficiency interviews.
4. The standard
Another issue that could be discussed under grammar teaching or testing is the issue of
what the target standard is. For instance, some researchers have claimed that as English
increasingly becomes the language of communication between non-native speakers), it
is likely that “ungrammatical, but unproblematic” constructions, such as “he look very
sad,” “a picture who gives the impression”(Seidlhofer, 2001, p. 147), once they exist
sufficiently frequently in non-native speaker discourse, would arguably become
standardized and exist as a variety (English as a lingua franca) alongside English as a
native language. Kachru and Nelson (1996, in Siegel, 2003) point out that considering
the non-standard features of indigenized varieties to be the result of L1 interference and
fossilization would be wrong because learners may not wish to emulate a standard, and
standard models may not be available in the environment.
5. Comprehensive English Language Test (CELT)
The Comprehensive English Language Test (CELT) (Harris and Palmer, 1970a, 1986)
was designed to measure the English language ability of nonnative speakers of English.
The authors claim in the technical manual (Harris and Palmer, 1970b) that this test is
most appropriate for students at the intermediate or advanced levels of proficiency.
English language proficiency is measured by means of a structure subtest, a vocabulary
subtest and a listening subtest. According to the authors, these subtests can be used
alone or in combination (p. 1). Scores from the CELT have been used to make decisions
related to placement in a language program, acceptance into a university and
achievement in a language course (Harris and Palmer, 1970b, p. 1), and for this reason,
it may be considered a high-stakes test. One or more subtests of the CELT have also
been used as a measure of English language proficiency in SLA research.
6. The First Certificate in English Language Test (FCE)
The First Certificate in English (FCE) exam was first developed by the University of
Cambridge Local Examinations Syndicate (UCLES, now Cambridge ESOL) in 1939
and has been revised periodically ever since. This exam is the most widely taken
Cambridge ESOL examination with an annual candidature of over 270,000 (see
http://www.cambridgeesol.org/exam/index.cfm). The purpose of the FCE (Cambridge
ESOL, 2001a) is to assess the general English language proficiency of learners as
measured by their abilities in reading, writing, speaking, listening, and knowledge of the
lexical and grammatical systems of English (Cambridge ESOL, 1995, p. 4). More
specifically, the FCE is a level-three exam in the Cambridge main suite of exams, and
consists of five compulsory subtests or ‘papers’: reading, writing, use of English,
listening and speaking (Cambridge ESOL, 1996, p. 8).
7. Multiple Choice Tests
Probably the most common way of testing grammatical knowledge is the multiple
choice test. These tests have the advantage of being easy to grade and being able to
cover a lot of grammatical points quickly.The most common type of multiple choice
grammatical item is one in which the test maker gives the testee a sentence with a blank
and four or five choices of a word or phrase which completes the sentence correctly. For
example,

Because my mother was sick, I _____ to go home last week.

a) had b) have c) has d) hadn't

To give slightly more context, this type of question sometimes makes use of a short
dialogue, with one person saying something and the other person responding.
8. Error Correction
Error correction items are also useful for testing grammar. An error correction item is
one in which the testee is given a sentence with an error. Four words or phrases in the
sentence marked with letters, and the testee needs to decide which of the words or
phrases has the error. For example:

Most of students believe that they should be getting better grades than they are
A B C D

The teacher may also mix in some sentences that have no errors, and students are
required to indicate that there is no error. In addition, the students might be required to
correct the error. Errors from students' actual writing are a good source of ideas for this
type of exercise.

9. Items to Test Knowledge of Word/Sentence Order

Other types of items can be used to test testees' knowledge of word order. The
traditional way is to present the testee with four alternative word orders. For example:

I wonder how she knows _____.

a) how it costs much.
b) how much it costs.
c) it costs how much.
d) it how much costs.

Another possibility is to give testees the four words and ask them to put the words in
order. For example:
I wonder how she knows __________
A. how B. It C. much D. costs
/__/__/__/__/

This can also be done in a way that actually requires the writer to do some writing. For
example:
I wonder how she knows ___________________.
A. how B. it C. much D. costs
Understanding of appropriate sentence order can also be tested in a similar way by
giving testees several sentences and asking them to put them in order. This type of test
tests knowledge of references, cohesive devices, etc.
10. Completion items
Completion items are items in which the testees are asked to fill in blanks in sentences.
For example:

Give the book to ______ woman in the blue dress.

For the purpose of a grammar test, the words which fit in the blanks should be function
words, such as articles and prepositions. (Completion items intended to test reading
ability or vocabulary knowledge, in contrast, use content words.) The advantage of
completion items is that they test production, not just recognition. The disadvantage is
that they need to be marked by hand and there will be some cases where the marker
needs to make judgements about whether a response is correct.
It is not always easy to write items for which there is only one possible answer. Using a
piece of continuous prose rather than disconnected sentences is one way of cutting
down on possible different interpretations of what goes into a particular blank, but it is
probably impossible to entirely eliminate the possibility of different answers.
11. Transformation Items
Another type of grammar item makes use of transformations. In this type of item,
testees are given a sentence and the first few words of another sentence to change the
original sentence without changing the meaning. For example:

1. Jim hasn't been home in a long time.

It's been a long time _________________________.
2. I don't need to go to the grocery store this week.
It isn't __________________________________________.
3. It is difficult to study when it is so noisy.
Studying ________________________________________.

There are variations on this type of item in which the word which starts the transformed
sentence is underlined, or the testee is given one word to use in the new sentence. For
example:

I don't need to go to the grocery store this week. (necessary)

Again, this type of test is difficult to grade because the teacher has to be aware of the
variety of possible answers. Another problem is that it does not in any way test the
testees' knowledge of when each of the possible transformations would be most
appropriate. For example, the testee might be perfectly able to transform an active
sentence to a passive sentence but not know when to use passive rather than active.
However, it is still sometimes a useful test of grammatical knowledge.
12. Sentence Combining Exercises
Sentence combining exercises can play a part in testing grammar as well as its more
traditional use as part of composition testing and training. For example, testees might be
instructed to combine the following sentences using a relative pronoun.

I met a man.
The man went to the same high school I did.

I met a man who went to the same high school I did

The Characteristics of Grammatical Test Tasks

As the goal of grammar assessment is to provide as useful a measurement as

possible of our students’ grammatical ability, we need to design test tasks in which the
variability of our students’ scores is attributed to the differences in their grammatical
ability, and not to uncontrolled or irrelevant variability resulting from the types of tasks
or the quality of the tasks that we have put on our tests. As all language teachers know,
thekinds of tasks we use in tests and their quality can greatly influence how students
will perform.
1. Characteristics of the setting
The characteristics of the setting include the physical characteristics, the participants,
and the time of the task. Obviously these characteristics can have a serious, unexpected
effect on performance.
2. Characteristics of the test rubrics
The test rubrics include the instructions, the overall structure of the test, the time
allotment and the method used to score the response. These characteristics can
obviously influence test scores in unexpected ways(Madden, 1982; Cohen, 1984, 1993).
3. Characteristics of the input
According to Bachman and Palmer (1996), the characteristics of the input (sometimes
called the stimulus) are critical features of performance in all test and TLU tasks. The
input is the part of the task that test-takers must
process in order to answer the question. It is characterized in terms of the format and
language.
4. Characteristics of the expected response
When we design a test task, we specify the rubric and input so that testtakers will
respond in a way that will enable us to make inferences about the aspect of grammar
ability we want to measure. The ‘expected response’
thus refers to the type of grammatical performance we want to elicit. The characteristics
of the expected response are also considered in terms of the format and language.
Similar to the input, the expected response of grammar tasks can vary according to
channel (aural orvisual), form (verbal, non-verbal), language (native or target) and
vehicle (live or reproduced).
5. Relationship between the input and response
A final category of task characteristics to consider in examining how test tasks impact
performance is seen in how characteristics of the input can interact with characteristics
of the response. One characteristic of this
relationship involves ‘the extent to which the input or the response affects subsequent
input and responses’ (Bachman and Palmer, 1996).