PROPOSAL

Item specifications for a second year Spanish listening

criterion-referenced test for a large language program
Chris Wright (Spring 2010)
This document introduces Criterion-Referenced Tests (CRTs) and is followed by two item
specifications for proposed test items in the second year Spanish listening CRT. Each item
specification is to be used as a blueprint for generating more test items that are directly related to
the course objectives of the program curriculum. These proposed items are to measure the
students mastery of content and objectives found in the chapters 15-16 in the Sabas que...
Spanish textbook currently used by the Spanish program.
To further validate the content of these proposed test items rating scale may be created to
determine the content applicability (the degree that content is appropriate for the given course or
program) and content congruence (the degree to which an item is measuring what it was
designed to assess). A rating scale, as seen below, may be used to survey at least five instructors
that teach the second year Spanish course with the Sabas que... textbook.
Not very
Well

Moderate

Very
Well

The item is related to the martial that I teach.

Comment: _________________________

The item matches with the objective its meant to test.

Comment: ___________________________________

Once a CRT is developed with sufficient test items that are validated by peers, a pre-test
and post-test may be carried out to see if the new listening test does what its supposed to do. The
CRT should be given at the beginning of the semester before instruction. It should have a
positively-skewed distribution. After three months of instruction at the end of the semester, this
1

same exact CRT could be given as a post-test. That is, after treatment of instruciton. The
distribution should then be negatively-skewed. Such a change is ideal because it shows a positive
effect of instruction. Below is example of these ideal distributions on a histogram.

Number of Students

Ideal CRT Distributions

Percent Scores
There are two quantitative item statistics that may be carried out on CRTs. One is the B-index,
which shows the effectiveness of each item in differentiating the masters from the non-masters in
one single test after instruction. The other is the difference index (DI), which does the same but
by comparing a pre-test and a post-test, looking for the ideal CRT distributions.

It is obvious that the rules and design of this proposed CRT is not compatible with the
rules and design of the currently used ACTFL Listening Comprehension Assessment which is
norm referenced. However, lining up the content and tasks of the currently used listening tests
with the content and tasks taught from the textbook would; (1) be an content validity
improvement, (2) still be graded by machine score for convenience, (3) are still candidate test
items for specific ACTFL proficiency tasks (ex. chapter 15 has a proficiency task level of Novice
and chapter 16 has a proficiency task level of Intermediate), and (4) are still discrete point,
meaning that they measure specifically listening comprehension and not memorization skills.
2

Test Item Specification for Chapter 15

1. General description
These items allow students to show ability to follow spoken directions for arriving to a
destination within a city. All content is found in Chapter 15 in the Sabas que... Spanish
textbook.
2. Sample test item:
Where did you arrive?
a. La clnica
c. El gimnasio
b. La escuela
d. La biblioteca
3. Stimulus attributes:
English instructions before stimulus exposure: You are about to listen to Spanish
directions. Start at the bottom left corner and follow the directions with your pencil until
you arrive to the destination. Then, answer the multiple choice question.

El gimnacio

Calle C

Calle B

La clinica

Calle A

Spanish audio: Siga recto dos cuadras en la calle 3 y pare. Doble a la izquierda y vaya al
norte dos cuadras. Despus, doble a la izquierda. Siga recto al oeste una cuadra. Cruce la
Calle B y siga una cuadra ms. Pare. A dnde lleg?

Calle 1
La escuela
Calle 2
La casa

La oficina

Calle 3
Start

4. Response attributes:
The map is a drawn on as students listen to the directions. They may track the map with
their pencil as if it were a car. Once the destination is determined, the student answers the
discrete multiple choice item on a Scranton. The opposite side of this map sheet may also
be used as writing space for recall protocols on other test items. This map sheet is
returned to the test administrator.
5. Specification supplement
English instructions must be used first to clarify how the test item works. The Spanish
directions need to be said one step at a time with short pauses between phrases. It is
important not to fill in all the boxes of the map with destination names because the
directions only give a general idea of the correct location. This proposed map would
work. Multiple test items, up to four, may be used for this same map size. It is also an
option to have a larger map with 5 by 5 blocks, being 25 blocks in total. This would give
more destination options for more test items. Also, the word recto (straight) must be used
instead of the word derecho (straight). Derecho (straight) may be confused with derecha
(right).

Test Item Specification for Chapter 16

1. General description:
The ability to understand a conversation related to traveling such as; clothing, modes of
transportation and types of lodging. Hypothetical phrases are used. Content found in
Chapter 16.
2. Sample test item:
What type of lodging does Paco finally decide to use?
a. Una pensin
c. Un hotel de cinco estrellas
b. Un motel
d. Una clase turstica
3. Stimulus attributes:
A photo of Paco and Maria in a travel agency is displayed. Students then hear this
preliminary background description in English: You are about to hear a conversation
between a travel agent named Maria and a tourist named Paco. Maria helps Paco organize
his vacation. You have 30 seconds to read the questions. (Pauses 30 seconds) Now listen
to the conversation then answer the questions.
Spanish audio sample:
Paco: Hola Mara, me gustara viajar al extranjero. Quiero viajar en clase turstica para `
una vacacin extica.
Mara: Excelente Paco, te puedo ayudar. Preferas en crucero? Podras ir al Caribe.
Paco: Eso no me parece bien. No me gusta ponerme los trajes de bao. Tambin, si yo
viajara en un crucero no tendra mucho espacio para alojarme. Las cabinas en
estos barcos con pequeas.
Mara: Puedo buscarte una cabina ms grande.
Paco: No gracias, es muy caro. Si yo tuviera ms dinero, yo comprara un boleto para
un crucero elegante, pero es muy caro para m. Tienes otra idea?
Mara: Si Paco, Te gustara viajar al Sur Amrica? Podras visitar a Per. Hay mucho
que ver all, como las ruinas de los Inca en Machu Picchu.
Paco: Interesante, dime ms.
Mara: Hay vuelos que van para all cada da. Hay clase turstica o primera clase.
Paco: Bien, voy a viajar a Per por avin, en clase turstica. Y qu tipos de alojamiento
hay?
Mara: Hay varias pensiones y hotel de cinco estrellas. Qu te interesara?
Paco: Bscame una pensin que sea barata y ms econmica para m.
Pause for one minute to answer four questions and continues for 4 more questions.
5

4. Response attributes:
The student answers the discrete multiple choice items on a Scranton.
5. Specification supplement:
Only two characters with distinct voices speak in the recording. They could be male and
female so that listeners may easily differentiate them in the conversation. Test items for
this level should avoid prompting listeners to memorize detail points of the conversation.
Rather, they should measure the comprehension of the main ideas. The sample recording
has enough material to create up to four test items. It can pause for 30 seconds to allow
students to answer the items. Then, it continues its second half of the conversation for the
next four test items.

Survey for Item Specification of Chapter 15

Very
Poor

Moderate

Very
Good

The item is related to the martial that I teach.

Comment: _________________________

The item matches with the objective its meant to test.

Comment: ___________________________________

Survey for Item Specification of Chapter 16

Very
Poor

Moderate

Very
Good

The item is related to the martial that I teach.

Comment: _________________________

The item matches with the objective its meant to test.

Comment: ___________________________________