Q1.

Discuss eight variables which are commonly used in research (20 Marks)
A VARIABLE is a measurable characteristic that varies. It may change from group to
group, person to person, or even within one person over time. The common variable
types are:
i.

Dependent variables

Show the effect of manipulating or introducing the independent variables. For

example, if the independent variable is the use or non-use of a new language teaching
procedure, then the dependent variable might be students' scores on a test of the
content taught using that procedure. In other words, the variation in the dependent
variable depends on the variation in the independent variable.
ii.

Independent variables

. . . are those that the researcher has control over. This "control" may involve
manipulating existing variables (e.g., modifying existing methods of instruction) or
introducing new variables (e.g., adopting a totally new method for some sections of a
class) in the research setting. Whatever the case may be, the researcher expects that
the independent variable(s) will have some effect on (or relationship with) the
dependent variables.
iii.

Intervening variables

. . . refer to abstract processes that are not directly observable but that link the
independent and dependent variables. In language learning and teaching, they are
usually inside the subjects' heads, including various language learning processes
which the researcher cannot observe. For example, if the use of a particular teaching
technique is the independent variable and mastery of the objectives is the dependent
variable, then the language learning processes used by the subjects are the
intervening variables.
iv.

Moderator variables

. . . affect the relationship between the independent and dependent variables by

modifying the effect of the intervening variable(s). Unlike extraneous variables,
moderator variables are measured and taken into consideration. Typical moderator
variables in TESL and language acquisition research (when they are not the major
focus of the study) include the sex, age, culture, or language proficiency of the
subjects.
v.

Control variables

Language learning and teaching are very complex processes. It is not possible
to consider every variable in a single study. Therefore, the variables that are not
measured in a particular study must be held constant, neutralized/balanced, or
eliminated, so they will not have a biasing effect on the other variables. Variables that
have been controlled in this way are called control variables.
Page
1
Ketter Robert

vi.

Extraneous variables

. . . are those factors in the research environment which may have an effect on
the dependent variable(s) but which are not controlled. Extraneous variables are
dangerous. They may damage a study's validity, making it impossible to know whether
the effects were caused by the independent and moderator variables or some
extraneous factor. If they cannot be controlled, extraneous variables must at least be
taken into consideration when interpreting results.

Q2. While using relevant examples, explain four types of reliability and four types of
validity commonly used in research(40MKs)
a) Reliability has to do with the quality of measurement. In its everyday sense,
reliability is the "consistency" or "repeatability" of measures used in research
methods.
Joppe(2000) defines reliability as the extent to which results are consistent
over time and an accurate representation of the total population under study is
referred to as reliability and if the results of a study can be reproduced under a similar
methodology, then the research instrument is considered to be reliable. (p 1)
Validity
Joppe(2000) provides the following explanation of what validity is in quantitative
research:
Validity determines whether the research truly measures that which it was
intended to measure or how truthfulthe research results are. In other words, does the
research instrument allow you to hit the bulls eye of your research object?
Researchers generally determine validity by asking a series of questions, and will often
look for the answers in the research of others. (p. 1)
'Reliability' of any research is the degree to which it gives an accurate score across a
range of measurement. It can thus be viewed as being 'repeatability' or 'consistency'.
In summary:

Inter-rater: Different people, same test.

Test-retest: Same people, different times.

Parallel-forms: Different people, same time, different test.

Internal consistency: Different questions, same construct.

i.

Inter-rater or inter-observer reliability(Different people same test)

Page
2
Ketter Robert

This is a kind of reliability that is at issue when the same stimulus (person, event,
behavior, etc.) must be rated by more than one rater.
Eg. In studies of the relationship between physical attractiveness and social
development, the researchers need to know how attractive the person is. (Research of
this kind asks questions such as, do prettier people develop better social skills?)
How can this rating be done? Calculate the ratio of length of nose to distance between
the ears? While some such physical indexes of attractiveness have been developed,
the most common way is to assemble a panel of judges to rate the stimuli. The
researcher looks at the extent to which the raters agree on their ratings. When interrater reliability is low, the researcher has to wonder if it is possible to classify persons
on a dimension such as attractiveness or whether his or her attempts to do so have
failed. Inter-rater reliability is measured in several ways, such as the percentage of
agreement among the judges or a correlation.
ii.

Test-retest reliability (Same people, different times).

Test-retest reliability refers to whether or not the sample you used the first time you
ran your survey provides you with the same answers the next time you run an
identical survey. There are factors that may affect their answers, but presumably if a
survey is taken twice and nothing has changed to affect responses, the results should
be the same.
The amount of time allowed between measures is critical. The shorter the time gap,
the higher the correlation; the longer the time gap, the lower the correlation. This is
because the two observations are related over time the closer in time we get the more
similar the factors that contribute to error.
Instruments such as IQ tests and surveys are prime candidates for test-retest
methodology, because there is little chance of people experiencing a sudden jump in
IQ or suddenly changing their opinions.
On the other hand, educational tests are often not suitable, because students will
learn much more information over the intervening period and show better results in
the second test.
iii.

Parallel-forms: Different people, same time, different test, different

persons simultaneously.

This refers to the ability of similar questions asking about the same item to have
similar/near identical responses. You want to make sure that all questions relating to a
particular item are being read as equal by the same sample, and thus generating
similar results. A researcher does this by creating a list of questions that are all asking
about the same things, then randomly dividing them up into separate tests with the
same sample, and seeing the correlation between the results.
iv.

Internal consistency: Different questions, same construct.

It refers to the respondents ability to answer similar questions within a single survey
the same. Such as the answer to: How much do you like this product and comparing
Page
3
Ketter Robert

it to On a scale of 1 to 10, what are your general feelings about this product. There
are a lot of ways to judge internal consistency, but the goal is to make sure that
questions asking similar questions yield similar responses.

b) Validity refers to how well a test measures what it is purported to measure.

Types of Validity
i.

Face Validity ascertains that the measure appears to be assessing the

intended construct under study. The stakeholders can easily assess face
validity. Although this is not a very scientific type of validity, it may be an
essential component in enlisting motivation of stakeholders. If the stakeholders
do not believe the measure is an accurate assessment of the ability, they may
become disengaged with the task.

Example: If a measure of art appreciation is created all of the items should be related
to the different components and types of art. If the questions are regarding historical
time periods, with no reference to any artistic movement, stakeholders may not be
motivated to give their best effort or invest in this measure because they do not
believe it is a true assessment of art appreciation.

ii.

Construct Validity is used to ensure that the measure is actually measure

what it is intended to measure (i.e. the construct), and not other variables.
Using a panel of experts familiar with the construct is a way in which this type
of validity can be assessed. The experts can examine the items and decide
what that specific item is intended to measure. Students can be involved in
this process to obtain their feedback.

Example: A womens studies program may design a cumulative assessment of

learning throughout the major. The questions are written with complicated wording
and phrasing. This can cause the test inadvertently becoming a test of reading
comprehension, rather than a test of womens studies. It is important that the
measure is actually assessing the intended construct, rather than an extraneous
factor.

iii.

Criterion-Related Validity is used to predict future or current performance it correlates test results with another criterion of interest.

Example: If a physics program designed a measure to assess cumulative student

learning throughout the major. The new measure could be correlated with a
standardized measure of ability in this discipline, such as an ETS field test or the GRE
subject test. The higher the correlation between the established measure and new
measure, the more faith stakeholders can have in the new assessment tool.
Page
4
Ketter Robert

iv.

Formative Validity when applied to outcomes assessment it is used to assess

how well a measure is able to provide information to help improve the program
under study.

Example: When designing a rubric for history one could assess students knowledge
across the discipline. If the measure can provide information that students are lacking
knowledge in a certain area, for instance the Civil Rights Movement, then that
assessment tool is providing meaningful information that can be used to improve the
course or program requirements.

v.

Sampling Validity (similar to content validity) ensures that the measure

covers the broad range of areas within the concept under study. Not everything
can be covered, so items need to be sampled from all of the domains. This may
need to be completed using a panel of experts to ensure that the content
area is adequately sampled. Additionally, a panel can help limit expert bias
(i.e. a test reflecting what an individual personally feels are the most important
or relevant areas).

Example: When designing an assessment of learning in the theatre department, it

would not be sufficient to only cover issues related to acting. Other areas of theatre
such as lighting, sound, functions of stage managers should all be included. The
assessment should reflect the content area in its entirety.

Q3. Differentiate between reliability and validity as used in research (10Mks)

Reliability

A study or test's reliability is the measure of whether the same results can be
obtained given differing circumstances. For example, having a reliable test
means a student will score the same or close to the same score on the test
under different circumstances. If five different people graded the test, the
scores should be the same or similar. If the student took the test on a Monday
morning or a Wednesday evening, it shouldn't affect the scores. In addition,
someone scoring the test on Monday and then again on Friday should not arrive
at different scores. Reliability controls the results of a test or study so external
factors -- such as when or by whom it is conducted -- do not change the results.

Validity

A study or test's validity is the measure of whether the results actually relate to
what the test or study developer intended to measure or test. If a test claims to
evaluate a student's ability to use multiplication, for example, but consists only
of subtraction questions, it is not valid. If a study was investigating the mating
calls of a certain bird but only investigated male birds, for example, or recorded
calls outside of mating season, it would also be invalid. Examining a study's
Page
5

Ketter Robert

validity allows you to decide whether the results should be accepted and used
to generalize or not.

Page
6
Ketter Robert