RELIABILITY

-is the consistency with which an instrument

measures the attribute
-the less variation an instrument produces in
repeated measurements of an attribute , the
higher its reliability
RELIABILITY
-An instrument is reliable if its measures
accurately reflect true scores. A reliable
measure is one that maximizes the true score
component and minimizes the error component
of an obtained score.
RELIABILITY
Many psychosocial scales contain two or more
subscales , each of which tap distinct, but related,
concepts (e.g measure of independent functioning
might include subscales for motor activities ,
communication and socializing.) The reliability of
each subscale is typically assessed and,if subscale
scores are summed for a total score, the scales
overall reliability would be assessed.
3 ASPECTS OF RELIABILITY
1. STABILITY
2. INTERNAL CONSISTENCY
3. EQUIVALENCE
1. STABILITY
-The stability of a measure is the extent to
which the same scores are obtained when the
instrument is used with the same people on
separate occasions.
1. STABILITY
TEST-RETEST RELIABILITY PROCEDURES
-the researchers administers the same measure to a sample of people
on two occasions, and then compares the scores.
1. STABILITY
TEST-RETEST RELIABILITY PROCEDURES
Ex.
We are interested in the stability of a self-report scale that measured
self-esteem in adolescents. Because self-esteem is a fairly stable
attribute that would not change markedly from one day to the next,
we would expect a reliable measure of it to yield consistent scores
on two separate test.
1. STABILITY
TEST-RETEST RELIABILITY PROCEDURES
Ex.
(cont.) As a check on the instruments stability, we arrange to
administer the scale 3 weeks apart to a sample of teenagers.
 Table 1 Data for Test-retest Reliability of
Self-Esteem Scale
SUBJECT NUMBER TIME 1 TIME 2
1 55 57
2 49 46
3 78 74
4 37 35
5 44 46
6 50 56
7 58 55
8 62 66
9 48 50
10 67 63
r=.95
1. STABILITY
TEST-RETEST RELIABILITY PROCEDURES
Ex.
(cont.) On the whole, differences on the two tests are not large.
RELIABILITY COEFFICIENT=a numeric index of a measures reliability
to objectively determine exactly how small the differences are.
1. STABILITY
RELIABILITY COEFFICIENT
-a numeric index of a measures reliability to objectively determine
exactly how small the differences are.
-designated as r range from .00-1.00*
-the higher the value, the more reliable (stable) is the measuring
instrument. In the example shown in Table 1, the reliability
coefficient is .95 which is quite high.
1. STABILITY
TEST-RETEST RELIABILITY PROCEDURES
Disadvantage:
-Many traits of interest change over time, independently of the
instruments stability
(attitude, mood, knowledge can be modified by experiences between
2 measurements)
-Therefore, stability indexes are most appropriate for relatively
enduring characteristics-personality and abilities.
Example of test-retest reliability
-Needham, Abderhalden, Dassen, Haug, Fischer (2004)
developed and tested a short form of the PERCEPTION OF
AGGRESSION SCALE, which is designed to measure
psychiatric nurses attitudes towards, and perceptions of
patient aggression. To select items with the strongest
stability, the researchers administered the full scales to three
groups of psychiatric nurses twice: to one group after 4 days,
to the second after 14 days, and to the third after 70 days.
The 12 items selected for the short form has a test-retest
reliability of .77.
2. INTERNAL CONSISTENCY
-An instrument may be said to have Internal consistency reliability to
the extent that all its subparts measure the same characteristic.
-This approach assesses an important source of measurement error
in multi item measures: the sampling of items.
2. INTERNAL CONSISTENCY
How to assess?
A. SPLIT HALF TECHNIQUE
-oldest
-The items comprising a scale are split into two groups
(usually odd vs even items) and scored, and then the scores
on the two half-tests are used to compute reliability
coefficient. If the two half-test are really measuring the
same attribute, the reliability coefficient will be high.
2. INTERNAL CONSISTENCY
How to assess?
B. CRONBACHS ALPHA (or COEFFICIENT ALPHA) This
method gives an estimate of the split-half correlation for all
possible ways of dividing the measure into two halves, not
just odd versus even items.
3. EQUIVALENCE
-used primarily with observational instruments
-determines the consistency or equivalence of
the instrument by different observers or raters.
3. EQUIVALENCE
Risk for observer error
-the degree of error can be assessed through
INTERRATER (OR INTEROBSERVER) RELIABILITY.
3. EQUIVALENCE
INTERRATER (OR INTEROBSERVER) RELIABILITY
-which is estimated by having 2 or more trained
observers make simultaneous, independent
observations.
3. EQUIVALENCE
INTERRATER (OR INTEROBSERVER) RELIABILITY
-(cont.) the resulting data can then be used to
calculate an index of equivalence or agreement.
-A reliability coefficient can be computed to
demonstrate the strength of the relationships
between the observers ratings.
3. EQUIVALENCE
INTERRATER (OR INTEROBSERVER) RELIABILITY
-When two independent observers score some
phenomenon congruently, the scores are likely
to be accurate and reliable.
EXAMPLE:
-Kovach (2002) observed the behaviors of older
persons with dementia over 30 minute observation
sessions. Interrater reliability, calculated as
percentage agreement between 2 raters, was .74 for
the variable activity, .92 for noxiousness, and .84 for
agitation.
INTERPRETATION OF RELIABILITY
COEFFIECIENTS
-Reliability coefficients are an important indicator of an
instrument's quality.
-A measure with low reliability prevents an adequate testing
of research hypotheses.
-If data fail to confirm a hypothesis, one possibility is that
the measuring tool was unreliable-not necessarily that that
the expected relationship do not exist.
INTERPRETATION OF RELIABILITY
COEFFIECIENTS
-Knowledge about an instruments reliability thus is critical
in interpreting research results, especially if research
hypotheses are not supported.
-Reliability estimates vary according to the procedures used
to obtain them. Estimates of reliability computed by
different procedures for the same instrument are not
identical.
INTERPRETATION OF RELIABILITY
COEFFIECIENTS
-reliability of an instrument is related to sample
heterogeneity. The more homogenous the sample (i.e. the
more similar the scores), the lower the reliability coefficient
will be. This is because instruments are designed to
measure differences, and if sample members are similar to
one another, it is more difficult for the instrument to
discriminate reliably among those who possess varying
degrees of the attribute.
INTERPRETATION OF RELIABILITY
COEFFIECIENTS
-longer instruments (i.e. those with more items) tend to
have higher reliability than shorter ones.
VALIDITY
-