How to Design and Evaluate Research in Education

By Jack R. Fraenkel and Norman E. Wallen

Cha ter !

"he Nature o# Research

Ways o# knowing
$ensory e% erience &incom lete'unde enda(le) *greement with others &common knowledge wrong) E% erts+ o inion &they can (e mistaken) ,ogic'reasoning things out &can (e (ased on #alse remises)

Why research is o# value

$cienti#ic research &using scienti#ic method) is more trustworthy than e% ert'colleague o inionintuition- etc.

Cha ter ! . continued

"he Nature o# Research

$cienti#ic /ethod &testing ideas in the u(lic arena)

0ut guesses &hy otheses) to tests and see how they hold u *ll as ects o# investigations are u(lic and descri(ed in detail so anyone who 1uestions results can re eat study #or themselves Re lication is a key com onent o# scienti#ic method

Cha ter ! . continued

"he Nature o# Research

$cienti#ic /ethod &re1uires #reedom o# thought and u(lic rocedures that can (e re licated)

2denti#y the ro(lem or 1uestion Clari#y the ro(lem Determine in#ormation needed and how to o(tain it 3rgani4e the in#ormation o(tained 2nter ret the results

*ll conclusions are tentative and su(5ect to change as new evidence is uncovered &don+t 0R36E things)

Cha ter ! . continued

"he Nature o# Research

"y es o# Research

 E% erimental &most conclusive o# methods)

Researcher tries di##erent treatments &independent variable) to see their e##ects &dependent

variable)
2n sim le e% eriments com are 7 methods and try to control all extraneous variables that

might a##ect outcome

Need control over assignment to treatment and control grou s &to make sure they are

e1uivalent)
$ometimes use single su(5ect research &intensive study o# single individual or grou over

time)

Cha ter ! . continued

"he Nature o# Research

&"y es o# Research continued)

Correlational Research
,ooks at e%isting relationshi s (etween 7 or more varia(les to make (etter redictions

Causal Com arative Research

2ntended to esta(lish cause and e##ect (ut cannot assign su(5ects to trtmt'control ,imited inter retations &could (e common cause #or (oth cause and e##ect8stress causes

smoking and cancer) 9sed #or identi#ying ossi(le causes: similar to correlation

Cha ter ! . continued

"he Nature o# Research

&"y es o# Research continued)

$urvey Research

Determine'descri(e characteristics o# a grou Descri tive survey in writing or (y interview 0rovides lots o# in#ormation #rom large sam les "hree main ro(lems; clarity o# 1uestions- honesty o# res ondents- return rates

Ethnogra hic research &1ualitative)

2n de th research to answer WH< 1uestions $ome is historical &(iogra hy- henomenology- case study- grounded theory)

Cha ter ! . continued

"he Nature o# Research

&"y es o# Research continued)

Historical Research
$tudy ast- o#ten using e%isting documents- to reconstruct what ha Esta(lishing truth o# documents is essential

ened

*ction Research &di##ers #rom a(ove ty es)

Not concerned with generali4ations to other settings Focus on in#ormation to change conditions in a articular situation &may use all the a(ove

methods)

Each o# these methods is valua(le #or a di##erent ur ose

Cha ter ! . continued

"he Nature o# Research

=eneral Research "y es

Descri tive &descri(e state o# a##airs using surveys- ethnogra hy- etc.) *ssociational &goes (eyond descri tion to see how things are related so can (etter understand henomena using correl'causal.com arative 2ntervention &try intervening to see e##ects using e% eriments)

Cha ter ! . continued

"he Nature o# Research

>uantitative v. >ualitative

>uantitative &numbers)

Facts'#eelings se arate World is single reality Researcher removed Esta(lished research design E% eriment rototy e =enerali4ation em hasi4ed

Cha ter ! . continued

"he Nature o# Research

/eta.*nalysis

,ocate all the studies on a to ic and synthesi4e results using statistical techni1ues &average the results) >uestion o# reality &are only individual erce tions o# it) >uestion o# communication &words are su(5ective) >uestion o# values &no o(5ectivity only social constructs) >uestion o# unstated assum tions &researchers don+t clari#y assum tions that guide them) >uestion o# societal conse1uences &research serves olitical ur oses that are conservative or o ressive: reserve status 1uo)

Critical *nalysis o# Research &some say all research is #lawed)

Cha ter ! . continued

"he Nature o# Research

3verview o# the Research 0rocess &Fig. !.?)

2ntroduction cha ter

0ro(lem statement that includes some (ackground in#o and 5usti#ication #or study E% loratory 1uestion or hy othesis &relationshi among varia(les clearly de#ined): goes last in Ch. De#initions &in o erational terms) Review o# related literature &other studies o# the to ic read and summari4ed to shed light on what is already known)

Cha ter ! . continued

"he Nature o# Research

3verview o# the Research 0rocess &Fig. !.?)

/ethods cha ter

$u(5ects &sam le- o ulation- method to select sam le) 2nstruments &tests'measures descri(ed in detail and with rationale #or their use) 0rocedures &what- when- where- how- and with whom):
=ive schedule'dates- descri(e materials used- design o# study- and ossi(le (iases'threats

to validity

?. Data analysis &how data will (e analy4ed to answer research 1uestions or test hy othesis)

Cha ter 7

"he Research 0ro(lem

$tatement o# the 0ro(lem &identi#y a ro(lem'area o# concern to investigate)

/ust (e #easi(le- clear- signi#icant- ethical

Research >uestions &serve as #ocus o# investigation- see . 7@ list)

$ome in#o must (e collected that answers them &must (e researcha(le) Cannot research AshouldB 1uestions $ee diagram- . 7C

Cha ter 7 . Continued

"he Research 0ro(lem

R> should (e #easi(le &can (e investigated with availa(le resources) R> should (e clear &s eci#ically de#ine terms used8o erational needed- (ut give (oth)

Constitutive de#initions &dictionary meaning) 3 erational de#initions &s eci#ic actions'ste s to measure term: 2>Dtime to solve u44lewhere E7F sec. is high: 7F.?F is med.: ?FG is low)

R> should (e signi#icant &worth investigating: how does it contri(ute to #ield and who can use in#o) R>s o#ten investigate relationshi s &two characteristics'1ualities tied together)

Cha ter H

6aria(les and Hy otheses

2m ortant to study relationshi s

$ometimes 5ust want to descri(e &use R>) 9sually want to look #or atterns'connections
Hy othesis redicts the e%istence o# a relationshi

6aria(les &anything that can vary in measure: o osite o# constant) 6aria(les must (e clearly de#ined 3#ten investigate relationshi (etween varia(les

Cha ter H . Continued

6aria(les and Hy otheses

6aria(le Classi#ications &Fig. H.?- . ?7)

>uantitative &varia(les measured as a matter o# degree- using real num(ers: i.e. age- num(er kids) Categorical &no variation8either in a category or not: i.e. gender- hair color) 2nde endent; the cause &aka the mani ulated- treatment or e% erimental varia(le) De endent; the e##ect &aka outcome varia(le) E%traneous; uncontrolled 26s &see Fig. H.7- . ?I)

*ll e%traneous varia(les must (e accounted #or in an e% eriment

Cha ter H . Continued

6aria(les and Hy otheses

Hy otheses J redictions a(out ossi(le outcome o# a study: sometimes several hy otheses #rom one R> &Fig H.H)

R>; Will athletes have a higher =0* that nonathletesK H; *thletes will have higher =0*s that nonathletes Clari#ies'#ocuses research to make rediction (ased on revious research'theory /ulti le su orting tests to con#irm hy othesis strengthens it Can lead to (ias in methods &conscious or un) to try to su ort hy othesis $ometimes miss other im ortant in#o due to #ocus on hy othesis & eer review're lication is a check on this)

*dvantages to stating a hy othesis as well as R>

Disadvantages

Cha ter H . Continued

6aria(les and Hy otheses

$ome hy othesis more im ortant than others Directional v. nondirectional

Directional says which grou will score higher'do (etter Nondirectional 5ust indicates there will (e a di##erence- (ut not who will score higher'do (etter Directional more risky- so (e care#ul'tentative in using directional ones

Cha ter ?

Ethics and Research

E%am les o# unethical ractices

Re1uiring artici ation #rom owerless &students) 9sing minors without arental ermission Deleting data that don+t agree w' hy othesis 2nvading rivacy o# su(5ects 0hysically or sychologically harming su(5ects

*0* statement o# ethical rinci les in research

Each student must sign one and have it signed (y work lace su ervisor

Cha ter ? . Continued

Ethics and Research

0rotecting artici ants #rom harm re1uires informed consent

$u(5ects must know the ur ose o# the study- ossi(le (ene#its'harm: artici ation is voluntary and they can w'draw without enalty any time &Fig. ?.H- . LC)

Researchers should ask; Could su(5ects (e harmedK 2s there another way to get the in#oK 2s the in#o valua(le enough to 5usti#y studyK Researchers must ensure con#identiality o# data &limit access: no names i# ossi(le: tell su(5ects con#idential or anonymous) Deceiving su(5ects is sometimes necessary &/ilgram study)- ask i# results 5usti#y ethical la se

When dece tion used su(5ects they should (e okay with it a#ter &and they can re#use use o# their data)

Cha ter ? . Continued

Ethics and Research

Research with children

0arental consent re1uired &signed ermission #rom arents *0* Ethics in Research Form addresses this also

Regulation o# Research &National Research *ct o# !CM?)

2# #ederal #unding received must have an 2RB to check; risks to su(5ects- in#ormed consent guidelines met- de(rie#ing lans #or su(5ects HH$ made changes in !C@! so that educational research is e%em t under certain conditions

Cha ter L

Review o# the ,iterature

6alue o# the ,iterature Review

=lean ideas #rom others interested in to ic $ee results o# related studies &must (e a(le to evaluated those o(5ectively)

"y es o# sources
=eneral Re#erences J inde%es &o# rimary sources and a(stracts &ER2C- 0sych *(stracts) 0rimary $ources J u(lications where researchers re ort their results & eer reviewed're#ereed 5ournals) $econdary $ources J u(lications where authors descri(e works o# others &encyclo ediastrade(ooks- te%t(ooks)

Cha ter L . Continued

Review o# the ,iterature

$te s in the ,iterature Review &manual or electronic) $ee e%am les . M?

De#ine ro(lem recisely as ossi(le Review some secondary sourcesN Review some general re#erence worksN Formulate search terms &keywords'descri tors) $earch general re#erences #or rimary sources 3(tain and read rimary sources &make notes'summari4e)

N/ay (e (ased on e%isting knowledge or revious reading

Cha ter L . Continued

Review o# the ,iterature

/aking notes

2nclude ro(lem' ur ose: hy otheses'R>: rocedures w' su(5ects'methods: #indings'conclusions: citationO $earching www8(e care#ul o# relia(ility 2ntroduction . descri(es ro(lem and 5usti#ication #or study: Body J discuss related studies together &P7- .@@) $ummary J ties literature together'give conclusions arising #rom literature Re#erence list

$earching strategies8use Boolean o erators &*ND- 3R- N3")

Writing u the ,iterature Review

Don+t re lace a review o# rimary sources with meta.analysis &a com(ined review o# all availa(le research on a to ic w' results averaged)

End 0art !

Cha ter I

$am ling

$am le J any grou on which in#o is o(tained 0o ulation J grou that researcher is trying to re resent
0o ulation must (e de#ined #irst: more closely de#ined- easier to do- (ut less generali4a(le $tudy a su(set o# the o ulation (ecause it is chea er- #aster- easier- and i# done right- get same results as a census &study o# whole o ) *ccessi(le o ulation J the grou you are a(le to realistically generali4e to8 may di##er #rom target o ulation

Cha ter I . Continued

$am ling

&Random v. Nonrandom $am ling) Random J every o ulation element has an e1ual and inde endent chance to artici ate
9ses names in a hat or ta(le or random num(ers Elimination o# (ias in selecting the sam le is most im ortant &meaning the researcher does not in#luence who gets selected) Ensuring su##icient sam le si4e is second most im ortant

Nonrandom' ur osive . trou(les with re resentativeness'generali4ing Cha ter I . Continued

$am ling
&Random $am ling /ethods)

$im le random sam ling

Names in a hat or ta(le o# random num(ers.. .CC ,arger sam les more likely to re resent o .

*ny di##erence (etween o ulation and sam le is random and small &called random sam ling error) Ensures small su(grou s &strata) are re resented Normally ro ortional to their art o# o . Break o into strata- then randomly select w'in strata /ultistage sam ling &see . C?)

$trati#ied random sam ling

Cha ter I . Continued

$am ling

&Random $am ling /ethods- cont.) Cluster random sam ling

$elect grou s as sam le units rather than individuals RE>92RE$ a large num(er o# grou s'clusters /ultistage sam ling &see . C?)

$ystematic &Nth) sam ling

Considered random is list i# randomly ordered or nonrandom i# systematic w' random starting oint Divide o si4e (y sam le si4e to get N & s'ssDN)

Cha ter I . Continued

$am ling
&Non.Random $am ling /ethods)

$ystematic can (e nonrandom i# list is ordered Convenience sam ling

9sing grou that is handy'availa(le &or volunteers) *void- i# ossi(le- since tend not to (e re resentative due to homogeneity o# grou s Re ort large num(er o# demogra hic #actors to see likeliness o# re resentativeness

0ur osive sam ling

9sing ersonal 5udgment to select sam le that should (e re resentative &i.e.- this #aculty seems to re resent all teachers) 3R selecting those who are known to have needed in#o &interested in talking only to those in ower) $now(all is a ty e &used with hard to identi#y grou s such as addicts)

Cha ter I . Continued

$am ling

$am le si4e a##ects accuracy o# re resentation

,arger sam le means less chance o# error /inimum is HF: u er limit is !-FFF &see ta(le) Re resentative sam le is re1uired &not the same thing as variety in a sam le) High artici ation rate is needed /ulti le re lications enhance generali4ation when nonrandom sam ling is used Ecological generali4ation &gen to other settings'conditions- such as using a method tested in math #or English class)

E%ternal validity J how well sam le generali4es to the o ulation

6ideo !M
Cha ter M

2nstrumentation
&/easurement)

Data J in#ormation researchers o(tain a(out su(5ects

Demogra hic data are characteristics o# su(5ects such as age- gender- education level- etc. *ssessment data are scores on tests- o(servations- etc. &the device used to measure these is called the measurement instrument) Where and when will data (e collected How o#ten will data (e collected Who will collect the data

Qey 1uestions in data measurement' instrumentation

Cha ter M . Continued

2nstrumentation

6alidity J measures what it is su osed to &accurate) Relia(ility J a measure that consistently gives same readings &re eata(le) 3(5ectivity J a(sence o# su(5ective 5udgments &need to eliminate su(5ectivity in measuring) 9sa(ility o# instruments
Consider ease o# administration: time to administer: clarity o# directions: ease o# scoring: cost: relia(ility'validity data availa(ility

Cha ter M . Continued

2nstrumentation

By the grou

&Classi#ying Data Collection 2nstruments) roviding the data

Researcher instruments &researchers o(serves student er#ormance and records) $u(5ect instruments &su(5ects record data a(out themselves- such as taking test) 3thers'2n#ormants &Hrd arty re orts a(out su(5ects such as teacher rates students) 0re#erence is #or e%isting ones &www.ericae.net- //< Can develo your own &re1uires time- e##ort- skill- testing: see . !7L) Written res onse J re#erred J o(5ective tests- rating checklist 0er#ormance instruments J measure rocedure- roduct

By where instrument came #rom

By res onse ty e

Cha ter M . Continued

2nstrumentation
&E%am les o# Data Collection 2nstruments)

Researcher Com leted 2nstruments

Rating scales &mark a lace on a continuum #or e%am le numeric rating !D oor to LD e%cellent) 2nterview schedules &com lete scales as interview takes lace: use recoding: (eware o# dishonesty) "ally sheets &#or counting'recording #re1uency o# (ehavior- remarks- activities- etc.) Flow charts &to record interactions in a room) *necdotal records &need to (e s eci#ic and #actual) "ime'/otion logs &record what took lace and when)

Cha ter M . Continued

2nstrumentation
&E%am les o# Data Collection 2nstruments)

$u(5ect Com leted 2nstruments

>uestionnaires &1uestion clarity to reader essential) $el# checklists *ttitude scales &,ikert is one ty e- how much su(5ect agrees'disagrees with descri tive statements a(out a to ic indicates a ositive'negative attitude toward to ic) $emantic di##erential &good'(ad: oor'e%cellent ratings) 0ersonality ro#iles *chievement'* titude tests 0er#ormance tests 0ro5ective devices &Rorschach 2nk Blot "est) $ociometric devises & eer ratings)

Cha ter M . Continued

2nstrumentation

2tem Formats

$election items or closed res onse &"'F: <es'No: Right'Wrong: /ulti le choice) $u ly items or o en ended &short answer: essay) 9no(trusive measures &no intrusion into event8 usually direct o(servation and recording) Raw scores &initial score or count o(tained8w'out conte%t) Derived scores &raw scores translated to meaning#ul usage with standardi4ed rocess)
*ge'=rade e1uivalence: 0ercentile ranks: $tandard scores &how #ar a score is #rom a given re#erence ointi.e. 4 and " scores): Which to use de ends on the ur ose: usually standard scores used

"y es o# $cores

Cha ter M . Continued

2nstrumentation

Norm Re#erenced v. Criterion Re#erenced "ests

Norm re#erenced scores give a score relative to a re#erence grou &the norm grou ) Criterion re#erenced scores determine i# a criterion has (een mastered
"hese are used to im rove instruction since

they indicate what students can or

cannot do

or do or do not know

Cha ter M . Continued

2nstrumentation

&/easurement $cales)

Nominal &in name only)

Num(ers are only name tags- they have no mathematical value &gender; !Dmale and 7D #emale 3R race; !D Blk7DWht- HDother) Num(ers show relative osition- (ut not 1uantity &grade level- #inishing lace in a race) Num(ers show 1uantity in e1ual intervals- (ut an ar(itrary 4ero &can have negative num(ers: degrees C or F) Num(ers show 1uantity with (ase o# 4ero where 4ero means the construct is a(sent

3rdinal &in name- lus relative order)

2nterval &in name w' order *ND e1ual distance)

Ratio &in name- w' order- e1. distance *ND a(solute 4ero)

Higher levels more recise8collect data at highest level ossi(le: some statistics only work with higher level data

Cha ter M . Continued

2nstrumentation

&0re aring #or Data *nalysis) $coring data J use e%act same #ormat #or each test and descri(e scoring method in te%t "a(ulating and Coding J care#ully trans#er data #rom source documents to com uter
=ive each test an 2D num(er *ny words must (e coded with numerical values Re ort codes in te%t o# research re ort

6ideo !@
Cha ter @

6alidity and Relia(ility

&>uality o# instruments is im ortant)

6alidity is most im ortant as ect o# measures

/eans accuracy- correctness- use#ulness o# instrument 6alidation is the rocess o# collecting and analy4ing evidence to su ort in#erences (ased on an instrument "est u(lishers usually give a statement o# intended use as well as evidence to su ort validity

Relia(ility &consistency in scoring) is art o# validity

Cha ter @ . Continued

6alidity and Relia(ility

&"hree ways to esta(lish validity) Content validity J is entire content o# construct covered (y test- are im ortant arts em hasi4edK

Esta(lished (y e% ert 5udgment Facial validity is art o# this

Criterion validity J is there consistency (etween the instrument and some redicted or concurrent criterionK

Esta(lished (y em irical evidence using validity coe##icient &.! to G! scores) Correlate scores o# the test with the criterion &$*" and =0* in college)

Cha ter @ . Continued

6alidity and Relia(ility

&"hree ways to esta(lish validity)

Construct validity J Does the measure correctly identi#y those with di##erent levels o# the construct
Esta(lished with em irical evidence Correlate scores on test with known indicator o# the construct & risoners score low on test o# ethics)

6alidity ro(lems come #rom systematic error &also known as (ias8 something the research did wrong)

Cha ter @ . Continued

6alidity and Relia(ility

Relia(ility means that scores are consistent #rom one time measuring to the ne%t Can have a relia(le measure that may not (e valid /ust (e relia(le to (e valid
$ee . !II- target shooting

Errors o# measurement J there is always some variation #rom measure to measure

,ook at relia(ility coe##icient to determine relia(ility

Cha ter @ . Continued

6alidity and Relia(ility

&"hree ways to esta(lish relia(ility)

"est'Retest J give the same test &o# enduring trait) to the same eo le at two times and correlate the scores E1uivalent #orms J give two arallel #orms o# a test to the same eo le and correlate scores 2nternal consistency J several methods
$ lit halves &score two halves o# test and correlate scores) QR.7! and Cron(ach *l ha J Correlate each item to overall score

Cha ter @ . Continued

6alidity and Relia(ility

$tandard Error o# /easurement J variations in measurement result in some error which is re orted $coring *greement J #or su(5ective tests or direct o(servations &check o# internal relia(ility) 6alidity and Relia(ility should (e addressed in all research &including 1ualitative) Cha ter C

2nternal 6alidity
&"he 26 really caused a change in the D6)

"hreats
$u(5ect characteristics'selection (ias J when su(5ects in study or in trmt'cont grou s di##er #rom each other &on age- gender- a(ility- etc) ,oss o# su(5'/ortality J must address 1uestion o# whether those dro ing out are di##erent than those not ,ocation'E% eriment varia(les J characteristics o# the school- classroom- etc. may (e inter#ere with the cause'e##ect relationshi &kee constant #or (oth grou s)

Cha ter C . Continued

2nternal 6alidity
&"he 26 really caused a change in the D6)

"hreats &continued)
2nstrumentation J need constant a

lication and scoring o# instruments

2nstrument decay J when scoring varies due to #ati1ue Data collector characteristics &age- gender- etc.) in#luence results) 8 use same collector or

randomly assn Data collector (ias J unconscious or conscious distortion o# data &use single or dou(le (lind techni1ue)

L. "esting J retest sensiti4ation can occur or su(5ects can #igure out acce ta(le answers

Cha ter C . Continued

2nternal 6alidity
&"he 26 really caused a change in the D6)

"hreats &continued)
History J an e%ternal occurrence that inter#eres with relationshi (etween 26 and D6 /aturation J changes in relationshi (etween 26 and D6 due to assage o#

time'growth o# su(5 *ttitudes o# $u(5ects J Hawthorne or guinea ig e##ects- novelty e##ects and demorali4ation may occur Regression &toward the mean) J ,ow scorers do (etter in su(se1uent tests: high scorers do worse 2m lementation J e% eriment di##ers #or grou s

Cha ter C . Continued

2nternal 6alidity
&"he 26 really caused a change in the D6)

How to minimi4e threats;

$tandardi4ed conditions Collect and re ort demogr characteristics o# su(5 2denti#y're ort details o# study $elect a design to minimi4e e##ects &true randomi4ed e% erimental designs are (est)

$ee age !@C- Fig. C.!F #or threats summary

End 0art 7

Cha ter !H

E% erimental Research

/ost ower#ul design 9sed to esta(lish cause and e##ect (y mani ulating &in#luencing) an 26 &inde endent varia(le- aka treatment or e% erimental varia(le) to see its e##ect on a D6 &de endent varia(le &aka criterion or outcome varia(le) =oes (eyond descri tion and rediction Cha ter !H . Continued

E% erimental Research

&Characteristics o# E% erimental Research) Com arison o# grou s &at least two grou s o# su(5ects- called treatment and control grou s) /ani ulation o# the 26 &e% erimenter changes something #or the treatment grou that+s di##erent than the control grou ) Randomi4ation &true e% eriments re1uire random assignment into treatment'control conditions8 a#ter random selection o# su(5ects to artici ate in study)

*ssignment takes lace at start o# e% eriment Do not use already #ormed grou s =rou s should (e e1uivalent &any di##erences due to chance) Randomi4ation eliminates threats #rom extraneous variables =rou s must (e su##iciently large to (e e1uivalent

Cha ter !H . Continued

E% erimental Research
&Control o# E%traneous 6aria(les)

*ll e%traneous varia(les must (e controlled to eliminate threats to validity'rival hy otheses

Ensure grou s are e1uivalent to (egin using randomi4ation Hold certain varia(les constant &i.e. age- 2>) or (uild them into to the design 9se matching when necessary 9se su(5ects as their own controls &treat same grou #irst in control condition then in treatment 3R use re.test' osttest on same grou ) 9se analysis o# covariance to statistically e1uate une1uivalent grou s

Cha ter !H . Continued

E% erimental Research
&=rou Designs)

Weak Designs

3ne $hot Case $tudy &R 3)

3ne grou e% osed to treatment then D6 is measured No controls E%am le; "ry new teaching method then see how students do on ost test

3ne =rou 0retest.0osttest Design

&3 R 3)

*dds a retest (ut no control grou Need control #or di## su(5 characteristics

R! 3 R7 3 $tatic =rou 0retest'0osttest Design &adds a retest)

$tatic.=rou Com arison Design

Cha ter !H . Continued

E% erimental Research
&=rou Designs)

True Experimental Designs

Randomi4ed 0osttest 3nly Design R R! 3 &random assign to trtmt'cntrl- then osttest) R 3 Randomi4ed 0retest'0osttest Control =rou R 3 R ! 3 &controls history- maturation- etc.) R 3 R7 3

Randomi4ed $olomon ?.=rou Design com(ines the a(ove two &eliminates testing threat: ro(lem is num(er o# su(5ects needed) Random *ssignment w' /atching

/atch airs on #actors that in#luence D6 then randomly assign to treatment or control &su(5ects limited (y no match elimination) $tatistical matching can (e done using redicted scores

Cha ter !H . Continued

E% erimental Research
&=rou Designs)

Quasi Experimental Designs

/atching only J di##erent #rom random assignment w' matching &uses e%isting grou s)

/atch su(5ects in trmt and cntrl grou s on known e%traneous varia(les 2# ossi(le- use multi le grou s- and randomly assign them

Counter(alanced J Each grou e% osed to all the same treatments (ut in di##erent order "ime series J Re eated treatments and o(servations over a eriod o# time &(oth (e#ore and a#ter treatment) Factoral designs J /ulti le 26s or D6s investigated simultaneously &i.e. look #or interactions (etween 7 26s)

Cha ter !H . Continued

E% erimental Research
&Controlling "hreats to 2nternal 6alidity)

$ee "a(le !H.!- . 7@? #or advantage'disadv. o# each design "o evaluate the likelihood o# a threat to internal validity in e% eriments ask;

What are the known e%traneous #actorsK Do the grou s di##er on themK How were they controlledK $ee . 7@@.7@C 1uestions to evaluate u(lished article $ee evaluation o# selected article on . 7CF.7CC

Researchers need tight control #or e% eriments to (e success#ul

Cha ter !L

Correlation Research
&0redicting 3utcomes "hrough *ssociation)

Correlational research involves study o# e%isting relationshi s (etween two varia(les

Descri tive in nature 3#ten a recursor to e% erimental research 0ositive correlation is Hi'Hi and ,o',o &coe##. Gr) Negative correlation is Hi',o and ,o'Hi &.r)

0ur ose is to e% lain relationshi s or to redict outcomes Cha ter !L . continued

.Correlation

Research

&0redicting 3utcomes "hrough *ssociation) E% lanatory studies e%amine relationshi to identi#y ossi(le cause'e##ect

Relationshi might or /2=H" N3" mean causation For causation; !) * (e#ore B: 7) * and B related: H) Rule out other causes o# B &need e% eriment) $catter lots with regression line'e1uation redicts scores numerically "he stronger the correlation the (etter the rediction

0rediction studies identi#y redictors o# criterions &i.e. H$ =0* and College =0*)

Cha ter !L J continued

Correlation Research

&0redicting 3utcomes "hrough *ssociation) Com le% Correlation "echni1ues- such as multi le regression allow use o# several redictors #or one criterion
Coe##icient o# multi le correlation &R) gives strength o# correlation (etween redictors and criterion Coe##icient o# determination &r7) is amount % and y vary together Descriminant #unction analysis is #or non.1uantitative criterion & redict which grou someone will (e in) 3ther techni1ues also used &#actor analysis- ath analysis- structural modeling)

Cha ter !L . continued

Correlation Research

&$te s in the rocess) 0ro(lem selection J usually it+s are % and y related or how well does redict c $am le J random selection o# at least HF /easurement J need 1uantitative data Design'0rocedures J need two measures on each su(5ect Data collection J usually (oth measures close in time Data analysis J correlation coe##icient- r- and lot &r is .! to G!- and the closer to lus or minus !- the stronger the relationshi )

Cha ter !L . continued

Correlation Research
&2nter reting Correlation Coe##icients)

=eneral guideslines;

G.ML to G!.F 6ery strong relationshi G.LF to G.ML /oderate strong relationshi G.7L to G.LF Weak relationshi G.FF to G.7L ,ow to no relationshi

Need .L or (etter #or rediction o# any use- and .IL #or accurate redictions Relia(ility coe##icients should (e .M u 6alidity coe##icients should (e .L u Cha ter !L . continued

Correlation Research

&"hreats to 2nternal 6alidity in Correlation Research) Remem(er correlation is not causation &lurking varia(les) $u(5ect characteristics J may get di##erent correl w' di##erent a(ility levels- gender- etc. &can control with artial correlation) ,ocation J testing conditions can im act results 2nstrumentation ro(lems J hel s to standardi4e instrument and data collection #or (oth grou s

"esting J retest inter#erence and sensiti4ation ossi(le /ortality J (e care#ul i# have large loss #rom one grou (eing tested

Cha ter !L . continued

Correlation Research
&>uestions to ask to avoid threats to internal validity)

What #actors could a##ect the varia(les (eing studiedK Does any #actor a##ect B3"H varia(lesK &this is where threats occur) Figure a way to control any lurking varia(les

Cha ter !I

Causal Com arative Research

&E% 0ost Facto)

Determines cause &or e##ect) that has occurred and looks #or e##ect &or cause) #rom it

$tart w' di##erences in grou s and e%amine them E%am les; Di##erence in math a(ilities o# male'#emale stu

No random assignment to treatment &it already occurred) *ssociational like correlation (ut rimarily interested in cause'e##ect 26 either cannot &ethnicity) or should not &smoking) (e mani ulated

Cha ter !I . continued

Causal Com arative Research

&E% 0ost Facto)

3#ten an alternative to e% erimental &#aster and chea er) $erious limitation is lack o# control over threats to internal validity Need to remem(er the cause may (e the e##ect: they may only (e related and there is some other varia(le that is the cause &lurker)
Remem(er three canons o# causation

Cha ter !I . continued

Causal Com arative &CC) Research

&CC versus Correlational Research)

Both are associational &looking #or relationshi ) Both are o#ten relude to e% eriments Neither involves mani ulation o# varia(les CC works with di##erent grou s: correl e%amines one grou on di##erent varia(les Correlation is measured w' coe##icient while CC com ares means'medians' ercents o# grou mem(ers Cha ter !I . continued

Causal Com arative &CC) Research

&CC versus E% erimental Research)

Both com are grou scores o# some ty e 2n e% erimental the 26 is mani ulated- (ut not in CC &already took lace) CC does not rovide as strong evidence as e% erimental #or cause and e##ect Cha ter !I . continued

Causal Com arative &CC) Research

&$te s in CC Research) 0ro(lem #ormation J identi#y henomena and look #or causes or conse1uences o# it

$ometimes several alternate hy otheses investigated

$am le J de#ine &o erationally) characteristics o# study care#ully- then select individuals who ossess

=rou s should (e homogeneous in regard to several im ortant varia(les &to control #or them as causes) then match control'e% grou s on one or more varia(les &smoking study matched on !C varia(les)

2nstruments J use any ty e to com are the grou s Design J (asic CC involves 7 or more gr s that di##er on varia(le o# interest &(asic design is one grou ossesses trait &athlete) other doesn+t com are D6 &=0*)

Cha ter !I . continued

Causal Com arative &CC) Research

&"hreats to 2nternal 6alidity in CC Research)

$u(5ect characteristics J since don+t select su(5ects and #orm grou s- there may (e unidenti#ied lurking varia(les
Can use matching to control #or any identi#ied di##erences- (ut limits sam les si4e Can #ind or create homogeneous grou s &#or e%am le com are only high =0* students to other high =0* students) on attitudes toward % $tatistical matching J ad5usts osttest scores (ased on some initial di##erence

3ther threats J location- instrument- history- maturation- loss o# su(5ects can (e concerns

Need to control as many as ossi(le to eliminate alternate hy otheses

Cha ter !I . continued

Causal Com arative &CC) Research

&Evaluating threats to 2nternal 6alidity in CC Research)

>uestions to ask

What #actors are known to a##ect the varia(le (eing studiedK What is the likelihood the com arison grou s di##er on these #actorsK How well did the design identi#y and control #or theseK

For e%am le consider su(5ect characteristics such as socioeconomic status- gender- ethnicity- 5o( skills: mortality rates in grou s: location &schools di##er): instrument &di##errent data collectors and' or (iases)

Data *nalysis in CC J o#ten com are means o# grou s: with 7 categorical use crossta(s &cross(reak ta(les) to com are ercents (y grou s "e%t gives e%am le study

Cha ter !M

$urvey Research
&9sed to descri(e what eo le think'do'(elieve)

"y es
Cross sectional rovide a sna shot in time ,ongitudinal collect data at di##erent oints in time to study changes over time
"rend study . random sam le each year on same to ic Cohort study . sam le #rom same cohort mem(ers year a#ter year 0anel study . same individuals surveyed year a#ter year &mortality a ro(lem over long time

eriods)

3#ten surveys are the data collection instrument in correlation &or cc'e% +l) studies Cha ter !M . Continued

$urvey Research
&$te s to conduct survey research)

De#ine the ro(lem

Needs to (e im ortant enough res ondents will invest their time to com lete it /ust (e (ased on clear o(5ectives

2denti#y the target o ulation

De#ined (y sam le unit or unit o# analysis 9nit can (e a erson- school- classroom- district- etc.) $urvey a sam le or do a census o# the o ulation

Cha ter !M . Continued

$urvey Research
&$te s to conduct survey research)

/ethods o# data collection

Direct administration to a grou &such as at a meeting) . good res onse rate- limited generali4. /ail survey &ine% ensive way to get large amount o# data #rom wides read o ) . lower res onse rates- not in.de th in#o- illiterate missed "ele hone survey &chea '#ast) . res onse rates higher due to encouragement &A2+m not selling8B): miss some o mem(ers- interviewer (ias ossi(le 0ersonal interviews &#ace.to.#ace has good res onse rate (ut time and cost high) . lack anonymity- interviewer (ias

Cha ter !M . Continued

$urvey Research
&$te s to conduct survey research)

$elect the sam le &randomly- (ut check to see res ondents are 1uali#ied to answer)
0ilot test can indicate likely res onse rate and ro(lems with data collection or sam le

0re are instrument &1uestionnaire and interview schedule)

 * earance im ortant . look short and easy Clarity in 1uestions is essential

Cha ter !M . Continued

$urvey Research
&$te s to conduct survey research)

>uestion ty es &same 1uestions need to (e asked o# all res ondents)

Closed ended &multi le choice) . easier to com lete- score- analy4e
Categories must (e all inclusive- mutually e%clusive

3 en ended . easy to write- hard to analy4e and hard on res ondents $ee e%am les . ?FH

Cha ter !F

Descri tive $tatistics

&"ools to summari4e data)

Descri tive statistics descri(e many scores with 5ust one or two indices &such as mean or median)

$am le o# a o is descri(ed w' indices called statistics Entire o is descri(ed w' indices called arameters >uantitative data J scales measure how much &test scores- amount o# money s ent- etc.

"y es o# data &words or num(ers)

2nterval- Ratio- and sometimes 3rdinal- varia(les Nominal and sometimes 3rdinal- varia(les

Categorical data J total num(er o# o(5ects in a category &ethnicity- gender- etc.)

Cha ter !F . Continued

Descri tive $tatistics

&$ummari4ing >uantitative Data)

Fre1uency distri(utions or ta(les show the layout o# the data &see te%t e%am le . 7F!)
Fre1uency olygons J shows where most scores are and how s read out data are
0ay attention to sha e & ositive- negative skews) Normal curves J smoothed olygons J most scores in the center- #ewer in the tails J many

varia(les #ollow a normal sha e &height- weight- age- etc.) Normal curves are the #oundation #or in#erential statistics

Cha ter !F . Continued

Descri tive $tatistics

&$ummari4ing >uantitative Data)

*verages J measures o# o# central tendency

"hree indices tell what is a ty ical score

/ode J most #re1uent score /edian J middle score &LFth ercent) /ean J takes into account all scores $ee e%am le . 7FL'7FI

Which to use de ends on what you are trying to show

$ reads J measures o# variation or dis ersion

"hree indices tell how closely scores cluster together

Range &highest J lowest): a crude indicator o# s read $tandard deviation &average distance o# each oint #rom the mean)

$maller $D means less s read out- larger one means more s read out

>uartiles- ercents- 2>R- (o% lots

$D and normal curves8I@'CL'CC.M rule

Cha ter !F . Continued

Descri tive $tatistics

&$ummari4ing >uantitative Data)

$tandard scores and the normal curve

$tandard scores use a common scale #or all scores 4 scores are sim lest J tell how #ar #rom the mean in $D units
$core on mean then 4DF: score ! $D a(ove then 4D!.F: !$D (elow then 4D.!.F- etc. 9se mean and $D to calculate 4 scores so you can com are a les'oranges & . 7!F) S D any score J mean

standard deviation

Cha ter !F . Continued

Descri tive $tatistics

&$ummari4ing >uantitative Data)

0ro(a(ility (ased on 4 scores

*ll scores in normal distri(ution are e1ual to !FFT * 4.ta(le gives ercent o# scores #rom any score to the mean &* endi%- . *.?'L) "he ro(a(ility #or getting higher or lower than any given score can then (e calculated

".scores are o#ten used (ecause negative 4 scores awkward &all ".scores are ositive)

/ulti ly 4 times !F- then add LF & . 7!7 "a(le !F.!L) $tandard test scores o#ten given with ".scores and ercents a(ove'(elow the given score

Note8use 4 and " scores only with N3R/*, distri(utionsO

Cha ter !F . Continued

Descri tive $tatistics

&$ummari4ing >uantitative Data)

Correlation e%amines relationshi s (etween two 1uantitative varia(les &interval'ratio data)

$catter lot shows the relationshi visually
9se it to check #or attern in data &hi'hi or hi'loK)
2# linear attern- can us 0earson+s r coe##icient

9se it to look #or strength &scatteredness) 0ay attention to outliers & . 7!L'7!I e%am les)

Correlation coe##icient is a numerical indicator or strength o# the relationshi

 0earson+s m &r) is #or linear data &.! to G!) Eta is #or curved data

Cha ter !F . Continued

Descri tive $tatistics

&$ummari4ing Categorical Data)

Fre1uency ta(les
=ive ercents #or ease in inter reting

Cross(reak or crossta(ulations #or relationshi s &26 goes on the sidethen give row ercents) Bar charts and ie charts used
Bars #or ordered categories 0ies #or unordered categories

Cha ter !!

2n#erential $tatistics

2n#erences a(out a o ulation (ased on data #rom a sam le *nswers 1uestions a(out how likely a sam le is to re resent some arameter a(out a o ulation 2n#erential test used de ends on the level o# data &1uantitative or categorical) Cha ter !! . Continued

2n#erential $tatistics
&"he logic o# in#erential statistics)

$am ling error

$am les di##er #rom their arent o ulations &no two sam les are the same) Di##erence is called sam ling error ,arge collections o# random sam les o# at least HF #ollow a normal curve attern 2ts mean &mean o# means) is the mean o# the o ulation 2ts $D &$D o# means) is the standard error o# the mean &$E/)

Distri(ution o# sam ling means &the sam ling distri(ution)

Cha ter !! . Continued

2n#erential $tatistics
&"he logic o# in#erential statistics)

$tandard error o# the mean &$E/)

2t+s the $D o# the sam ling distri(ution $ince distri(ution is normal- then G!$E/ has I@T o# cases: G7$E/ has CLT: GH$E/ has CC.MT
3nce we can estimate the mean and $D o# the sam ling distri(ution can determine how likely it is that a articular sam le mean came #rom that o ulation i.e. /ean o# o D!FF- $DD!F and draw a sam le with a mean o# !!F- yes could (e #rom that o 8(ut i# draw a sam le with a mean o# !?F- most likely N3" #rom that o 8since is G?$E/ #rom the mean &almost 4ero ro(a(ility)

E% ress means as 4 scores: a 4 score move that 7$E/ is going to occur less than LT o# the time &7.LT each side)

Cha ter !! . Continued

2n#erential $tatistics
&"he logic o# in#erential statistics)

Estimating the $E/

2t is estimated #rom the $D o# the sam le- ad5usted #or sam le si4e; $E/D$D'Un.!

Con#idence 2ntervals &C2)

9se the $E/ to indicate (oundaries CLT o# the time a o mean will (e within G7 $E/ #rom the sam le mean &actually G !.CI $E/) 2# sam le mean 2>D@L &V $E/D7) then CLT o# the time the o mean 2> will (e @LG!.CI&7) or @L GH.C7 which is @!.F@ to @@.C7: CCT C2DMC.@? to CF.!I Can (e CLT con#ident that true o mean is @!.F@.@@.C7

Cha ter !! . Continued

2n#erential $tatistics
&"he logic o# in#erential statistics)

0ro(a(ility is a redicted occurrence such as L in !FF times &LT or .FL)

2n revious e%am le- the ro(a(ility o# the o ulation mean (eing outside the CLT C2 &o# @!.F@ to @@.C7) is LT

9sually com aring more than one mean

E%amine di##erence in 7 sam le means to see i# how likely the di##erence in the sam le is to re resent a true di##erence in the o ulation8is it due to a true di##erence in the o or only due to sam ling error "he $E/ o# the di##erence (etween sam le means- called the $ED or standard error o# the di##erence is used and w'in G!$ED is I@T: G7 $ED is CLT: GH $ED is CCT

Cha ter !! . Continued

2n#erential $tatistics
&Hy othesis "esting)

* hy othesis is a redicted relationshi

9sually com aring means- ro ortions- or looking #or correlations (etween grou s "he heart o# in#er. stats8is the relationshi #ound in the sam le most likely due to a relationshi in the o - or 5ust due to random sam ling errorK

"he null hy othesis is stated and tested THE NULL ALWAYS SAYS THE E !S NO ELAT!"NSH!# " $!%%E EN&E''' Cha ter !! . Continued

2n#erential $tatistics
&Hy othesis "esting)

Research hy othesis is what you really think is going on: o

osite o# the null

E%am le o# hy othesis test

HF &null) is that mean!Dmean7- meaning the mean scores are e1ual 3R the di##erence (etween the mean scores is F "he distri(ution #or a di##erence o# 4ero (etween the means is a normal curve centered on 4ero *s di## (etween means gets larger- meaning #urther #rom the center &in $E/ units)- the more likely it is to re resent a true di## in the o means 2# the ro( is .FL or less- re5ect null8called a statistically signi#icant di##erence &some #ields use .F! or .FF!)

Cha ter !! . Continued

2n#erential $tatistics
&Hy othesis "esting 0rocess) $tate the research hy othesis &Ha or Hr) $tate the null &HF) &Remem(er N3) 3(tain the sam le statistics &means- ro ortions- correlations) Determine the ro(a(ility o# getting the sam le results 5ust (y chance i# the null is true $mall ro(a(ility & E.FL) means re5ect null: there is a signi#icant di##erence &or correlation) in o . ,arge ro(a(ility & W.FL) means do not re5ect: there is no signi#icant di##erence &or correl) in o . Note; Just (ecause #inding is statistically signi#icant does not mean it is a ractical di##erence &given a large enough sam le most are signi#icant)

Cha ter !! . Continued

2n#erential $tatistics
&Hy othesis "esting)

3ne tailed versus two tailed tests

When literature strongly indicates the need #or directional hy othesis then do a one.tail 2n a one tail all LT is on one side &7.tailed cuto## is !.CI$D while ! tailed cuto## is !.IL)

"y e 2 &al ha) versus "y e 22 error

$ee Figure !!.!I- . 7?F "y e 2 J re5ect true null: "y e 22 J acce t a #alse 2nversely related errors

Cha ter !! . Continued

2n#erential $tatistics

&2n#erence "echni1ues) 0arametric tests &#or 1uantitative 2'R data #rom normal distri(utions o# sam le si4e HFG)

t.tests com are means o# two grou s &can (e inde endent or correlated' aired sam les) *N36* tests com are means o# two or more grou s &use ost hoc) Correlations t.test &with com uters 5ust use signi#icance o# r) /ann Whitney 9 com ares ranks o# two grou s Qruskal Wallis 3neway *N36* com ares ranks o# two lus grou s Chi.s1uare test &com ares ro ortions)

Non arametric tests &#or categorical data and 2'R #rom non.normal o s or small sam les)

0ower o# tests J use arametrics and increase sam le si4e

Cha ter !7

$tatistics in 0ers ective

roaches to research
Either 7 or more grou s com ared 3R varia(les in ! grou studied *ND data are either categorical or 1uantitative

Com aring grou s on 1uantitative data

Can com are #re1 distri(utions &histograms)- m. o# center- and m. o# s read 3R all three 2nter retation J im roves with e% erience8need to know when something statistically signi#icant is not ractically signi#icant Calculate e##ect si4e . look at si4e o# di##erence or delta X8i# it is greater than .L- ractically signi#icant 9se in#er. stats 5udicially aying attention to si4e o# di##. and sam le si4e and method it is (ased on

Cha ter !7 . continued

$tatistics in 0ers ective

Relating varia(les within grou w' 1uant data

$catter lot and correl coe## J e%amine lot care#ully Beyond signi#icance ay attn to si4e o# r and es ecially to r.s1uared E%amine how sam le data collected 9se #re1 and ercent in cross(reak ta(les ,ook at summary stats care#ully and ay attn to sam le si4e

Com aring grou s w' categorical data

Relating varia(les within a grou with categorical data J use one sam le chi. s1uare

Cha ter !7 . continued

$tatistics in 0ers ective

Reca

9se gra hics and num(ers 0ay attention to outliers 0ay attention to magnitude o# di##erences 9se in#erence tests #or generali4ing ur oses and e%amine sam ling 9se multi le techni1ues and C2s