eee Kare POT RCN aL aL ens De ee rey Perc nee he ree profesional” mode aimed at complex behavioral change. The evasion peer ree re ett an eee Pee ee een ne era seer ek ent ee ted Car eer eee ern ee eet ee et nt Arn eee eer eerie earn) eee ee er ee Esra ener en een ey eee sir fie ipanonin cr ee me i mennesenn ein ner aero a need ee Prnepeetai nai earn, tabi ener a ee eee Enna mv oar ees eee eT Pema one nee ee Ye erin var ei rim neat peer tbe eae scarcer desk Err etait nen TROT A Guide to : Implementation ResearchA Guide to . Implementation Research from the Ueban Ii Digest of Sova Eperioents, Thi ALAN WERNER and Mark Shroder iam, by David Greenberg Pic nt Acton: Implementation. Refer, edited by Mary Clare Lennon and cited by David Greenberg, we The Next At edited by Alan Welland TH URBAN INSTITUTS PRESS on ‘Washington,DC.amin han i: lhe xy et ts recat ting pcp cng roomate Indade toppled pA) Its Resane Che Meee aby Won Ninn fre MB tie uRnan ivstrrUTE | ecomens end edscational organi. BIBI cassie in Washing ton, DC, in 1968 te staff invents the social, economic, and governance prob lems confronting the nation and eats the public and private meen to lleite them The nite dismiss research findings through pulcatonsts web st, themed, eminar nd forums “Through wor tht range rom broad conceptual studies o administrative and technical asistance, lsat searchers contribute the stock of knowledge va ‘He oie decisionmaking inthe public Conclusions o opinions expressed in Inaitate publetons are those of the thors and do not neces eect the views of fcr or trusts the lst, ior group, or ay organs hat roid nani suport tothe estateContents Preface ix J Iroduction to Implementation Research 1 2D Data Needs, Soures and Colston Strategies ” 3; Pocumentng Implementation 8 A Asesngand Explaining Implementation un References ww About the Author ast Index 1Preface Avi to Implementation Research was conceived inal 52 ‘companion volume tothe clletion of ths inchuddin oiy Jno Aton (Lennon ad Corbet 203). The books weet comple sent each other, withthe guidebook intended as a more practical ‘xpreion ofthe points nd perspectives raed inthe ear ame of Sl, However each work stands om own, Both books arose ost ff series of eonfeenses organize by Tom Corbet ofthe University of {Waconsns Instat for Research on Poerty IRF ad Baara Blum of Colnbia Universtiy’ Recuch Forum on Children Faia the ‘eo Fdesalis atthe National Cente for Children Beginning in 199, the conferences brought together practitioners of implementation rear rom universes and search costing fms, swell &cO0- Somers of evalaton research fom sate wells agence. The focus ofthe confrencs was toler abot recent research efforts to describe and evaluate the progress made by tes and loa in Implementing national wellae reform, but tet rer purpose wa 0 consolidate advances made in implementation rcarch on soil pro trams. With support fom the Offie of the Assistant Secretary fo" Plan fing and Evaluation of the US. Deparment of Heath and Henan Service and from the Joyce Foundation several ofthe conrence pt "eipants were comonsioned to wre ales that explore approaches to implementation research and thir thorealbases and asics thiesfuiness to policymakers and progeam practitioners. These articles ‘were organized and published in Poly in Actin ‘The onganlzersofthe implementation conferences asked meto develop & practi guikbook or student and resezchers engaged Inimplemey latin ecarch a wellas or consumer ofimplerentaton research Ths book the ea A Guide to mpleometaion Rssarh draws onthe article Ply no “Action, aswel a on recent policy restrch and program evlaction Because arose out of the confrencesand because of iy esearch back ‘round the Book focuses on sue inthe design and implementation of weliare reform poisis and programs to ilsatespectceseatch eh Digues and strategies. Those techniques and stratepes, however bane broader application to research on range of social progres To sat "eadr les far with te context wolfe poly and walla rer, chapter includes bref introduction to the challenge of denigning and implementing welfare reform in the cucrent policy context ‘would ike to ada inal not bout theory Altough the Bock «ses several theories underlying specific approaches to implement. sion research it does not isl est ona particu theory. This devon Sessa judgment about che usefulness of theory in informing imp ‘entation research asi is resolution tallow the broadest posable pplication ofthe steaeges ad techniques presente. As such the Book is organiged around what I fd be the core questions or implemen tation research: "What ihsppening?"Is it what is expected or dsied?™ and"Whyisit happening ts" Alan Werner ‘Cambridge, Massachusetts Introduction to Implementation Research tic polcmars and program manages ae responsible or fotvely and eficlenly axing community eSources 0p ms ses info upp the socal goals Ertan esearch po nase hey make This book explores how oe tpe of el Seceone 1 ZimennatonFoareh can as hose designing ad ‘operating social programs ‘What Is Implementation Research? Ce ean i ms qn ped nthe propa ‘But although implementation studies ‘What i happening?” but also "I it what is expected or desired?” an “Why sit happesing ati?Comprehensive ealutions of socal programs often include both ties Within his research content nile. ‘eniation sais can have mulipe purpose, such x supporing tee ure of the program being ings overtime or aeons ‘nd explining the pater of mpi Program ties lations hat include bth mpct an implementation stadie are ‘ually undertaken when policymakers, program adiiniees soa Policy exper are unersin about whether a given prorat ox peg {novation will work. Often, however, new programs oy poles ae implemented on thesis of exeetive or legilatve manda chek say reflect some combination of chaning public sttudes a eee and koowdedge already established through pie peace and woe ‘These mandates oblige federal, state and local pench Cerone ey "rogram managerto implement ne progr o ome changes {iting programs. Particularly when the mandated choc me ae sive and/or let othe creation of new programs he biggest cones maybe tg the programs"upndrunning"and mnkineeeh ee instances in particular, implementation esearch sepeete ftom impact sty may be warranted and desirable ‘The core mission of implementation research to describe, assess and ln “wat is happening and why” maybe spel competing wen browpht to bea onthe flowingmsor acs of pga eae hat are te progr gol cancepe, and desig? Ar they based on sound theory and practi, and fot in what respect Sac programs mandated by Congress bu implemented at he sate or loca ew are often launched with blockgrat to sates reer "es, open-ended funding for an eaiement, oF gueranteed ledee] ‘matching funds. nd ar accompnied by ules abn the we of funds ach clit Denes serviee) ands of pei and gos ral social goal, Wihin this famewoek. deals af progam design ee offen eto state and acl ficial Sometinesmplenesreton ene ‘maybe concerned with andatertl quetonsoftesoundnes ce fram concept and design. For example, do the proposedearhans ech theneeds ofthe argetpopultionandaretheylkclytocoutrbee ee ogra gol Ae the planned adninitation thd tories ae ems eile and suited to the dsc “len flow and tamaponet formation needs tn general questions about thai behind pro uo dng ep eb at the conrectins among poem Scien pen serve aa en outcomes ve me a search ad exec ad tha hve ative pte and SV Ire pren cen technology and eis net onoty Implementation teschmy e dened tnlde the along ‘sant itinetarntsmhaes ec ‘ett shes a Se shy ogre nace Free a Flies 7 For example, even if issues of program capacity are settled enough fe+ AouIDE To PLEMENTATION aeseancit computer on ery caseworkers desk Moreover smite ‘asec adios programming revubmens oe he xs components pmgrnape ose imager. wrk andchent ands Pao edge Ether el terion? vil programs dont ext ina tact, Tob Slr Tobe succestey may ie a mei nd veined chen amb sonoee aa pie as favorable social, political, and economic condi tions, Pr = should ike he progam cont ino acount nea ‘groups. ‘Are program process and scorn + stems opereting as planned and a, how ‘tough program ay havea a ave adequate sors nd ma spent barca io component pice my ntopoe ee tsa ise. or eam a wr pice se raga eahing hinted re option ith he aprep- tise he plan rs and "dot thom ah Ainough he qa an quantity of progran serves ny be thought of prof footnotes coe foes nerves tpt tis purty re see ele 8 kde gov viet ‘Sic program escarole es in eo sonal ne rif eves nappa oo ke ay, eral frog gpabmay ao be me oar the nus eetaton Sls ofen ober matt an st vee roo a well a Glee sve otc anderen changes Ave clots achieving sind outomes and, fro, Row and eh? ‘Athough client outeomes are the revult of all the program aspects ds ‘assed above outcomes are often the Fist signs that rogram may oF tay not be working well Sometimes, dissatisfaction with results may tsa romp program manages, sate executives, o the state legla~ ture to undertake an implementation study i the fst plac. In un- ‘vering the reasons shy 2 program is unsuccessful in achieving {esl mplementstionrestch may calito question any oral of the parts ofa poogeany including, for cxample, the program’ design land theoretical underpinnings relationship is environment, admin- istrative and! management structure, resoures, policies and proce- lures, ad services. ‘What Research Methods Are Included in Implementation Studies? “While some researchers think of implementation research as a specific methodology thi book defines implementation research as the set of if sues that fal within the cov questions, "Whats bappening?™ VA shat s expected or deed” and" Why 5 it happening ti? ‘When characterized by the pursuit of these questions, implementation research ielectie and pragmatic in its methodoogis—the data, data Cellection strategies, analyses, snd presentational ties required are ‘etsmined by a combination of ts specific research guestions and an ‘lua gus by the eearchers about where and how to fok forthe Snswere The following action introduces and summarizes the variety‘A GUIDE TO IMPLEMENTATION RESEARCH of ats snes th aye include in inpementaton rar lar chaner dosriveanilweas then oper Data and Data Coltection Stu geoy manager lc fhe ee provider management and staff, ‘be ae Interest groups and client Re ‘ater information on site Towing expe state and local advocacy and pubic escarchers tse a variety of methods 10 2m these respondents, including the + Open-ended interven. Te ae ht focus on thoe pars ofthe Program or esas ee 4 osirceat othe norman ng imtvie PENS gps. Fos groups ate open end diceston goupn, ss of to more dan bo 18 epee shumnaed soindvome anal wtoftpisortiens en Periods often sprog stand the program rom the lets vienpoin, ee —————— set ten tan yal a oF the “average” caseworker, ot those that ask about how a fer groups or bears erent fom ober gerne eee {General thera types fdas tance eden ; Sore ) about latge numbers of people: adr I ta and survey data. Administrative data meds cos les Atos initrd ate yan or cat “Sorel ech anger or etic but hard-copy dinate ata thyabindudecse so parnives and tbe quate information. “rey inv colin data os lag mbes repens ‘The wl mo sues ats eon, ann peo inte ws pease conan of hee oni evar dh sre gustan hates formation brrconssmaybecary cv in qin rests st ao IIc opened or gultatie roporse "Lamia aa arson often sen static pot that ine importantsxresof oration about program operations vera outcomes Te fal goverment ste elas ‘Shetinsegue se agente esate repr tht monitor Jog petormanceand propa spending Ot sais ports Roy bros in undestning ogra envrrment sch US Cersusor ate vn statstice rte Other sl ti report ‘sCcomyied by bss groups bebe pi nd spines our Many publi og ove oped we sts ht oe ae ‘Shit sumparis of pny sts and acempens aeney docoments edt ropa sing, eign, nao admanstaton ind pblc inosine inl oe of dt or implement rseh.Thse cients ma ine leva ed {ted te lepatonsnd anne smn rl and Stic progam plans and docements supporting he pansimpemen- thon pm eter ams nd hanks program rms pes sndncesper ee Documenting Assessing, and Explaining Implementation Implementation esearch variety of ana statges ats dips pending on te dita used and the goa ofthe analyss—docomenting (Canis happaping”) assessing (kaha x essa ep [pea ephinng Why busaemineasat 7) program imple FRenttin, Why arasesnentandexplanaton tested separately inthis tok? Altra within the content of an impact study, assessment and ‘lanai ar nary the same. Thats impoct toes ate designed 10 ‘Sse the dpc to which program or pokyiscausing sed change. In social science, demonstrating that bis caused by a usually means that Bisexpined by aSree hater comes eeloceeeremnia ‘no etn Docomseine IueLeeiranion Document ton research may adopt whim Sot ribing sacl progeinn iat of, he try sl rom the pin of nae eae sens sare tough he eject decpion say aed {ftakeholder opinions and attitudes, it presents them as, ‘importer {ee sow the progam rather than ar aera oem ris ofthe progam, Soman, boner Fh oe {0 ase th program fomthe intl view of hepa er ving toandotnd whys rogram oer cats insider's view can often clarify the program actors"motivitions kee <ée an ies hs hing expt rcs wae ream an proving ih nwa nay aegis Researcher designing and conducting sty to doc ‘mentation face several. "Major challenges” el + Developing an ntl ea of what ot eligi ie of what obs, wom teen * Sontng though config or even contatory esr assessments of & program; “ory desripensead 1 Being ih vats ore tne and acs rogram sie nd Gambinguniaiesndguatae de nope ene a ses MmanTATON Tpieestaon ss vide sessment of rogram desig. oper ‘dnc ondeucome ingen mpamenston scenes cnet of ‘Sparing dts ov poban ojeaons sce service a ot ‘Smo wh some nnn or anardond doling eration be Sirti conpurton,nendacg seamen racers may spp {Cams norms estan depeingen tes ras some wn oe «the lnpoga i, lo. ose: erat au eel ora nance standards z «Profesional or industry stands: ‘Implementation researchers professional experience and judg sent and + Opinions anjudgments ofthe programs stakeholders. ie eas and perfor _Exruanone IMPLEMENTATION “Through explanatory analysts plementation research sees fo unde and why program operates a performs asi does, 2s well develop recommendations fr change ot lestons for effective program design Und edminstation. The iene of whether Implementation research tone can establish caual links between program polices program ‘tivities client outcomes, ad broader social change isan ongoing con- toversy. This back does not ngage inthe debate, but tkesthe position ‘that whe well lesgned implementation research an uncover plausible esons why «program is working oF not-—can bull hypotheses and | Whorieet should not be expected to demonstrate causityconcle- sively (se endnotes 3and 6). ‘Most explnatory analysis n implementation research ses one oF smage fre general approaches in developing hypotheses about aust ‘ cannecton (1) using stakcboer accounts of why they tke or fl taker specications rele o program activities and goals (2) associ ‘ting natural or planned variations n management, polices operations {eve other tre wth observed differences in resis and (3) mon itoring ele aaa predictions or expectatonsbated onthe program's mode! or theory: Laced aus a1 A OUIDE To IMPLEMENTATION RESEARCH ‘Advantages and Limitations of Implementation Rescarch ‘As with ay research approach nplementation research hs both advan {agesand limitations when compared withallrnave mebodleges ny ‘mentioned above, implementation research conducted in counrecn rth an impact evaluation provides detailed information “ia te back box'—informationabouthow progam operste and bow ieans ‘ppen. However, standalone implementation sie sss hove nn {antadvantgesand imitations compared with impact sds Anon ‘he advantages of implementation resuch ae the following: "+ Provides rapid feedback o program managers. When necesay, ieplerentation audi can be designed and elie quickly. Besar, {he study of “wha is happening” and prow'am operons conscnce On average ow long do clients rain inthe att or servic? Isthisfora longer at shorter period than expected?29 AGUIDE To nupLeMENTaTiON neseARCH © On average, how many hours per weck does the ctviy or ser ie require? How many hous per week do lens spead en the city or service? ‘© What the content and quality ofthe activity or service? It safcent to meet program expectations? IC not how and why? ‘+ What supportveservcs ae vase to pplicans and recpiems For each sportvevervice, © How many clients need or rss the sevice? © How many rective or we the service? © Isthe service aralable fra who need i If not why and how? (On averageshow long does someone have to walt for the sevice? © What the content and quality of the supportive service! I i suifsen to meet program expectations? Ina, how and why? *+ What ae the implitions ofthe content, timings and quit of services for program operations and performance! How does this vaty by locality? How may services be improved” {she program achieving desired outcone and if at how and why? ‘inal eof questions foases ox progtam res, at bth the "micrs” and"macto" lve. Al the mcr ee mplemertation eseatdhers wah now what happens so individual sppicante and cents asthe pose {hough the program andar subject ot plc, procedures beh, ioral requirements, and services. At the macro level, studies focus o ‘overall changes in the adinisrative and institucional calute of welfare and whether he reform effort ha substantially changed worker, Chen, nd the public's stndes and expectations about were. Some impor ‘ant questions about progrem outcomes includ the following examples * How many clin become employed rd in what timeframe ‘What yes obs in what nur ae cents Bg? What {he arcs of cents job, inctuding for example, wages hoard ten! Waheed of et + How many amis are achieving anil independeme vi vel in what time frme snd for wha reasons? ~ ¢ tare the changin mii ounesof ame and tino? * Whataethe changin amie housldndvingarengomen + What arethe chang den wl eng + Ho nd wy i hove ay yal? By demographic or INTRODUCTION TO IMPLEMENTATION RFSEARCH «To what degre have cient and worker expestacons aswell the cultura welfare’ changed? «1st program achieving oa for cnt ndotber outcomes? IF ‘ot how and why: low may program performance be improved “The Flexibility ofthe Implementation Research Agenda “hislonlist of reseac qeson tt ude implementation esearch saecrens rrsiond undestandngot wht he progestin {anon t tyingto do Beginning wth speci uso bls ‘he sehen the sey quarry data ad cate sours dete eal uss Wha hpi hit recipes Wy it hppenngan i One dara Tiphenaton reach, howe evi Becasethe tay 8 locked nesting at of bytes pected efor can {hit flocs gor dts a ses ot the Progra ta Sonne egos my Become moot and new que ‘ons mate. Prtclriy modes Of ew Progam ofa wie pea anges iin prog secret eae Bonen as aloe nw ins of uetining ee rowan fcr ape ene ‘Onc xempleot tc need to ein aimetaton esearch supe! bya craton of were fem demorstaton ov ost Se we inthe Ite Competes Employment ‘Opertumiy support Cente (CEOS) program se Were nd Nat Pobch 198), CEOSC was an intense sfation, employment ad trang gtr ne sr ele len ah preschool hen ‘he pez war pete by pci county wee eprtment oF by pte sor prof scl sce agence Part of ihceaonk rs appa esta CEOSC of fom buses taunt wie depres eur come mite unis ina inevalvaton teach pan ont onthe amine at cs nd aves of he HOSE prog se ot ‘hes it vist CEOSC programs made i der ht participa ‘nap wi th egy workers a9 riders of [pogrmsuc Fist CEOSCpaipnt clon city works Tertcmiursemens for tensportaton, chilis, apd ote ning ‘penn Sond and mre by, theclgbiity woken aceptance22 AGUIDE TO IFLEMENTATION RESEARCH ‘CEOSC was an important element in building participants’ conf- pertain at callin is ete the “ep aod he tmp sof he a ep icon ea Ito etn quate data ore ones onan uit it Samp se henamber nasa if ents gy wre ral TM fs) led ats cot for Win a badge an scone ee "teh enn depth ply inp dca naps sy exlring he the of TANF cen and exces snl oundestand wat abo the nae ay he 4 dt cobcion tate that nds lpenrod interviews with hee hdd and toc Yamin Tit lea [DATA NEEDS, SOURCES, AND COLLECTION STRATEOLES 41 approach limits the numberof individuals and families that may be Bude however. Or the other hand, an implementation study that {eka s generalized description of how client fae after leaving TANF ‘would regi greater breadth inthe numberof inva corered by The dat but researchers would aok normal expect olsen 28 mech ‘bout enc individual or tryin the sty sample as inthe in-depth ‘Staly Table 1 shows ho the tradeot between depth and sample size ‘Sreimpli yfouralteraivesourees of data about TANF cents ta ‘Sard atocclveports administrative systems, surveys, and in-depth fpenended interviews How do researchers make decisions about a study's dats needs and shout the inevitable adeofs in sleting dat collection met? Per~ ips the most important gue 5 the undedying purpose ofthe sta Teremample implementation studies hat are exploratory in nar end fo reguie more depth fo reduce the chaness of missing something Important inthe data cllecion effort On te other hand staies pr araedy anni the degree to which the program isoperatingand per {amin at planned (for example, "How well the program being implemented across the state ard whete ae the robles of exemplary program sits") probably requ ager empl sizes and may nce to etepromiie on depthby maturing ited mumber of summary ar Ses serosa rclany large number af clients workers, and/or Jol ‘ices Ones a decon ie made about the basic natere ofthe stay ‘hnloratry of defintine—speciic data needs follow fom the study's resarch questions” ‘Conducting Interviews and Observations on Site “The data that may be quired for an implementation study ae varied And eallfgrs ange ol ealestion statis Some dita maybe collected ‘Sectonialy, by mail of telephone. Other data collection efforts— Sth io-depth open-ended interviews, focus groups, and observ om for cxample--reqite an in-person approach, Mast thorough implementation stodies sequie researchers o spend some amoont of Time'in the fld or atthe location where te program is adminis- tered, whete services re delved oF wiereclints workand ive, This Section sunmarzes how tocorganizeidwork and how to collect dat inthe eld"and Collection st esa pina ate hinted mane teat nadie terme lero ‘re drng spoon Ta porte Today sto saat cay ‘loan set oy ‘om on Ire gtine “ania ‘Swine nlite mater aa nvetha ‘trea Sil ay eter encom “Ae ‘mona etd ety oa ote ae tel nti (nr pe A GUIDE To IMPLEMENTATION RESEARCH ‘ble 21. Tadeo between Det and Breadth n Data Sources Sorter Scowrntin icine esa lee caer Seimei imcompurtle eter ‘ciara tay ne nile evalorer finiysre ee and Sc tincecini Sop ta ‘Bales exh nial Siegen Siang font rig Snort ot _ sas sete Siro ecg oe cmon, mannan Aieedcinet monte bo. opel tei Searcy toric pad Deets ‘yee eng ‘Sbeeoond ‘yess ot se DATA SEEDS, SOURCES, AND COLLECTION STRATEGIES 43 Planing On-Site Interviews and Observations Before ongeizing the on-site data collection effort of an implementa fon study researchers mst make a number of decisions, nding he following examples Whi napondonstoinevew and hich tvs to obser. pen sng the gevera nd spec research goals pnetted by an implemen tation study what ype individuals shouldbe interviewed or organized tnt focus groups sd which activites, events, or services sould be ‘Sucre For comprehensive implementation study that seks to unde “Rand the design dewlopment,inrodaction, ongoing adminsraton and Teatlisof new prgsam or service, thet carb age For example ta fied respondents could ine the following: sate leve TANF agency tanagement ad taf rangement and sof rested tae social service tnd employoent agencies stat legslors and staf representatives ofthe fine ecutive bra cal TANF agency management aod tf, ma [geet and sal felted ca social sevice agencies tepresentatves of Enc provi epreseataties of eal employers or Busnes groups ‘epreenaives of stewie and local public inteestand advocacy ous SnUTANF clntsandex-enisand thei fis, Shtivies or services dered ould include for example, intake te views redetermination iteviews, employment program orientations ‘Tad tecaments jb chibs and other ob- finding group and individual Severo serves ill tring tes, program ext nevis and time limit r sanction appeals esons, or studies that are more focssed on specifics the target espon- dents shouldbe those with rt hand expeience of he isu, For eat pean investigation of how the TANF application process wotks would Formally gather information from applicants, clerks, and eligibility twoers end supe visors. Simla information about how employabi- Jp asenaments ate conducted shold be obtained from TANF cen fh the case manages or employment specialists tha normaly conduct the assessments, ‘How may and which "itso interview or observe. Paral to the iasteof which typeof individaor event interview nd ober athe Fanuc of how many of those “units of analysis” to choose and how to ‘choose among ll otetal unite Thisis the problem of sampling st Sy und it nuded pat y the oveel purpose of the sud. For ‘Song fora purely exploratory study tat seeks oem" wbatsorsofthings are happening” in the implementation of «new program, one straiegy might be to interview or observa small momber of unt but fo choose wits that ae likely to represent some variation, such a loa ‘ofces in diferent regions ofthe sate, oe TAN fares with ferent demographic characteristics. This approach helps incteage he chance of observing diferent implementation experiences an cam then fore mote stems sty Astudy that seks to characte the average fstresr outcomes of !programor to assesthe prevalence of some pater tras mas wove «18 local offices. faces the sue of clectng» epresentative stp of the units 20 be characterized o assessed, Fra gven level of precio, sampling satis may be uscd to tint how many ofthe sncyoe ‘nis shouldbe contacted In general ager sample with es vaaon among the units wily more prese and more representative et ‘mates, The research profession bas ationd standardeof precio hat areigh and often equi large rp. However sell ifort to support design or management decisions can certainly be based on smaller samples, with the caveat that hese the sample the higher thei that the derived estimates wil deviate from the ruth Ther ee sharp ditinton between calectien of ancodotes and anette based ‘scientific sample but only gradations of sats! precon, These sonsderation abou smaller samples ofen core Into lay with Bek work becase ican be ery expensed impractical cond og interviews with many respondent, The decison of which waits should be sampled is usually ade eps rately and before the decision about how many units to samples Pe answer othe former question isnotabysstraghtor ware, but lends inparton guesses about where the most variation occurs For exam study wants to aes the degre to which workers and applicants de uss relevant employmen: services during the egiity terse, the sampling statcyy shoud be guided by whether the bigger erences pctice are among individual workers, individual sapersaory ur ot Individual offices The most effective strategy would be to sap the level ofthe unit with he lange amount of veation., Designing interviews or he eld. Decisions about he type of infor ‘maton needed affect howe interviews shouldbe conduc inte Bld Foresampleifrsearchers hve few peconcived ides about how ar rogram or servis has been implemented the ssruments seo guide Interviews and observations ate likly tobe less strututed std more ‘Bes dried oslo he rogram as efenc te inrvews and ‘of activities on sit, may deaw on the time and resources {og munaumnt el sf los on the ine TAN ap abc of it cfd thorgh preperation Pa avin inring tes, Een no oral itn rena ve tcaroed TANF apnceso = verses que ane ad dence, aly re pew he igh evel of he gecy unde stay He fe me ANE apy of oar cence my m8 Shot een sting nl TANF ofr Howe Pe emaon may be eed te nso sie oven ea tc Tor the wa, Anne clearance may 0 BEead for iss tothe TANFagecy directors of local TANF fies {obe sited shouldbe noied well before the site vst A cormmeon ‘approach sto send lterssgnedby he cet agency director Ce the commissioner ofthe TAN agency, informing eter agence snd cal TANF offs about the ty and that someone Rom the ‘dy willbe contetng ther about arranging for «ie + Establishing an on-site contact per. One eflectiveseategy for Fakdwork isto request that each site (excl agency ofce sevice provider ageney, or traning st, for example) designate + a member toaet asa contac forthe stad The contact wo be som between the ite and the sty and wuld be the normal conn "munication ink between the wn. Te it contact can hep dehy specie espondens, cate needed documents, help range sid Schedule interviews and observations, and distribute say sty imutetas to onsite respondents. Depending on the level of effort required the ite coniat may ao best to help ecu staff os ens fr fcusgroupe *» Arnanging for and hauling interviows and ober vations onsite. Ons sit iterviws and observtins -boul be scheduled several weeks Jnadeance ofthe vist, One fective approach iso deaw ap ten tatveschedule indicating the individu or ype ot rependent sd "he approximate length of time for the interview. The tentative Schedule shou be ses tothe sit cont who then attempt fil Interview slots withthe appropriate individuals, The schedale ould be exible, so ha the ste contact cn fit the ned ofthe stay with respondent schedals and work demands, 1+ Sending materials in aves of the te ast. Depending por the specie purpose ofthe interview frst of questions ison & ‘hod idea to send the merview or npc guides in advanceof the ie Vist particular, for example, when the major purpose of the imtrvew is desrptve, having the interview o ope pe before. hand can ep prepare respondents forte interview: Often, as ‘ants wilt down notes ancl wseflinformatontelting to ‘opis to he covered in the Inteview Sending materials prior ida ss ia help forthe parts ofthe interview dealing with aor ‘mants’opisions and secommendaions Partcuarly instances when the study is more interested in considered and supported opinions and assessments, rather than in "nap adgnents 8 ‘nif wallow time for informants ore before he itrviewe two tuations ens probably not god ida to sed onions trent bsreinicie the interview. For example. when part ofthe purpose of the inter Belo ames manapmest or sa kee ne Police o Tite ipping the of the questions in anc il cley ‘Seti tat goal A more sublease which te intervie of ‘Sp eh ts nce ben rant ep ‘tei oes tr anweinar nin ch teers tt aso, vqustion icing onions end seommendtions “uclibc weighed au ie bneis afore roughly epaed "ahr nt sytem decides lo send ample or pata incre oon suns nabs iin an information sheet epanng the stad. te prpae of te at ‘neta iesenanden amplt ore ofthe piso be ‘Sher shoud be made valet il prospect informants «Singin. alan ec at rg Stat ie lipeseenriting mae {Sifwanngandrerater. Aly impeston i) ‘ae ae ype nd nic cn thorogh raining hoe pune andar f tse IMGIE iitespcc pci range epee pers ‘idigutn ra ne nde here Itai operon iy hole eed Spt evi unrsendnpons ane ferent mai Se aeeeer acme mee concn te mening ech {hesiog all archers pie i aning ers prepa, and posts at umn Toe mal arden sul cove telling pe - Ope hedaman of progam or og Sisal icudingt Gos fem ilo be encounter Grenewof the study design and goa, an how te vistselate toate ata© Overview of data collection activities on sit, inching hard copy material tobe collected, activities to be observed, indivi uals and groups tobe interviewed, and overall data collection responsive, © Preis preparation and site communications © Behivior and euguete onsite, © Condactng the interviews incaling a detaled cicusion ofthe intention gols and expect ange of rsponses an a question by-question "0 by") has © Taking notes and Keeping a record ofthe interviews and, © Post-vist activites inelding organising and reporting responses snd other data and citcurstances governing pov conta! wth informants, Figure 2.1 presents the contents ofa taining mantal for the fekdvork na mulistate implementation stalyof the Low-income Child Cae progam, + Postvitactvres, Sve vistors usally have several activities to ‘complete nce the fl vit sever. For example, interview noes have wo be ongaized and put ito a form thats accessible to other ‘sty tal Ech question or topic inthe interview or observation sides must be addressed therein answer othe question oF sem, the researcher shoul explain why. Forte, alter thorough review of interview notes, site vistors should expect 10 conduct follow-up telephone calls informants for larfcation o missing details, Finally, as described in grester deal below site vistors should meet co compare thee findings and expercnces and to begin the process f analyzing dat, Collecting Data On Site joe in-person, onsite data collection activites include open-ended lnterviws and observations. Although other forms of dat cllecton may take ple on site—such a focus groups, ethnographic sui survey ‘questionnaires and case record abstractions —those actives ar wtaly nt by speci rather thanby tpl impleercation sty lt worker. Thissction discusses howto gn abou! degringand ea mcting ‘open-ended interview ar structured observations Figure 21. eldwork Training Maual Contents of Tring Maus! forthe Hana! Stuy ot out e |Care for Low-Income Families: Sule and Commun Say chapec:trtrodution tothe National Study of Cid Car for owrncome Fame 1x "Pupows ae print ofthe rahi anus 2. Oral fe tur ©. Cojo ar des ofthe ste and commen sti ‘Ranerahip oversea ajcives (0, Pret shea ad lveables cout 2: Overvi oftan Clacton agree forte Ste canny Sty Fst a on a ncn sates at) sn om eto FM nnrows nd oer te achonas 5 Petey cco endcmaon aot 3: Pre-Flel Vil Preprtonsan Da Calton ShRSGguree mrad se cormeny sie OR onan and erg te probs rene ‘hater: Conducting hes: eso open 1 raton shi crn gona en 2 Celene mere cc tay prey o. Coe noson ttn rans csc ay rem a 1, Cin TAN apy ors 2 Congreso stonnt awe sr a7 ‘Sune taco elie nnn Ani treo imonrchscnsPoaDeving nares fr oerendeneviaes, The wslinatranest for condcing open-ended ize inthe fl sn mero ue, Interview ges ca ranging an comply om sg oe beso icon pc or eh ner t+ mae on Pend of ented querions nth pomp an per ie {soto atypeot interment axhare TANF gow Whey sip tpc i my bea ican coment yo gee et has some important dwbck sansa Gy Fn, the ee vient may get orp verse potatoe deals a “pre end temo ech ill rs ‘each and pethaps acs raps pouty muligi sont das corral, Using sre erie pi can poe Shecompihsnenes comptes and computa da, “ho inary and ete deg ows or ere stument te hichopstaindudeandhow castor the topcase “ciment for cifeent ype apna freaks that quston abot program dagn wil be incoded ian nrc {uid foram sight worker Moree, dough fat ate ‘eerste TANF pny or maybe shed uso sat leat respons fae rogram or ply thes lwp gr ‘iors and rte wiley iftoet rh te pond Forte ‘rae purposes, mee the trp ofan eli worker ory TARE [ogra witha" fin? dag that ese pid move oa ‘rr longer-term traning and we preprton ‘come ae Stat contrite trv mae it ef the topi you want cver Two mporant eter te tp et ince) atte pies dec tet formants ch ine foe and (2) ae htop ed othe a tech gus on? An ier wie TANF egy wore inthe emt comprehensive implementation ud of ates TANF progam ight ince the owing tee areas wore teckgound and aperetes woke ining rte new program oly rogram sare DINE 2ppleation process apse an cent eh eqcnonecrooe csttand cts educating and wort splot sisi sprees mgd ning cei itsand wanton procedures snd secs an vel poem sce mentand ommendations a ‘ comenin net ep eeoping nese puis it athe Snes tobe conrad unde ech pic heading For cumple forthe pe DATA NEEDS, SOURCES. AND COLLECTION STRATEGIES 51 area “aptian and cent cao, ting, and work rete ser ‘ebro be covered myn the owing rang of eres ‘palbleacs seve content and drain of serves, logncyandqusiy of \ricesterce pote and communion Tihsewee power, Tei of rede nthe comer hair devdoing specie questions prompts ana probes for eplagroup of pe ‘Ri uct fs o the wae ent Somat Serie” and adequcy and aly of serve” might ook thesequene nfgute 22 et nections othe eevee ae inparenbos) ‘though evoping ful tof interview guides (opis) seat may wat oct guide fresh pel espn ome topestnd some specie questions may ste sac atm espondent- ‘Whercappropriae tee of he sane tpi or question fer avarey UW etrvn allow for some sya of difrence i sponses By Sf or manag onto. " Conducting the erin Reseach sng opet-ended nerve estes een sasoamer swe ane of penis sd 5 ont norants steel of interview aude quran and to colecsy nts let oma Sons pb om ssh nm aac stn eps the somerset foenant In pare spor ntiokep fe cnerstiono he opi st hand and hep heir ‘New iin the sceded ne ot Ath oro he pps of Spevanaed evi allow formant to cxandon he esis ‘ed awl fot nerve eo prbe fr more dp the inte Towers seman conle he nerves pce enh and Tetjes mater Wh sme experince, th can be done pic and courteous manne, “hough sme informants ate more natural forthcoming han over ld esearch sou expert ponent to answre te g Gnaeus und nor eesti ner neta ormaton Forms bing guesion se ery helm cing eee for Enema at apna moran reve the ‘ponent For examples whch ated to exp he bbvral equi ‘Renee TaNrappicns an lent ese mange might net o ‘eon lot thestusons that may expan nil om pa ina wok ced acy. Teer my be eo remyFigure 22. Masrative Questions for Fre Questions for Open: Bed Interviews with TANE Imtervew Guide for TANF Case Manager “opie Vs Applicant andeten. Tope Anta and cnt eaten aang snd ware 2c ety an aan serie Fe . sri Foca tbe cat Sanches series avaiable TAN sopanis an “ eerie ne nate eee el We seca, Fr ‘rap ests nw carta ae ea ha) eno steer eer oon india sec rb each ssa isch poaw Porters ie nteing taut bm easton noe cnng vremperens| | Sean | Ebates argo yen! | syamer ont sorce fox ench of hsv re, as ‘eae tein oat a amon oe Anca ay of air ech ee aon ea rd woe series avalale TANF appears ar lors nos ‘maar rvs ease bain whee yas nave rere fo ob een anon Sain ea) oro mar rnb ca _tfecve? For exampie, do you get regu feedback fram : av you cian anf evolu Oooo wok You clegies and ober TAN programa? ftw sats scot, tha wasnt be Som bmprore te coven” ‘sao rence nati ne be uty tse Pease ‘Sires Pesos te thro ay Pr le lt an ag sets Mp Fan Pa npc orto fale o revise pata station san exentin* ‘guard apis soch oer the qton abo exemptions might {dod isto common exemptions ued ebever “Another wel practise contin opted interview a oask respondent the tess andor evklenc or Dei fgets ‘Gem there"calenge ae weiter nas arto thequstins then {SIS ne es appenct ay fceinfrans to thinker acl ‘Soortner repos ado quali shen iathe ihtf thi ound or iolungthcpons Son, ae esac o weigh theif thon pinion palate seth ofthe evden edt Soppor i. “Rn important ssc in cndacting om ite nerves concerns hot nay informants ocd any sng nerve session. There are ‘Sim pracia teal in mang ths dco Taking to mote nr ‘Mana once an lp dewelop tore comprehensive answers as each Brant may ops ifroton let by the tes so sing Stal group interview oppored to iid! interviews cleaty ‘feat te umber of nds that my be seen during ste isk ‘teehee, weve terete more ha oF he spon Sent he eseachrean elec eerie ashe ite play among the informants tay overpower temps to focus the Ramet on the qucstion stand Also opinionated informants Sinton ith Sronge personals ay vate judgmen's the Stes in th os nerve "Coney oy ean sport fue forope-endedinevews partly nen norman ie the oison bout how wel s Brom oc weking Froth eh able sons it etal ‘Sinton respondents aout wether and wt dpe conesality lb prteed Informant conencemananyty (he pose) “ial improve the chances for andor nthe espns Desig and ondacting structured oberations ons. let antec dork sth opprtant floes toute tsar cea progam cvs actin sas open ended ne fiom roque sonic ore to coe ror pando ensure compe Tay cos evo anid aco archer servations bene From rutare A vommon tos ote is of questions fr the tbeaser wave rng that abe. “ie approach to developing observation guides sina to the apponch fr the nerve pd Tat the srutured aber an Fide shld be ogated wrod ast of sue eed othe stentand purpose of the event or activity and shoud be related othe tuys ‘overalls. The resulting observation ide shouldbe ast of questions ‘that fi one or both of those erteria and dat focus and organie the observe’sexperince in ways wel othe sty. Figure 2. isan exam ple ofan observation guide developed forthe evaluation of the 10 Srenghen Michigans ames welfare orm demonstuion Obs ate expected to anser each question on the guide or explain why the ‘question snot relevant or eanot be answered ‘Other Data Collection Activ On-site open-ended interviews and observations are usally a major part ofthe data collection efor of an implementation study, As mentioned shove, there are many other forms of dat and data cllectonsrteies that may be neededo adres the vated escach requirements scam prchensive implementation study: Mow ofthe tae dcssed law ave bon tated extensively in thesia sence research methods iter sture (see Charlesworth and Born 2003). This section docs not cover ‘hem alin deh, but indicates how data colletion strategies may be rele vant to implementation research and some ofthe decisions rseuchers may nead to make when adopting the statis or ther project Administrative Data Administrative data are data normaly coileted spat of engoig po ‘am adminisraion Two types are typically valable: had copy ad tomate. Callestng hard-copy adminiraive dats, Hand-copy program flex may bea source for eer quantitative data such ascent characteris) ‘or dscursive data ach as ase narratives). The former ae usually sed bya study to characterize or measure large groups of peopl, whe the later ae mot often wed in interive tues of ele sal ures ‘of intividal cers or fms. Themsjor challenges to collecting usa sn reliable quantitative dat fom Bar copy fle ae t oat the mee ‘essary formsand other information withineach cen oseect the quired dat fom each paper aie and to eord the dst in uniform ant well-onanized way. Ultimately these date wil ily be ented ita ‘computer or atta ana, gue 23. Observation Guide ‘Obcervation Gude fr Inthe and Radeterination Seslons Cot, To sessnn began: 2 ante 62. Deeibe ie phyaeal sating or he asso (og, pt ction, freien spe sasowoas ciclo, et) (68. Has this oem been poo shedledor tho cant ‘wake? (4 Dest te cents ocunstanens For sxanple rumbrot hen an aspen, employment as, provovs cet ic oasanea.ot How it sant? Oe ct appear be Jenantodgeatie about TANF or TSMIF? 8. Hon nates na hr pepe sesso? Hae let Teenpovsey rior toa rons craton? an ‘nan at lhe races sos ret row dete ‘otra nine ncn ee a msg roma 7. How et stn psc Mas gtd son 3. Whatman vas roles ean How eae ne encnersemnato? 69, Wat questing any, lena? What nase cons ‘uc gh snore et pear ean 10. Ome ch matin rg moved ay ahi ho oe counted towed tuttilment of he Soci! Contract? if 0, wnat Sevbess waar? a 611. Was cla gh ny ten imation dig te sea SSrort wath cn grt (Pesneer Oban cist Singomeciionecon) C12 iat wash ate sess? we me sxanson osu? coe or egby ater ee tp? han nes mech eeicecontes x TOE?) awew (610, Tne sasdon ended, “owe ip tn pir oma i nf eT pheA convenient way to tart organizing data clleton from hard- marly fr program administration rather than for eatin research Nevertheless the automated administrative data represent the riches, ‘most accesible oureof standardized information forall clients during ‘the perils in which they receive TANF beefs. Moreover, becuse the sudminsratve data ae microdata, or related to spec indus and ‘ase the dat maybe manipulated fra variety of statisti analyses, ‘There are wo basic approaches colectng automated administrative {at from tae or local TANF agencies One to ve the TAN agen "tract the specific varables forthe range of caer needed fo the sta, ‘Tis approach avoids some programming labor for the reserches and «can yi implied client reo that may be ease fo manipulate and ‘aly. The disadvantages that the analysis constrained bythe var ables chosen beforehand iting eit. Als, the researchers have lesscontrol over the quay ofthe data extracts anf sakes ae made bythe TANF agency (and detected, the extracts would bare tbe pro vided agin. The wecond and more usu, approach isto sk the TANF agencyto copy the ene administrative system dataset an end the copy tothe researchers" Ths apprach alow restr fei in choosing sppropite data iter for ana but wally means more effort in dat storage and proesing For ether approach, researchers mt take ne olan the strate and content ofthe administrative system, The systems le layout is = pod placetostar. Tis isessentlly roadmap ofthe data ites that the System s designed to tecord foreach individual case, and household cvng benefits Isao important to anes beforehand the reliably ‘ofthe dita tems that may ser worth fo include in the implemen tation stad. Tw sues in particu are worth eonsidetng. First its important i rack any changes in thease offs on cit records Although states arly implement an etl ew stoma tte doma feet chang tire ate ep ‘Tenterden ers naan igh our pan ensures fin ech duo ard ees care nga “Seumeor est tot wt ey poe ‘Segnapodieso chek nyse ah a Sona anced bors om dan cdc aoc antwilpbiyer ge icteminason dame sing fie orinmerte Benne compere ng Meares dn cnet mir er hee “Tae maybe isu foray eae at Str ers eo rT ei ee sone te Thacher ant pen cig an net en Tec: heer wispy ve tenet one revo rg Since hy dot inne ean rity ae nary ‘Dati pce Ge Chg Sang Sen mc f torte smiteve de ald poly ego ea thre nt nd dc sprees smn oom dt rusedby compu pam Focus Groups Focus oups he een wel wily in averting an politic con Sutin or any yaad ave mre recent made theta tbe ‘ori of social ence rescnch, Foes group can every tefl in Inplermtoneachss pln ts The neal oa fe {Boup ito nip tener and delete the options cones bebe Sea ferot pope who shar achuatersicor ot of haces of interest the seach" For rampl an plementation tu tbe fpoup maybe gy wovers TANF sppliants or TANE chen ho {Te combining wok and wale Using omogenou group of pape dows the focus up fo oie in dept the des and opinions of ind with sbared et of experiences. Focus pupae tot far xls sate eau the luge anc st ofthe data they In pat ss ecase thee nese vay of earring bow prea tbe dictions Tartu he oupishomopenous sro sme key race i there no way of kong weber the responses of one oupof {Sah or nine ndidule would be he sane he sponses of atertnt eh etc dymamithat mas thedicson nap way ora be sed Sirist nasetent Aannemeeean Patent rnerni aed Sef iieauied nance Stipes mneharnas ink ete etapa tease efceatan aiacne mSetpcamestcettn aan agama Ege masa hacia sieht tse eee icureeigt anitemsPracmecnt Sint enc ace Society gis geet Tiseapsiemirerete regenerate ae we SSD eaten rte eevee cae eaeate Seeaieen aie eronees Zerit oversimeee er Sepa cemehete een eet Seeiiemecinennce Sor Meerut ee eee ieaeanee eet rien trnnun Shethchaeecateenn eet Sh itoee morakegineenrec ehocmtenitamesieentearen rate inherent tempting to change the nsttasonalcalture of the TANF ofce, focus ‘groups of workers might help clr the range of reactions from work frconironted with asc change in how "business done” “The question of whom tone a Fous groups turns on researchers views of wheter there willbe important diferences inthe view = ‘Silows andbalelsof the varios grouped bythe new plc or po tram. For etample given the importance of chil care to TANF cents Sih must Bind employment, the study might consider organizing di. rt focus groupe for TANF amis with preschool chile, o for TANE Fame with adit available at home t ere for children. Silay, dleent focus groups lity workersmay be organied onthe Bass ‘of year of service inthe TANF agency. Therma terion is whether the ‘Specie characteristics of people invited to parpate defines grovp that ‘likely to haveshared opinions an exesenes. ‘Aertel tool for eondting wel organized fous group isthe mod craters guid, Depending the modereor® se apd experience, the Suid ray be es genera a i of tpi or detaie seripted gus thos. The important aspect ofthe guide sto inde she range of topics to be dace. Most guides ao incade probes use when the group thor not espn to the moderstors questions o topics. An example of prof afore group moderator guide forthe evaluation of a welfare reform projet Artaone since igure 2.3." Surveys Surveys ae well xtalished methods of collecting detailed comparable information rom large rsmber of individ. Surveys conducted over the wephone orn person can be relative expensive and should be sed ‘aly pede o address ane or ore ofthe research goals. One impor- {ent ertesion fora survey ina implementation sua isthe need for Jafocmation that nay be generalized overs large group of people. For fuample ihe study ake wat proportion of fais leaving the TANT Follsareconered by heath insurance, escaches need oaskthis question ta suficenty large sample hom tht group tbe ble to make elisble Semen about al amis leaving TANF. Tf researchers ace stitial generalizations about groups of ind viduals or familion survey may be necessary. Thats ithe characters. ticor outcome being messed and/or the target group are not inluded jn an aval administrative dataset, the only way cll the dt isFigure28. Fcis Group Madetor’s Guide | FOCUS GROUP GUIDE ‘Group 9: Sanctioned/Benefts Terminated 1 Psa tbackii te en ot aed a Ih oyu beets Bene. compe ‘routers of JOBS paper) Isto cy nto wont et on sor bang sre wou wea hora sr noted ut you wou an or ah aac Dre ‘Pete you une aso st hs es gor mepe, {na igo areas a woe + Dayouhove ct te wou ang lw weapon an “why it happened? ms + Duyou meet oy cova yur econ yo ‘eave eeu ao You tees bang tees 2 Latest e momento even. The nde cash asco ‘niet sch You nope bo JEBS oan ington fr ward. Ta Paras er not eogetaing win e088 open mes Eanes banish Sar How mary fu el at Neca? How many ‘have ben cons a al bout ths le? What has Soar ‘confusing? | “Srainsoomn + How mth oa coat ws you ck tune ‘ansion bout the JOBS reqrements in yourbeing - 91 ator your berets ware ented how concer wee you abou he possi cf beng sanoned fo nt comping wih ‘Fe requirements ofthe JOS prageaen? " cy ona Wy wee ye Scr ‘being sanctioned? yee ‘en + Rgutonet ayo met + ion verteaton ure + Pe sont sy otc + ante ert et ch | I i | boy survey. For example i the study requires statistical information shout the edvatinal lve of TANF reipients, it will need to use 3 Sey ifthe TANF administrative system does no include those data Similars study needs statistical information about TANF program Testers, or TANE clgbilty workers, it will probably needa sary 2s Comprehensive dat abou those groups ae not likly to be avaiable ‘through administrative data ‘Sree one. In part cae Surveys can be an expensive dats cl lec stay patel respondent smplesare ange and include populations that are dict 0 locate), and a study often only pts “one Shot" at alli suey data, researcher should ake ret care dec ing what information the survey sould age. As wall data collection, the raional or any particule question or topic ea should come rom the esearch goat and the set of speci esearch questions forthe ud Moreover it ustlly not ecommended to rely on survey fo infor maton that may require open-ended answers as opposed to precoded Choices. The extra expense associated with post (0° tn tn In any event the standard wed to aes program implementation are mot immutable, nd may themscves come into question. They may hot be appropriate o spite program applications they may be ovriy ‘optimise oF pessimistic in their expectations for program revits oF they may be based on wed theories or poor empire recarch Uitic ‘mately the usfltss of any ctlative standard rns ons aceptance bythe researcher, pliymaka, and program practioner commutes, the objectivity andeliblty oft appation across multiple programs: dis ability to guide practical decsonseboat fatare program design fd adit, “The remainder ofthis section dicases each of the evaluative stam ards lited above and lusts thei se in implementation research, Program Mode, Pan, or Design “Among the most common sandarés wed ln asessng rogram imple rmentaon ae programa’ xin and expectations The pica te ‘when the progam model isa design that har alec ben shawn ef sibe and efctive When policymaker and program operas hve sound reaons for implementing a given set of polices and services they frst ‘want to know ithe implementation i thi tothe baepins. More= ove, juging the sucess of + prorat (or policy or service, or ain= istrative approach) in its own terms has 2 certain fae valty ean objective way eo asesing program implementation, Fil for some ‘aluationappreaches mast oti "theory of change’ dcused Blow), the program’ design and expected shor- and longterm oucmes ae the ‘ole sandards against which wo ass implementton succes ‘A recent evaluation stady of effort to improve the guaaiy and quality of child cate for low-income files provides good extnple ‘ofthe use of detailed program plans as the major emt criteria for judging the suces of program implementation, The ist step inthe research project entailed developing detailed program model and expectations withthe el of program manager and af tthe vsius project sites Like the example presence infiguze 3.2, the models expose the ways in which program structure, ate, nd eres are hypothesized o lead to program goals Th evasion coms largely inthe periodic observation of project activities an este and acon parson of actual program experience with he detailed program ma i Figure 41 illustrates this proach ss wed to eva the sce of | Figure 4.1, Evaluating Program Implementation Using a Program ‘Model asa Standard SITE XGOAL1 Desiedaysemic change __Ststagenactven ‘Grete waetsutaning new Concucta vane preven Sihighaaty omy ehizcae” —crfrence tence by 2m eres ‘hiseare rises) Mest wm prove atoning te Coen promt tea Snalecuns be fomatonc! 2 prowde' acoaton Conca aren based care Corfe to prarate cally an ‘eon mes te promt ‘irctee ans deeopebshess on esearch ndings inropr text ‘Ova, Ste Xeinitve dos mak oppea 6 be mpeg Stopes faring a vies ssocston ae lene Ne Domes es beon nase anne cevekpmnt of oy rganzsbna tue 0 ‘iets lan forth sesso. Tus far, ee fas boon il, # ny mesg oureach povaaain oun X seu =U Froid tpn av aascaton Wt Pas no arson of end “Te srtgy has nw boa hangeo—rate than fring ssn ‘Shntynide asocinton sept asciaors wl be ome i each of ‘curparsct ne coy. (continual)ne Figure 4.1, Continued proc site in implementing strategies and activities preserbed by its program model. SITE XGOAL2 ‘Program models or designs may also prescribe quantitative goals, such Dested wystenie canes Stetpobciin slipper tones ne nine oh eee Sinem may bperiobetraioread compursos of Pos peat eae eee iy care ory ‘eal wih propa gals Homere searchers sold ae care ape " sianhchrlend = ify definitions of observed activities or outcomes, such as "participation Develop community qualy in awork-rebated activity” and “entered employment,"for example. The Standards with appropri Waining specifications for these measures may differ by intensity (such as “hours motels and assassin 198 ‘per week") or length of time (such as number of weeks or months). The Cena mn ig see ‘Beare tod by re shal be dened the ame ay 8 thse ‘cmrna seven eae hatne or wine “ae! incumple of teen of rogam kpleesaon teomse pene stg gurtattemense Theat nea easton of nc see ay fren Seltcce mrt program new Yor Sete agin servis and protons batae “Se munigm ic Compete Employment OpportsntesS Srocuibe menses wh Sie Comer (CEOSC) prota Each vit profeted prio oe ‘Std anc forvrou service ratte, The cai nats how ano he seman eee projets mth “number fo prlpaton specie pops hata ay sanded ne ferieo or ace cm eotch ndings noon tet “The goa nereate he spy of qty cr inte cau hasnt ‘een by achieved. Tha a ep ows scieig hs oa was ‘ey pol of 100 otra pps ka coy Te mao be acconpahed by conducing mony ep uses. Ts acy riko pas epee “ible 4.1, Feluaing Progra Iplerentation Using Quantaive Program Goals Standards Siac Ming ryan oak ne CHOSE Pep Tho itts has had seneahat mor scsassindevelpng sl Nmirof Nema‘ Nambrafst quay sansrds cd care. Aconaiant was Rios ove a et Pepenactity sine —smeing meng Spent ‘commnty Quay sanders an ascosaren ol The santa aa ctr incor tetnarh were devises, be astessrai tots ta were denen cnnat - bes asses quay obecey a > 3 7 Prongoymemaiing 7 ‘ > Thae ono apart be ary elt lanes tye an he Ecnira ing 3 3 > Segre areting a som ncn ase the ayo Staite 5 3 3 cheat Cond cin 4 4 ‘ acingFederal an State Legislative or Administrative Rules Speifing rogram Content and Performance Standards ‘Often legislation creating social programs specifies or mandates led agencies to develop standards for program cootent,pariipation, and ‘uteomes. Moreover, meeting or exceeding participation or outcome Standards often eslsin bons funding, while ang omc he an- dards may resin penalties this she case fr state deve work potic= Inston rats for TANF recipients, for example. The standards tstlly Impose these set of expetations on mutple program implementa ‘ns and so allow fr meuningfaleomparoons sero programs ‘The legilation cresting the Even Str Pail Litray Program in 1989 required tha ail local projets provide ast of Basic program de- ments, including core instractonal elements ad suppor servis Te quirements tp shor of specifying curiclum and eave the deta to individual project sites. An obvious issuefor the evaluation of ven Start as to measure the degree to which projects mt the minimum stan dard for program content. Table 4.2 summaries the degree to which ven Start projets met the specif program cantent requirements Legilaively or adminsteavely mandated participation lees area relatively como wa to impose overarching performance standart “Tabled presents an example ofthe we of partipation standards nan implementation evaluation ofthe Job Opportunity snd Basie Skills (BS) progam. Performance tandadsaresomatines applied o program outcomes. For ample, the national welare efor legisation passed in 196-— the Personal Kesponsblty and Work Opportunities Reconeliation Act, (PRWORA}—esalshed bonus payments states that lowered their ‘tes of out-of wedlock bith by agen pereniage Sometimes specie benchmarks ate juste for local condition tha may ale oxtemcs sequrdless ofthe programs performance. For example, performance Standards for employment outcomes fr the now obsolete Job Tain- Jing Partnership Act (JTPA) program were adjusted for local economic conditions Broader Social Standards Public interest in a program or ast of social policies may extend beyond the performance sandard for prtipation and outcomes set ‘Tble 4.2. Boaluatng Program Implementation Using Leiaiely Mandated Standards or Program Content ecenapof Pros Tet Di ot Net ad Hol Mining Dio renin Ten Med Popa Eat aon 0 Pn 98-108). Pan don rare (0) rent) Mig de) Tae nde * 2 ‘ee mo a ‘thes use interim : , sched sn pie Prot igh liy Secspae cena Prone | oc oabein 9-199 progr data sone ning Pomieyeunransoee 3 * ald en soamuty o ° i Pasepeiniiepdont u 2rat A GUIDE TO IMPLEMENTATION RESEARCH ‘Teble 3. Ue of Fadl Pasciprion tended asa Bate Bencnark Mindary — Mandy 1982 poicpton Nag Tae kas 5 me vane fo Quan “8 ewok om Now Yk forth in egisation or ina program’ administrative rales nd guide- lines. Ths arial tre nthe cae of welare reform. Indeed, the debate surrounding the passage of national wellae reform lgisation went far Beyond seting benchmarks for the TANF work pategation fate. Among the broader goals anociated with wllae reform ae “ecreases in poverty and out of wedlock births, and increases in chil support payments “An example of rescech concerned with broader scil measires nd standards evaluating well reform are the so-called "eaers sie” “Those studies have oeused on the po TANF experience of frais that hve exited welfare ine national welfare reform Inlided among the ‘many outcome measures observed ae poverty fates and instances of -ardship snc leaving welfare, Table 4.4 isan example of outcome me srs of miter hardshipefore and aflerleaving TANF ora syatess report on aseleed roup of leavers studs, (Nowe that ers studies are 4 spelled ype of implementation research tat focuses on the out- Comes experianced by program purtcpuns They aenotimpac: ties, {they mike no attempt oink outomes topics series oF admis istrative practices) Industry Standards or Exemplary Programs Industry tandadsare quantitative and qualitative measures of individ ta worker local ffi, or program att output codes of conduct "ny other performance ret benchmarks or expectations In welfare Feform programing, for example, sich standards could potentially twablsh anges of reasnable worker cneloads act of bet races fo ‘seework, or national eticaion standards for TANF caseworkers ‘While dusty standards ae widely used in other professions, such ae ‘medicine education 2ecounting chiles services ard law, however, ‘ible 4.4, Use of Broader Soil Standard as an Evaluative Benchmark ‘eae pec Sane Di isp Se CrTANF I) la ‘ae oiigped mason on » ° had ieee 2 ‘ en wos oa at ence a s Reid ind tom bobs 5 “ Sted nls 2 1 (iar ep 2 a Chernov 2 3 ‘Soha tethee ary ben dsp inthe adminitain of cahassnace relict work pgm ‘ne pos cen the ck of nds standards TANE pro wansisthat he abo an elighiy worker the ox comon eof Stier in TANF programe diferent crn sates and rae Ica county oes within he same wate Boh he ls giding trevor andthe fps hil and expnsiy equ iy the ph may difer pending om se cicumtancce were TTANE ogame ws incesigy onthe ending ofthe se ‘ore inemngclsbars tempoymet inking er th {porate evs and monkoring ht prog ond como indeperdenesindney dads rete staal ann ® ‘Far more common than industry standards are references to exem- lar porams and program performance For expe wiles Ipc ton of the Ellon Greer ven for ndcordenee {GAIN program dein ofthe progam chctrenes nal ‘dy count it found partis pre ets or Rr Count i, Fedander and rsdn 954225, Fcexps ‘heide GAIN peu podaced bth using is and elie tring and ws he omy dy coantyt have pose Bnet ot ‘inom bh arpa and goremment popecties “The inings fr ede Gales prey onportanin wo inj way the mle erm debates preceding te are of PRNORA in. hey demons tts poet ond implementa welarcto'merk pram in wheh ceverybodh ine? they nsngespecoe and ped enc) rae oto tk programe Sen thet ocdtaone he pe th Reid GAIN progr tht seemed expan cee By ompaingin a otal contladiateedopc Salinas anaioncramongt iA sta cote ie stad suthors were abe to develop hypothess stout why Renae Cathie invest Secta 1 due Reco anand 19) Onc he mt npr sews as tt Rese ph ‘ophialandadmiiratonphacon pupae ins lavofa chen work sil eprence, or dationastinment 2 sine rogram eet ya shin the dike sue Ta pote ered 0 make ee dee ‘ordinances pospecieson th vende pop The cxemplary performance of Riverside GAIN has helped led 9 wide dept he" Work Pr mnde of wee refor inmany othe states andeountes Profesional Experience and Judgment Semesinesaprgram maybe nique or ene ne, witout spec program moss set ofrnune sandr or dt pit ogra Sapeince pits whit may be cated. Inthe tance pe ‘meration srs may ace callupon te kaowegeof prior feeearch and exercnce in sexing Siar programs to develop & {Game eerene for lain "\ common apizaion of ths approach st evdoplsons abou exenpary program mods ort practi bascdon easement ‘expr int eld. or exemple in aceprt ed el Sree in Shngul Elcom, Mecatun an Ras (99D) cane posts of ie ecpr stts ekted wth he suitance of pul of exper shoe ange a rn sn nan Hey ems This ype rich prod can then nfra the expansion Wit new or inmate sve. hy important drawback of he expe irom approach tht the standards wed fen reeset one oF ited mar of persteson what eos bet races These typesot rear findings at stngest whebcled up by mor obj: {he standrd of pogeam perormance. ‘Stakeholder Opinions and Judgment ‘final st ofan sandr frase program implementation {restlcbolde opinion an jamente Thos cles pope ob “Mons and slow aoty progr pps ad fontine Sroka’ pepe ometimes en bie veo Towel progam i working "The previ chepee duced gathering information about take- teeter and opinion asa deri eee o 4 30a Shourthepogramentonmen and ode culturalconten, Busta Hela epbacato hae pal aly assesment tai ped teh cation and wih sume imight so what coum "suceosal Implementation foreechstcheboler goapor erspete. Fom the profi of plcynkos an ageny manage sch aleholersesmens may be valuable tole for targeting rxoures nceded to Improve program opeatons andres, A clint survey fom the National Evaluation of Wifire-to-Work ‘Strategies included client assessment of specie implementation sues and progam effectiveness (Scrivener etal 198, 135-36) Table 4 pre. sents selected items from the sorvey ‘A survey fr the evaluation ofthe To Strengthen Michigan Failiss (TSME) weliae reform program included some questions asking work: esto anesthe program's effectiveness and quality an level of imple. entation (Hargreaves eta. 1995). Table 4 summarizes some af the findings rom the survey. ‘Tetle4.5, Ue of Cen erspecesinEvlueting Program nplementition ‘ational hatin a ilo i Seago Sel Cle Surya ‘ais sine ptt Taper tne abet t sti an ecennpevaobeioeFO8Swaswould 25 ak) ny ak enorcenet and anoang Terra yh etme ‘ont ene i mca cen vin ete JOBS aon STB ated mentnctierals Fretonot dfecien of ORS ecevagrwtothaktheprgm 83 SMHS Sample sie kas om “ebieAS, Uo Worker erative in Eauating Program Inpeentarion “hs Rayos fm Wr ane Fos Baio Toston Mic Fer Fe ne Be qi andrei hewn you were wh wer mn bn afc the ligemeston of TSM? Dec ret crea! Neva hey hoy coe that tonod wie! Seve it ta a Explaining Lmplementation From pa poi fe: xing papa een — urundigwapotanseyersing an perorngni Separate prof implements reerd fr exlning, Tapmmaton eseetry ocx eth oe eowting epee so polars nd program pes‘can make neded adjustments. Moreover itis through explanation that searchers try to uncover why succesful programs work wel so that their lessons maybe used ffi other program ites orto guide fare progam replication Although explsning program implementation canbe welt can sto be controversial nd iia. The general reson that explanation implies cawsation, To explain something in socal sience to make s him about causality. Hypothesizing about xual connections nimple inelation serch isk ray ofc ne elo for example: “ + Thee are no wie enabloked a acted techie sadarde forcast anays i plementation ear tegen ‘mpit cations. implementation sais dost havea we stele ody fc snr or eign ok Ingfor casa connections Inptehisiseeae e purpose of inpementaton esearch aro vied tha thee common {nate gl wound wht onto rece rc approaches ‘Withous core boy of ace ths. mpementato re seach camot ognize dc about hich ipptoch oa what purpose and in what cacuntnces, Deano abost Impact analy amos sveys bein wth he shared spun Shttheipect of progam Xs the aie betwen what ap tenet under prope X and what would have happened he hence of The jor es of date sho bet eae “na wouid hte Happened inte aseee OFX" Ro woh oe cones yet ext aerom il inplamertaon rear appeaches srdimetds The approach edn ipementton reach sometne re on ara ake fom fons sec dns, One ot Smplemeatsion esearch rest stengthe is eles sto bene ofis grat weakness Tht i method ane ison en tn de seek 2s ola seer corm, pct noel 2 emngy~wih ite here pein aie toute dori The rca plato ipmentation archon oe tern teri how pops rete degli rated The vey ise of where ook for eases expty oF ‘et reson evry of ort sbst how prams wk Fares nan ssa hein may View ol programs rer nrg ose exon et il ge oro nox rae a 9 tha in Font ce Be ane Coward 19) I hs ements eee becomes he sted how ey programs an serene en power andi Suc PICS noel charged hors sboutprowam desig and implementation ott ifering views of whee to ook for sons of prem operators oes For exams cy sation of plemcron earch conceained progr encon aol poss of waingsiin “ote aretnes down rough ste anda age nis n [Edt param woke In pr he er of eel ura cee Ted in more deta below) est tur the more lau Seon then. ina. scorn tthe psp “te ecbaremerny pty satay determined the ine Spine ta froin wore inert wth he public In min Sobleagencs suches tose povide cash stance, tne ree Mang lise of setanaton mach endorser ‘Rare sevcop coping mecaninms that determine bw hy “Tiere ad aru benefit Tos coping necheisa Satter any des clash of poly. Zo ude. JR JRow and wy ply prmnpemsted a doestet eve PEt segues senor ms undentand the ines Pei anaes fhe fone aff and ot mecsrly SEE y unc drecies om above ans lon he evel ain site the luck consensus arn ce ones in inplementr son aca us fe tl and get pass choi See appningsi i The eine tein reviews soe ci tieappences dt cxplin implementation Using the Program Model as a Provisional Theory ofthe Pegeam ut he pera noel may bed veer pint or ag eA atncunst wo hep whys proma OPE=NSand performing at As dhcused in chapter. progrns odes her implitlor exp bued on hor oreo tease Causaon and buman beavon’ Within the context of pov ‘cpu of hyped guy re neo ree "why pop do nocopeae ss "Wi this approsch, when eseacersolerve vane fom planned peaions or epee ret they may ret the Pog de of thot to determine which prs a ec chan a aang der seffom the ode Ne that nt varanone neta he erate Brograrsepsting as planned a pacha se behing fxpeted no exlation beyond the program ery needed: Howe eve ifll-pars ofthe progam made ea pac and operating es aie, and he rogram tl cs ot shee expected real the Dean mon ei son. example the wc of rogram models to! rep impemenationprblems come fm an elunon oa xy we fae corm éemnszaon in Asbny te Anuar Sat Saliciency trough Employment and Tsing Service (ASSETS) prota ee Hargresen ct l 39:Wer tal 1987) Ths Semon ‘ration program ested swt of poly nave ining corsa ‘ating bod sampsand AFDC in einglecsh gra sinpifngand ‘seaming eligi ks and proctaues, und bree he ‘airmen for fecipentsa partite ncmpaymer nd ng sevice One ofthe impr ons the pret waoncea the ceployment and eins of wel en and deere te depen Enron welae bench {tpt ofthe ASSETS program dg, enemas ase ert and feral cia wheto be aed an stated sn Stave satan However, eats thesis sem went pe when ASSETS wa imple, he rploymet an ain lg requiem war admired sete By ne naa ind eiten rot enforced arly aya ek of ASSETS: sami eghty rocesandbeaweaf te db innpementing the stoma adn “etineser ASSES scaly iced wee parton rc pilot counts wibootncesng cies crpaymen nd euriogs Au the employment an trining sat ws ena yp an running the wef thes series neve vaed po imendd scale ‘er the cous ofthe evauton sty Recap pr of the arora nl wastes plane, ASSET i a po er np oteme te ratte af he CAIN program in Carn a exam of tow emt Fasano hg roan ode no 40 REO etal Bor, Wan 9!) CAIN was ananbitow serch set erm hat ater plement ong Fe Moussa for else cpr cna nas i dened ast elm nd anc pnd dough cick county iplemesting GAIN had soe Dewy in Seige sae seam cng he roa’ as mot cies Snemploty ascamer cme acing sata dacs om te at eng mre sc etcaon ‘futcetng uation we ashy gens che of wate Tete ito seh or emer a6 eset comonert. Te Fee oe oui sped ry pupae an eens at lynne he ole, Those ots “fg bs eduction fr chs ith poor testo tai eerie evs onl ead ber td more sae rye ax demonstration counties also made diferent decisions regard ing soviet ampsaeegine eyo pre ‘Sept diate Reco engausiedty isempnatsonssione nort.acnent esas se tr wenty ethos os props ome Breaths os of he eters outs were ae we 12S thre ceding uso pate nbs eduaon TER brcenringin jos the theory feed ye osm) tt il eset nena cpl od ae paret hte and mov sal poe Albough the aera ec ence corer png seach Soe isa wih i aston stamens ad beer Spent rests Daastton ih hess erento EReTRas Count atando senha oben sees ethene osamesoncl GA poss Sookie? pnp ‘oscar farmer or xing promos sonnets cue in ope easton rare == ERIE ory of change approseh-scmingl promis to produce Selon sau pgm pac dopo of cle and EEE EEE EEE438A GUIDE To IMPLEMENTATION RESEARCIE beers feteomes"Te ant leverge ofthe pprsch arises frm the deceive smal anil tot soil reas mel tier emit or ipods sso predictors ora heory ofthe progr") out hove the progam itrvetons wil change len baviorand cestanes Once the program hor is eet in deta its predictions maybe tested jum ke how ay eet potest if the pce evento happen and ed te rece hangs ncntbchaviorsndouiee pon hoy Sr tse dso mini pon Aug tether of change approach conceal inks rogram eign operations an tans sho hs ns ar be ‘Gramoretaditonel inact espn’ Forexmplet focscsnensely ‘on prticpant experiences in pogram score services and the “hangs n print ators allowing ts enn ade tinea oneratnaegeene oc ofthe progam ‘Those ube experiences my infnce sd hnge pcan ot commen Moree een the pga hsm eon en ome, itnay note forthe reson xpued by the pagan try. “Tetheoy of changes areata when ts impsbleor ndesiabl to impleneand sign xpernent However Siete acelin sh cn rn mew spistiow ois ppronch se edger aout theo of th Pra hery Pa tec eile tented ayplnned and the pede changin outconnes happen ts Posie tate program es hoe changes ato Kaw ta th erin and'o know how much ofan iac the program ape Almost aay requis each deg that can exit te counter inctah or wha thon oures wold ne been nthe see he Progam ‘The Perspective of Steet-Level Bureawertey™ Sinton andar oration in undastandingthe serine pb ttn tn ty as some ‘nyoranconttsons to miensrnon rary egg at ‘ontine wks nner sn otha spe bam puyol inte sete reser ped cs ttn an ee negested ans ue sisson is ercarenaon Sar ‘cual Oring of ras change notes and oh a erative care, sch under vlfve reform. 1 pricy during, hone tins that essed routines and behaviors of one age? ‘Mork maybe ahd fo change. Acordingto the theory behind re Tee bueayrac tue poly ceed out ofthe tension betwen he (Gomandsof er administrate lesan procedures and frontline wok {ined devlopanew-comfort ein dir thts, As cae fractions ef the swec-evel orientation implementation rscrch (ries (ogkin 20515) ‘hap sont sae en ky pletion ae no TRS pte duane woe dears ee ln nemesnaty sty n ncray By fg oo me acon nde med oss oat ch Pee sey henna eer ponte pe pene aren Sauer tae pun od pve stdbe we thw ty = Melina ene enya “The get apes ofthe siete! perspective that i focses rears aunaonn wst hapers to tial palette fs Six dtr uth approach av on some portent ions FOr ‘Sine the bas for sometimes ther intel conhsons about how SEI wore drt shapes py penta ina ier pro Gramineae om the sey eves few worker ines Sd pera ebscresons Osea the sere perpctie td seit cmt be td to cra Tesch base rd ‘irerycsns abot how ne poly or ogra wile reed Fring workers snot abeay cl ba ngs oat wt sey fpongonet the ste leean be aise. ‘iconic este ep: svc inauping ply pgra manage. Ahough oka “Re tratan managers Geary nave to tik the ety of set evel eens nt acu iis ancy miter deve ways iit Tee worer dsm of pang hat aren nested drecions Somutimes upper level mage! concen abou oo ma worker ‘Teo in the implemenstion of poe cn ed eedteting Stent nes fo empl an ci eason 2 ‘tae saci progam soncoriance could harper csenorts Inthe elt t nd the mow flee wo o move TANE fee tows economic ndependene ina stady ofthe pletion of rt exponsiy contrat seep Bin 1997 emis ow worher deo ihn hes