University of Pennsylvania

ScholarlyCommons
Marketing Papers

11-21-2011

Illusions in Regression Analysis

J. Scott Armstrong
University of Pennsylvania, armstrong@wharton.upenn.edu

Armstrong, J.S. (2011). Illusions in Regression Analysis. Forthcoming in International Journal of Forecasting, 2012. This paper is posted at ScholarlyCommons. http://repository.upenn.edu/marketing_papers/173 For more information, please contact repository@pobox.upenn.edu.

Illusions in Regression Analysis

J. Scott Armstrong (armstrong@wharton.upenn.edu) The Wharton School, University o !ennsylvania, !hiladelphia !A "#"$% &orthcoming, International Journal of Forecasting, '$"' Abstract Soyer and (ogarth)s article, *The +llusion o !redicta,ility,- shows that diagnostic statistics that are commonly provided with regression analysis lead to con usion, reduced accuracy, and overcon idence. .ven highly competent researchers are su,/ect to these pro,lems. This overview e0amines the Soyer1(ogarth indings in light o prior research on illusions associated with regression analysis. +t also summari2es solutions that have ,een proposed over the past century. These solutions would enhance the value o regression analysis. 3eywords4 a priori analysis, decision1ma5ing, ex ante testing, orecasting, non1e0perimental data, statistical signi icance, uncertainty

The *+llusion o !redicta,ility4 (ow 6egression Statistics 7islead .0perts,- ,y .mre Soyer and 6o,in (ogarth, is dedicated to the memory o Arnold 8ellner ("#'91'$"$). " + am sure that Arnold would have agreed with me that their paper is a itting tri,ute. :iven the widespread use o regression analysis, the implications o the article are important or the li e and social sciences. .mploying a simple e0periment, Soyer and (ogarth ('$"", herea ter *S;(-) show that some o the world)s leading e0perts in econometrics can ,e misled ,y standard statistics provided with regression analyses4 t, p, &, 61s<uared and the li5e. S;( ollows a rich history on the illusions o predicta,ility associated with the use o regression analysis on non1e0perimental data. A loo5 at the history o regression analysis suggests why illusions o predicta,ility occur and why they have increased over time = to the detriment, as S;( show, o scienti ic analysis and orecast1a,ility. ' Historical view of illusions in regression analysis 6egression analysis entered the social sciences in the ">9$s with the pioneering wor5 ,y &rancis :alton. ?ut *least s<uares- goes ,ac5 at least to the early ">$$s and the :erman mathematician 3arl :auss, who used the techni<ue to predict astronomical phenomena. &or most o its history, regression analysis was a comple0, cum,ersome, and e0pensive underta5ing.

"&or more a,out Arnold 8ellner, see *+n memory o Arnold 8ellner, a great person and scientist,- International
Journal of Forecasting, '9 ('$""), #@"1#@9, ,y Antonio :arcAa1&errer.

'There is also an e0tensive literature in which leading econometricians have warned o illusions associated with
regression (e.g., see the review ,y 3ennedy, '$$'). 3armi and Shapiro ("#>$) made un o regression analysis procedures in their *data torture- paper.

Bonsider 7ilton &riedman)s e0perience more than orty years prior to user1 riendly so tware and the personal computer revolution. Around "#%%, as part o the war e ort, &riedman was as5ed to analy2e data on alloys used in tur,ine engine ,lades. (e used regression analysis to develop a model that predicted time to ailure as a unction o stress, temperature, and some metallurgical varia,les representing the alloy)s composition. C,taining estimates or &riedman)s e<uation ,y hand and calculating test statistics would have ta5en a s5illed analyst a,out three months la,or. &ortunately, a large computer, ,uilt rom many +?7 card1sorters and housed in (arvard)s air1conditioned gymnasium, could do the calculations. +gnoring time re<uired or data input, the computer needed %$ hours to calculate the regression estimates and test statistics. Today, a regression o the si2e and comple0ity o &riedman)s could ,e e0ecuted in a,out one second. &riedman was delighted with the resultsD the model had a high 6 ' and the varia,les were *statistically signi icant- at conventional levels. As a result, &riedman recommended two new improved alloys, which his model predicted would survive several hundred hours at high temperatures. Tests o the new alloys were carried out ,y engineers in an 7+T la,oratory. The resultE The irst alloy ,ro5e in a,out two hours and the second one in a,out three. &riedman concluded one should ocus on tests o outputs ( orecasts) rather than statistically signi icant inputs. (e also opined that *the more comple0 the regression, the more s5eptical + am- (&riedman and Schwart2 "##"). :iven that doing regressions was e0pensive, it was sensi,le to rely heavily on a priori analyses. Cpinions a,out the proper model (e.g., which varia,les are important) should not ,e ,ased on opinions or untested ideas even though they may ,e o ered ,y amous people. (The names 3eynes and Samuelson spring to mind.) +nstead, the evidence should ,e ,ased on meta1analyses. 7eta1analyses produce more accurate and less ,iased summaries than those provided ,y traditional reviews as shown ,y Bumming ('$"'). When possi,le, meta1analyses should ,e used in ma5ing decisions as to what varia,les to includeD speci ying the e0pected direction o the relationshipsD and speci ying the nature o the unctional orm, ranges o magnitudes o relationships, and si2e o e0pected magnitudes o those relationships. +t is also important to determine relationships that can ,e measured outside the model ,ased either on common 5nowledge ( or e0ample, ad/usting or in lation or trans orming the data to a per capita ,asis) or on analyses o other data. Analysts should use simple pre1speci ied rules to com,ine a priori estimates with estimates o,tained rom regression analysis ( or e0ample, one might weight each estimate o a relationship e<ually, then re1run the regression to estimate the coe icients or the other varia,les). This approach was recommended ,y Wold and Jureen ("#FG), where it was called *conditional regression.- + thin5 o it as a Hpoor man)s ?ayesian analysis.- + pre er it to ormal ?ayesian orecasting methods ,ecause o its clarity a,out the nature o each causal relationship and the related evidence (and ,ecause + have a strong need or sleep whenever + try to read a paper on ?ayesian orecasting.) +n this paper, + re er to it as an a priori analysis. +n the mid1"#@$s, + was wor5ing on my !hI thesis at 7+T. While the cost o regression analysis had plunged, it still involved punch cards and overnight runs. ?ut the most time1consuming part o my thesis was the a

'

priori analysis. ?e ore doing any regression analyses, + gave John Jittle, my thesis advisor, a priori estimates o the coe icients or all varia,les in a demand1 orecasting model. As it turned out, these purely a priori models provided relatively accurate orecasts on their own. + then used regression analyses o time1series, longitudinal, and household data to estimate parameters. These were used to revise the a priori estimates. This procedure provided orecasts that were su,stantially more accurate than those rom e0trapolation methods and rom stepwise regression on the complete set o causal varia,les that were considered (Armstrong "#@>a,,). Iespite warnings over the past hal 1century or more (8ellner, '$$", traces this ,ac5 to Sir (arold Je reys in the mid1"#$$s), a priori analysis seems to ,e giving way among academic researchers to the ,elie that with enormous data,ases they can use comple0 methods and analytical measures such as 6 ' and t1statistics to create models. They even try various trans ormations or di erent lags o varia,les to see which ,est it the historical data. .inhorn ("#9') concluded, *Just as the alchemists were not success ul in turning ,ase metal into gold, the modern researcher cannot rely on the Hcomputer) to turn his data into meaning ul and valua,le scienti ic in ormation.- Crd ('$"') provides a simple demonstration o how standard regression procedures, applied without a priori analyses, can lead one astray. Forecast accuracy and confidence We have ample evidence that regression analysis o ten provides use ul orecasts (Armstrong "#>FD Allen and &ildes '$$"). 6egression1,ased prediction is most e ective when dealing with a small num,er o varia,les, large amounts o relia,le and valid data, where changes are e0pected to ,e large and predicta,le, and when using well1esta,lished causal relationships = such as the elasticities or income, price, and advertising when orecasting demand. (owever, there are illusions that reduce the orecast accuracy and lead to overcon idence in regression analysis. + discuss ive o them here4 Complexity illusion: +t seems common sense that comple0 solutions are needed or comple0 and uncertain pro,lems. 6esearch indings suggest the opposite. &or e0ample, Bhrist ("#@$) ound that simultaneous e<uations provided orecasts that were more accurate than those rom simpler regression models when tested on arti icial data, ,ut not when tested out o sample using real data. 7y summary o the empirical evidence concluded that increased comple0ity o regression models typically reduced orecast accuracy (Armstrong "#>F, pp. ''F1'G'). 8ellner ('$$") reached the same conclusion in his review o the research. (e also ound that many users have ,ecome disillusioned with complicated models. &or e0ample he reported *the &ederal 6eserve ?an5 o 7inneapolis decided to scrap its complicated vector autoregressive (KA6, i.e., Kery Aw ul 6egression) models a ter their poor per ormance in orecasting turning points, etc..vidence avoring simplicity has continued to appear over the past <uarter century. Why then is there such a strong interest in comple0 regression analysesE !erhaps this is due to academics) pre erence or comple0 solutions, as

(ogarth ('$"') descri,es. Somewhere + encountered the idea that statistics was supposed to aid communication. Bomple0 regression methods and a loc5 o diagnostic statistics have ta5en us in the other direction. The solution is that, when speci ying a model, rely upon a priori analysis. &ollow 8ellnerLs ('$$") advice and use Cccam)s 6a2or.G +n other words, 5eep it simple. Start with a very simple model, such as a no1change model, and then add comple0ity only i there is e0perimental evidence to support the complication. And do not try to estimate relationships or more than three varia,les in a regression ( indings rom :oldstein and :igeren2er, '$$#, are consistent with this rule1o 1thum,). Illusion that regression models are sufficient: &orecasts are o ten derived only rom what is thought to ,e the ,est model. This ,elie has a long history in orecasting. &or solutions, + call your attention to two o the most important indings in orecasting. &irst is that the naMve or no1change model is o ten <uite accurate. +t is to orecasting what the place,o is to medicine. This approach is especially di icult to ,eat in situations involving comple0ity and uncertainty. (ere, it o ten helps to shrin5 each coe icient toward having no e ect (,ut remem,er to re1run the regression to cali,rate the constant term). Second is the ,ene it o com,ining orecasts. That is, ind two or more valid orecasting methods and then calculate averages o their orecasts. &or e0ample, ma5e orecasts ,y using di erent regression models, and then com,ine the orecasts. This is especially e ective when the methods, models, and data di er su,stantially. Bom,ining orecasts has reduced errors rom a,out "$N to F>N (depending on the conditions) compared to the average errors o the uncom,ined individual orecasts (:rae e, et al '$""). Illusion that regression provides the best linear unbiased estimators: Statisticians have devoted much time to showing that regression leads to the ,est estimates o relationships. (owever, studies have shown that regression estimates produce ex ante orecasts that are o ten less accurate than orecasts rom *unit weights- models. Schmidt ("#9") was one o the irst to test this idea and he ound that unit weights were superior to regression weights when the regressions were ,ased on many varia,les and small sample si2es. .inhorn and (ogarth ("#9F) and Iana and Iawes ('$$%) show the conditions under which regression is and is not e ective relative to e<ual weights. Cne good characteristic o regression estimates is that they ,ecome more conservative as uncertainty increases. Un ortunately, some aspects o uncertainty are ignored. &or e0ample, the coe icients can *get credit- or important e0cluded varia,les that happen to ,e correlated with the predictor varia,les. Adding varia,les to the regression cannot solve this pro,lem. +n addition, analysts searching or the ,est it, and pu,lication practices avoring statistically signi icant

G8ellner traced this idea ,ac5 to Sir (arold Je reys in the "#G$s.

results o ten de eat conservatism. Thus, regressions typically over1estimate change. +oannidis ('$$F, '$$>) provides more reasons why regressions over1estimate changes. Cne solution is to com,ine orecasts, and among the alternative models, include the naMve model. +n particular, or pro,lems involving time1series, damp the orecasts more heavily toward the naMve model orecasts as the orecast hori2on increases. This is done to re lect e ects o the increasing amount o uncertainty in the more distant uture. Illusion of control: Users o regression assume that ,y putting varia,les into the e<uation they are somehow controlling or these varia,les. This only occurs or e0perimental data. Adding varia,les does not mean controlling or varia,les in non1e0perimental data ,ecause many varia,les typically co1vary with other predictor varia,les. The pro,lem ,ecomes worse as varia,les are added to the regression. Jarge sample si2es cannot resolve this pro,lem, so statistics on the num,er o degrees o reedom are misleading. Cne solution is to use evidence rom e0perimental studies to estimate e ects and then ad/ust the dependent varia,le or these e ects. Fit implies accuracy illusion: Analysts assume that models with a ,etter it provide more accurate orecasts. This ignores the research showing that it ,ears little relationship to ex ante orecast accuracy, especially or time series. Typically, it improves as comple0ity increases, while ex ante orecast accuracy decreases = a conclusion that 8ellner ('$$") traced ,ac5 to Sir (arold Je reys in the "#G$s. +n addition, analysts use statistics to improve the it o the model to the data. +n one o my Tom Swi t studies, Tom used standard procedures when starting with G" o,servations and G$ potential varia,les. (e used stepwise regression and included only varia,les where t was greater than '.$. Along the way, he dropped three outliers. The inal regression had eight varia,les and an 61s<uare (ad/usted or degrees o reedom) o $.>F. Oot ,ad, considering that the data were rom 6andLs ,oo5 o random num,ers (Armstrong "#9$). + traced studies on this illusion ,ac5 to at least "#F@ in an early review o the research on it and accuracy (Armstrong "#>F). Studies have continued to ind the it is not a good way to assess predictive a,ility (e.g., !ant and Star,uc5 "##$). The o,vious solution is to avoid use o t, p, &, 61s<uared and the li5e when using regression. Using regression for decision-making 6egression analysis provides an o,/ective and systematic way to analy2e data. As a result, decisions ,ased on regression are less li5ely to ,e su,/ect to ,ias, they are consistent, the ,asis or the decisions can ,e ully e0plained = and they are generally use ul. The gains are especially well documented when compared to /udgmental decisions ,ased on the same data (:rove and 7eehl "##@D Armstrong '$$"). (owever, two illusions, statistical signi icance and

correlations, can reduce the value o regression analysis. tatistical significance illusion: S;( incorporate the illusion due to tests o statistical signi icance. 7eehl ("#9>) concluded that *reliance on merely re uting the null hypothesis . . . is ,asically unsound, poor scienti ic strategy, and one o the worst things that ever happened in the history o psychology Schmidt ("##@) o ered the ollowing challenge4 *Ban you articulate even one legitimate contri,ution that signi icance testing has made (or ma5es) to the research enterprise (i.e., any way in which it contri,utes to the development o cumulative scienti ic 5nowledge)E- Cne might also as5 i there is a study wherein statistical signi icance improves decision1ma5ing. +n contrast, it is easy to ind cases where statistical signi icance harmed decision1ma5ing. 8ilia5 and 7cBlos5ey ('$$>) document in devastating e0amples ta5en rom across the sciences. To o er another e0ample, (auer ('$$%) demonstrates harm ul decisions related to automo,ile tra ic sa ety, such as the *6ight1turn1on1red decision.- Bumming ('$"') descri,es additional e0amples o the harm caused ,y the use o statistical signi icance. The commonly recommended solution is to use con idence intervals and avoid the use o statistical signi icance. Statisticians argue that statistical signi icance provides the same in ormation as con idence intervals. ?ut the issue is how people use the in ormation. Signi icance levels lead to con usion even among leading researchers. Bumming ('$"', pp. "G1"%) descri,es an e0periment showing that when researchers in psychology, ,ehavioral neuroscience, and medicine were presented with a set o results, %$N o FF participants who used signi icance levels to guide their interpretation reached correct conclusions. +n star5 contrast, #FN o the F9 participants who thought in terms o con idence intervals reached correct conclusions. Correlation illusion: We all claim to understand that correlation is not causation. The correlations might occur ,ecause A causes ?, or ? causes A, or they are each related to B, or they could ,e spurious. ?ut when presented with sophisticated and comple0 regressions, people o ten orget thatD 6esearchers in medicine, economics, psychology, inance, mar5eting, sociology, and so on, ill /ournals and newspapers with interesting ,ut erroneousPand even costlyP indings. +n one study, we had an opportunity to compare indings rom e0periments with those rom analyses o non1 e0perimental data or '% causal statements. The directional e ects di ered or > o the '% comparisons (Armstrong and !atnai5 '$$#). 7y conclusion is that analyses o non1e0perimental are o ten misleading. This illusion has led people to ma5e poor decisions a,out such things as what to eat (e.g., co ee, once ,ad, is now good or health), what medical procedures to use (e.g., the re<uently recommended !SA test or prostate cancer has now ,een shown to ,e harm ul), and what economic policies the government should adopt in recessions (e.g., trusting the government to ,e more e icient than the mar5et). According to 8ellner ('$$"), Sir (arold Je reys had warned o this illusion, and, in "#@", re erred to it as

the *most undamental allacy o all.The solution is to ,ase causality on meta1analyses o e0perimental studies. Discussion An o,vious conclusion rom the study ,y S;( is to de1emphasi2e descriptive statistics or regression pac5ages. So tware developers should provide statistics on the a,ility o alternative methods to produce accurate orecasts on holdout samples as the de ault option. They could allow users to clic5 a ,utton to access the traditional regression statistics, a warning la,el should ,e provided near that ,utton. S;(, echoing &riedman, emphasi2e that scienti ic theories should ,e tested or their predictive a,ility relative to other methods. Crd ('$"') deplores the act that ew regression pac5ages aid in such analyses. +t would ,e help ul i so tware providers would ocus on ex ante testing ,y ma5ing it easy to simulate the orecasting situation. &or cross1sectional orecasts, use /ac55ni ingPthat is, use all ,ut one data point to estimate the model, then predict or the e0cluded o,servation, and repeat until predictions have ,een made or each o,servation in the data. &or time1 series, withhold data, then use successive updating and report the accuracy or each orecast hori2on. These testing procedures are less li5ely to lead to overcon idence ,ecause they include the uncertainty rom errors due to over1 itting and errors in orecasting the predictor varia,les. So tware pac5ages should provide statistics that allow or meaning ul comparisons among methods (Armstrong and Bollopy "##'). The 7d6A. (7edian 6elative A,solute .rror) was designed or such comparisons and many so tware pac5ages now provide this statistic, so it should ,e among the de ault statistics, along with a lin5 to the literature on this topic. Io not provide 67S. (6oot 7ean S<uare .rrors) as it is unrelia,le and unin ormative. Allen and &ildes ('$$") note that since "#>F there has ,een a su,stantial increase in the attention paid to ex ante orecast comparisons in pu,lished papers. This is consistent with the aims stated in the ounding o the Journal of Forecasting in "#>', and su,se<uently, the International Journal of Forecasting. 6egression analysis is clearly one o the most important tools availa,le to researchers. (owever, it is not the only game in town. 6esearchers made scienti ic discoveries a,out causality prior to the availa,ility o regression analysis as shown ,y &reedman ("##") in his paper aptly titled *Statistical models and shoe leather.- (e demonstrates how ma/or gains were made in epidemiology in the ">$$s. &or e0ample, John Snow)s discovery o the cause o cholera in Jondon in the ">F$s came a,out rom *the clarity o the prior reasoning, the ,ringing together o many di erent lines o evidence, and the amount o shoe leather Snow was willing to use to get the data.- These three characteristics o good science, descri,ed ,y &reedman, are missing rom most regression analyses that + see in /ournals. We would ,e wise to recall a method that ?en &ran5lin used to address the issue o how to ma5e decisions when many varia,les are involved (Spar5s, ">%%). % (e suggested listing the varia,les related to the choice ,etween

%?en/amin &ran5lin)s letter to Joseph !riestley can ,e ound at4 http4QQwww.procon.orgQview,ac5groundresource.aspE

resource+IR"%9%.

two options, identi y which option is ,etter or each varia,le, weight the varia,les, and then add. !ic5 the option that has the highest score. Andreas :rae e and + have ,uilt upon &ran5lin)s advice in developing what we call the index method or orecasting. The method relies only on a priori analysis (pre era,ly e0perimental indings) to determine which varia,les are important and what is the direction o the e ect or each varia,le. &ran5lin suggested di erential weights, ,ut the literature discussed a,ove suggests that unit weights are a good place to start. 6egression analyses can then ,e used to estimate the e ects o an inde0 score. The inde0 model allows analysts to ta5e account o *the 5nowledge present in a ield- as recommended ,y 8ellner ('$$"). The ew tests to date suggest that the inde0 method provides use ul orecasts when there are many important varia,les and su,stantial prior 5nowledge (e.g., see Armstrong and :rae e '$""). +t can ,e e0pensive to do a priori analyses or comple0 situations. &or e0ample, + am currently involved in a pro/ect to orecast the e ectiveness o advertisements ,y using the inde0 method. + have spent many hours over a "@1 year period summari2ing the 5nowledge. This led to "#F principles (causal conditionQaction statements), each ,ased on meta1analysis when there was more than one source o evidence. The vast ma/ority o them were su iciently comple0 such that neither prior e0perience nor regression analyses were a,le to discover them. They were ormulated than5s to a century o e0perimental and non1e0perimental studies. Oone o these evidence1,ased principles were ound in the advertising te0t,oo5s and hand,oo5s that + analy2ed. 7any are counter1intuitive and are o ten violated ,y advertisers (Armstrong '$""). .arly indings suggest that the inde0 model provides use ul orecasts in this situation. +n contrast, regression analyses have met repeated ailures in this area ,ecause there may ,e over F$ principles that were used = or misused = in an ad. As S;( suggest, urther e0perimentation is needed. We need e0periments to assess the a,ility o alternative techni<ues to improve accuracy when tested on large samples o orecasts on holdout samples. Journal editors should commission such studies. Allen and &ildes ('$$") provide an o,vious starting point or research topics as they developed principles or the e ective use o regression ,ased on the e0isting 5nowledge. They also descri,e the evidence on the principles. &or e0ample, on the matter o evidence on one o the diagnostic statistics they state (p. G""), *The mountains o articles on autocorrelation testing contrast with the near a,sence o studies on the impact o autocorrelation correction on Sex anteT orecast per ormance.Un ortunately, so tware developers typically ail to incorporate evidence1,ased indings a,out which methods wor5 ,est, and statisticians who wor5 on new orecasting procedures seldom cite the literature that tests the relative e ectiveness o various methods or orecasting (&ildes and 7a5rida5is "##F). To ma5e the latest indings easily availa,le to orecasters, the +nternational +nstitute o &orecasters has supported the &or!rin.com we,site. The goal is to present orecasting principles in a way that can ,e understood ,y those who use orecasting techni<ues such as regression. S;( recommend visual presentation = and 8ilia5 ('$"') adds support. This seems li5e a promising avenue. Bonsider, or e0ample, the e ectiveness o the communication provided ,y Anscom,e)s Uuartet (see Wi5ipedia).

>

Conclusions Io not use regression to search or causal relationships. And do not try to predict ,y using varia,les that were not speci ied in the a priori analysis. Thus, avoid data mining, stepwise regression, and related methods. 6egression analysis can play an important role when analy2ing non1e0perimental data. Karious illusions can reduce the accuracy o regression analysis, lead to a alse sense o con idence, and harm decision1ma5ing. Cver the past century or so, e ective solutions have ,een developed to deal with the illusions. The ,asic pro,lem is that the solutions are o ten ignored in practice. S;( shows that this pertains even among the world)s leading researchers. 6esearchers might ,ene it ,y systematically chec5ing their use o regression to ensure that they have ta5en steps to avoid the illusions. 6eviewers could help to ma5e researchers aware o solutions to the illusions. So tware providers should in orm their users. To me, S;()s 5ey recommendation is to conduct e0periments to compare di erent approaches to developing and using models. +t is remar5a,le that so little e0perimentation has ,een done over the past century to determine which regression methods wor5 ,est under what conditions. Arnold 8ellner would not have ,een surprised ,y these conclusions.
Acknowledgements: !eer review was vital or this paper. &our people reviewed two versions o this paper4 !. :eo rey Allen, 6o,ert &ildes, 3esten B. :reen, and Stephen T. 8ilia5. .0cellent suggestions were also received rom 3ay A. Armstrong, Jason Iana, Antonio :arcAa1&errer, Andreas :rae e, Ianiel :. :oldstein, :eo Bumming, !aul :oodwin, 6o,in (ogarth, !hilippe Jac<uart, 3eith Crd, and Bhristophe Kan den ?ulte. This is not to imply that the reviewers agree with all o the conclusions in this paper.

References
Allen, !. :. ; &ildes, 6. ('$$"). .conometric orecasting. +n J. S. Armstrong. !rinciples of Forecasting, ?oston4 3luwer Academic !u,lishers, Armstrong, J. S. ('$""). .vidence1,ased advertising4 An application to persuasion. International Journal of "dvertising, G$ (F), 9%G19@9. Armstrong, J. S. ('$$"). Judgmental ,ootstrapping4 +n erring e0perts) rules or orecasting.- +n J. S. Armstrong. !rinciples of Forecasting. ?oston4 3luwer Academic !u,lishers. Armstrong, J. S. ("#>F). #ong$%ange Forecasting. Oew Vor54 John Wiley. Armstrong, J. S. ("#9$). (ow to avoid e0ploratory research. Journal of "dvertising %esearch, "$, Oo. %, '91G$. Armstrong, J. S. ("#@>a). #ong$range Forecasting for a Consumer &urable in an International 'ar(e t. !hI Thesis4 7+T. (U6J to ,e added) Armstrong, J. S. ("#@>,). Jong1range orecasting or international mar5ets4 The use o causal models. +n 6. J. 3ing, 'ar(eting and the )e* cience of !lanning. Bhicago4 American 7ar5eting Association. Availa,le at http4QQgoo.glQo3VU( Armstrong, J. S. ; Bollopy, &. ("##'). .rror measures or generali2ing a,out orecasting methods4 .mpirical comparisons, International Journal of Forecasting, >, @#1>$. Armstrong, J. S. ; :rae e, A. ('$""). !redicting elections rom ,iographical in ormation a,out candidates4 A test o the inde0 method. Journal of +usiness %esearch, ,-, @##19$@. Armstrong, J. S. ; !atnai5, S. ('$$#). Using Uuasi1e0perimental data to develop principles or persuasive

advertising. Journal of "dvertising %esearch, %#, Oo. ', "9$1"9F. Bhrist, B. &. ("#@$). Simultaneous e<uation estimation4 Any verdict yetE .conometrica, /0, >GF1>%F. Bumming, :. ('$"'). 1nderstanding the )e* tatistics: .ffect si2es. Confidence Intervals and 'eta$"nalysis . Oew Vor54 6outledge. Iana, J. ; Iawes, 6. 7. ('$$%). The superiority o simple alternatives to regression or social science predictions. Journal of .ducational and +ehavioral tatistics , /3 456, G"91GG". .inhorn, (. J. ("#9'). Alchemy in the ,ehavioral sciences. !ublic 7pinion 8uarterly, 5,, G@91G9>. .inhorn, (. J. ; 6. (ogarth ("#9F). Unit weighting schemes or decision ma5ing. 7rgani2ational +ehavior and 9uman !erformance, :5, "9"1"#'. &ildes, 6. ; 7a5rida5is, S. ("##F). The impact o empirical accuracy papers on time series analysis and orecasting. International tatistical %evie*, ,5, '>#1G$>. &reedman, I. A. ("##"). Statistical models and shoe leather. ociological 'ethodology, '", '#"1G"G. &riedman, 7.A. ; Schwart2, A. J. ("##"). Alternative approaches to analy2ing economic data. "merican .conomic %evie*, 0:, Appendi0 %>1%#. :oldstein, I. :. ; :igeren2er, :. ('$$#). &ast and rugal orecasting. International Journal of Forecasting, /;, 9@$1 99'. :rae e, A., Armstrong, J.S., Bu2Wn, A. :. ; Jones, 6.J., Jr. ('$""). Bom,ining orecasts4 An application to election orecasts, Wor5ing !aper. :rove, W.7. ; 7eehl, !... ("##@). Bomparative e iciency o in ormal (su,/ective, impressionistic) and ormal (mechanical, algorithmic) prediction procedures4 the clinical = statistical controversy. !sychology, !ublic !olicy, and #a*, ', '#G1G'G. (auer, .. ('$$%). The harm done ,y tests o signi icance. "ccident "nalysis and !revention, 5,, %#F1F$$. (ogarth, 6. 7. ('$"'). When simple is hard to accept. +n !. 7. Todd ; :igeren2er, :. (.ds.), .cological rationality: Intelligence in the *orld 4in press6. C0 ord4 C0 ord University !ress. +oannidis, J. !. A. ('$$F). Why most pu,lished research indings are alse. !#o 'edicine, ', @#@19$". +oannidis, J. !. A. ('$$>). Why most discovered true associations are in lated. .pidemiology, "#, @%$1@%>. 3arni, .. ; Shapiro, ?. 3. ("#>$). Tales o horror rom ivory towers. Journal of !olitical .conomy. >>, Oo. ", '"$1 '"'. 3ennedy, !. ('$$'). Sinning in the ,asement4 What are the rulesE The ten commandments o applied econometrics. Journal of .conomic urveys, "@, F@#1F>#. 7eehl, !... ("#9>), Theoretical ris5s and ta,ular asteris5s4 Sir 3arl, Sir 6onald, and the slow progress o so t psychology. Journal of Consulting and Clinical !sychology, %@, >$@1>G%. Crd, 3. ('$"'). The +llusion o predicta,ility4 A call to action. International Journal of Forecasting, '> 0001000. !ant, !. O. ; Star,uc5, W. (. ("##$). +nnocents in the orest4 &orecasting and research methods. Journal of 'anagement, "@, %GG1%%@ Schmidt, &. J. ("#9"). The relative e iciency o regression and simple unit predictor weights in applied di erential psychology. .ducational and !sychological 'easurement, 5:, @##19"%. Schmidt, &. J. ("##@). Statistical signi icance testing and cumulative 5nowledge in psychology4 +mplications or training o researchers. !sychological 'ethods, :, ""F1"'#. Soyer, .. ; (ogarth, 6. ('$"'). +llusion o predicta,ility4 (ow regressions statistics mislead e0perts. International Journal of Forecasting, '> 0001000. Spar5s, J. (">%%). <he =or(s of +en>amin Fran(lin. ?oston4 Bharles Tappan !u,lisher. Wold, (. ; Jureen, J. ("#FG). &emand "nalysis. Oew Vor54 John Wiley. 8ellner, A. ('$$"). 3eep it sophisticatedly simple. +n 3eu2en5amp, (. ; 7cAleer, 7. .ds. implicity, Inference, and 'odelling: ?eeping it ophisticatedly imple. Bam,ridge University !ress, Bam,ridge. 8ilia5, S. T. ('$"'). Kisuali2ing uncertainty4 Cn Soyer)s and (ogarth)s *The illusion o predicta,ility4 (ow regression statistics mislead e0perts.- International Journal of Forecasting, '> xxxxx 8ilia5, S. T. ; 7cBlos5ey, I. O. ('$$>). <he Cult of tatistical ignificance: 9o* the tandard .rror Costs 1s Jobs, Justice, and #ives. Ann Ar,or4 The University o 7ichigan !ress.

"$

Oovem,er '$, '$"" (6%9)

""