On the value of non-experimental data

Healing in a Former British colony

Prepared by J. Scott Armstrong (details on him at jscottarmstrong.com). Please inform Scott about errors and also make suggestions (armstrong@wharton.upenn.edu) Scott has taken these slides from adprin.com, a site that he founded. That site contains interactive versions of these slides, along with linked references, videos, and webcasts, all in PPT and PPTX format that you can download.

In a former British colony, healers believed that a distillation of fluids extracted from horse urine, if dried to a powder and fed to aging women, could preserve youth, and ward off diseases.
The preparation become popular and was used widely by older women. Studies of hundreds of thousands of women showed that those who took the drug were healthier.
The source is Avorn, Jerry (2004), Powerful Medicines. New York: Alfred A. Knopf, p. 8.

What potential problems might there be with using these studies of the powder and why?
Adapted from AdPrin.com 2

The former colony is the United States; the time is now; the drug is female hormone replacement products such as Premarin.

Women who watched their health sought best procedures. They were compared with people who put less emphasis on health. Thus, non-experimental data misled doctors.
Experimental studies revealed that long-term use of the house-urine extract was useless; worse, it caused tumors, blood clots, heart disease, and perhaps brain damage.

Conclusion:
Analysis of non-experimental data is not capable of finding what works in complex and uncertain situations, such as advertising.
Adapted from AdPrin.com 3

When does non-experimental data help?

Write your answers, then see the next slide for the evidence.

It helps: 1. in simple situations. 2. When one has much data. 3. In comparison with unaided judgment.

Evidence on the value of non-experimental data

Surowiecki (2004) reports that some football coaches benefit from analyses of non-experimental data. Lewis (2003) describes how analyses of nonexperimental data have aided decision-making in baseball. Finally, the 15 National Basketball teams with at least one statistician on the team won 59% of their games in the first half of the 12009-2010 season, while the 15 teams with no statisticians won 41%
(See Persuasive Advertising, p.8)

Based on this exercise, write a small application step for yourself, and set a deadline, preferably within one week. If you are working with someone else, share your application plan and the results of your application. For example, there is much non-experimental evidence to suggest that investments in education are useful for countries. Do you have any experimental evidence to support that? What type of an experiment might be done? (If you have any interest in this example, you might want to read this paper on Natural Learning.)

Adapted from AdPrin.com