Angry by Choice: Confirmation Bias

8/9/12 4:03 PM

FS AA AbC AL AS BF C6 CC CH CO/VL CW DM EP GEP GW HG IF LA MDB MPM PEM PX PY R6 RRR SW TPP VM

Science Education Standards: Can
New Recommendations Help
'Mediocre To Awful' States?

Again, drug design and airplane

design are not the same

How to Unstick a
Gecko

Squirrels and
climate change

2 hours ago in The Curious Wavefunction

6 hours ago in inkfish

11 hours ago in The

Allotrope

1 hour ago in The Phytophactor

New ICS International

Chronostratigraphy Chart Uses the
Long Norian
18 hours ago in Chinleana

Angry by Choice
Discussions on the interface between Science and Society, Politics, Religion, Life, and whatever else I decide to write about.

Confirmation Bias
By The Lorax on Wednesday, July
07, 2010

Search
Recommend this on Google

In order to do good high quality and fun science, we general have a question
we want an answer to, which is more often referred to as a hypothesis. (Not all
good science is done this way, some is strictly and/or initially observational.)
Now when we ask a question, we may very well have a personal stake in the
answer.
An example may be warranted. Lets say its 1940 and youre a scientist
interested in the question does smoking cause lung cancer? Your hypothesis
is probably along the lines of I hypothesize that smoking leads to lung
cancer. If your dearest grandmother and father died of lung cancer, you may
already think the hypothesis is correct; if you happen to collect a paycheck
from Phillip-Morris and smoke yourself, you may think the hypothesis is wrong.
Now having a pre-established bias is not inherently problematic, its inherently
human. What you do with this bias is the issue. In the above example, it is
fairly clear what the biases are and why they exist and these biases are likely
to be identified by a scientist whos worth her salt. One of our jobs as good
scientists is to identify sources of bias. In the above example there are strong
personal sources of bias, but in the day-to-day workings of a research
laboratory there are more subtle biases. The I-think-this-is-a-cool-idea bias
(probably the most common), or the this-result-could-result-in-a-glamour-magpublication bias, or my-competitor-has-another-idea bias., the list goes on. So,
it is important to identify your biases. But this begs the question WHY?
One reason I want to address here, because it goes well beyond the
laboratory, is the issue of confirmation bias. Confirmation bias is what
happens when you conflate positive data (results that affirm your
preconceived notions) and/or diminish negative data (results that contradict
your preconceived notions). Confirmation bias helps the casinos make several
billion dollars in profit every year. Confirmation bias leads to the retraction of
some high profile publications every year. Confirmation bias makes pundits
and those that parrot them look like idiots.
So, using our smoking/lung cancer hypothesis above scientist #2 might use a
few 2 pack/day smokers without cancer as her sample population. Whereas
scientist #1 might recruit her study population in the cancer wards. Although
my examples are over the top, the choices made by a scientist that are a
product of confirmation bias could be much more subtle and not apparent to
other scientists who may be reviewing the work for a publication. This is why
good scientists are, or at least try to be, vigilant about their biases and take

http://angrybychoice.fieldofscience.com/2010/07/confirmation-bias.html

I am an Associate Professor at
the University of Minnesota with a
background in Biochemistry and
Molecular and Cellular Biology.
Oh, the opinions expressed here
are my personal viewpoints and
not those of my employer, family,
or dog.

Subscribe To AngryByChoice
Posts
Comments

Blog Roll
Traveling Small with a Nucleus
Field of Science
Dispatches from the Culture War
Evolving Thoughts
Why Evolution Is True
Mike the Mad Biologist
Parasite of the Day
Metamagician and the Hellfire Club
DrugMonkey
Tangled up in Blue Guy
Sandwalk
The Renaissance Mathematicus
Pharyngula
Rationally Speaking
Genomicron
Greg Laden's blog
Quiche Moraine
Blue Lab Coats
Digitalcuttlefish
The Hyphal Tip
Panda's Thumb
Inversesquare
Adventures in Ethics and Science

Page 1 of 6

Angry by Choice: Confirmation Bias

8/9/12 4:03 PM

steps to ensure they arent screwing up.

Say for instance you think there is a link between Chronic Fatigue Syndrome
and XMRV or that there is a link between the MMR vaccine and autism. Do
you find the one poorly designed study that supports your position while
simultaneously excluding the plethora of other studies that dont? If so, you
may be a confirmationally biased redneck.
This goes well beyond the sciency stuff. Do you go to Vegas and proclaim the
$250 jackpot you won, while failing to point out the 32 $20 bets you lost? The
casinos arent making billions in profits every year on paying out more than
they take in.
Do you have a problem with illegal aliens in Arizona as a recent threat to our
well being? Maybe you note whenever an illegal is arrested for a crime as
proof of a problem while ignoring fact that crime is lower* in Phoenix in 2009**
than in 2002**?
Do you think atheists have split into the old pleasant docile type and some
New type that is the root cause of most problems in America? Maybe youll
take the word of a sockpuppet identity thief because they agree with what you
already think over many real voices that disagree.
Hot today...global warming is true. Cold tomorrow...global warming is a myth.
*This is a 10 year comparison for Phoenix specifically.
**Compare pages 21 and 14 in the 2009 and 2002 years for a simplified
breakdown.

Reddit

Popular Posts
101 Dalmatians, Not an Animal Rights
Documentary
Look I realize that reality currently has a
liberal bias and that sucks for some of the
conservatives. Hell, four decades ago reality
had a ...
Amy Dickinson: Dumbass
Amy Dickinson: First ever awardee!! Grats
Amy! This week in the Star Tribune advice
column (it's next to the crossword so I often
r...
Why Students Cheat
As a member of an academic integrity
committee at my university I have learned a
lot of things. Some of these things I wish I
hadn't. Maybe ...
On scientific writing
In science, communication, both oral and
written, is a critical skill. I know the
stereotypical scientist is some extremely
awkward asocial ...
New blog time
For the couple of people that may (or may not) follow this
blog, I have a question. I enjoy writing about science and
education and other s...
2011 Cartoon Contest Contestants | Union of
Concerned Scientists
Check out the 12 entries to the UCS editorial
cartoon contest (2011 edition) AND VOTE. I'll
add my favorites later, but they are all
winners...

Labels: Logic, Scientific Method

Ultimate Death Match: Fungus vs Worm

As the newest penultimate addition to the
FoS collective (should any group of bloggers
ever use the term collective?), I thought I
sho...

4 comments:
Peter said...
Confirmation bias:- very interesting, logical comments. Unfortunate choice of example with
XMRV and CFS/ME. In the clinical community we are astounded that the many go no
further than count headlines. 1 for, 3 against. As if an outcome can be determined by some
sort of gut-feel vote. Never mind reconciling the differences.

Does size matter...or shape..Yes we're talking

about the penis
An older study brought to my attention
recently (subscription required, though I also
found this link ). This is strictly to avoid ...

A question for you:- What do you call it when public health officials direct the withdrawal of
a scientific paper, already accepted for publication because it goes against their intuition
[FDA XMRV paper by Harvey Alter] while allowing a study that agrees with their intuition to
be released [CDC Switzer study]? See Vincent Racianello's coments here
http://www.virology.ws/2010/06/30/publication-of-xmrv-papers-should-not-be-blocked/
Could it possibly be called "Confirmation Bias"?

Genetics and Poker and Sex Oh PZ!

PZs title pic I got from here The Minnesota
Atheists talk today by PZ Myers was a lot of
fun. Hopefully, he is making good progress
back...

What do you call it when a purportedly medical research paper finds patients for a serious
illness by phone survey rather than clinical diagnosis? Would this be accepted for say
cancer? [The CDC paper "found" patients by phone interview. In contrast with Lombardi
and Alter papers that accepted the work of their scientific peers for the last 25 years that
had established clear clinical diagnostic criteria]
Could this possibly be called "Confirmation Bias" - looking for something where you know it
will not be found? Perhaps that accusation is unfair. But why ignore the work of many biomedical researchers before you, as though your insight was "special". [CDC ignoring
Canadian consensus criteria and a large body of scientific papers establishing neurological
abnormalities in CFS/ME].

Santorum Is an Ass, a RANT (You Were

Warned).
So here is Santorum going after colleges and
universities last week: The question is 'Why?'
Has America reached this point, where we
cheer...

Followers

The scientific process needs defending. Unfortunately in this instance it is being left to
clinicians and patients to express that defence. Voices not generally accepted. Fortunately
a few more "acceptable" voices have stepped up to the plate.
Whether there is a link or not is not known. But please, a bit of anger at the interference to
the scientific process. Let science take it's course.

http://angrybychoice.fieldofscience.com/2010/07/confirmation-bias.html

Page 2 of 6

Angry by Choice: Confirmation Bias

8/9/12 4:03 PM

Join this site

Peter Wachtel,
Melbourne Australia

with Google Friend Connect

July 30, 2010 5:44 PM

Members (21)

The Lorax said...

Funny, I don't have a horse in the XMRV CFS pseudodebate and it was ~4 words of a ~650
word post. However, it prompted a ~300 word response from you.
While I understand the desire for quick fixes and definitive explanations for our ills, that is a
problem not a feature.

Already a member? Sign in

Personally, I agree that the paper submitted to PNAS should be published, unless there are
additional factors you are leaving out. The peer review process is flawed, but should be
respected.

What I am Reading

Regardless, you, Dr. Racianello, the WPI etc all have horses in this debate yet you only
point out the biases of those who disagree with you. One might call that poisoning the well.
I suggest you check out ERV's postings on XMRV and CFS (I know you already have), she
has some pretty damning technical problems with the XMRV CFS link data, which as far as
I can tell have not been adequately addressed.
August 3, 2010 9:06 PM

Peter said...
Thankyoufor taking the time to reply. My apologies if my expansion of a 4 word comment
into an essay seemed confrontational. I accept that my response was actually at a tangent
to your original discussion of human bias. And that you were making a broader point with a
few examples.
As to "horses", patients and clinicians do have a horse. But not the one you are suggesting.
eg., the technical issues ERV raised. They could be 100% correct. Or maybe not. I'm not
sure that I am entitled to care too much. "We" outsiders expect this technical issue to be
sorted out by the scientific process. We can only very generally judge the credibility of
those who speak. Listening to Ila Singh's discussion of techniques what I hear is a very
circumspect discussion of the detection difficulties by a hands-on, experienced researcher.
I hear that yes, actually, the technology that ERV derides as "1970's technology" is indeed
the most reliable way of detecting this virus. ERV's comment on this issue read like a
fundamentalist approach - "it is written". 21stC PCR amplification technology is the most
reliable, anything else is backward. I'm guessing that when "you guys" (the scientific
community) finish assessing this, the scientifically circumspect viewpoint of those like Ms
Singh will be the correct one. But maybe not. At least we'll have an answer.
The horse for patients/clinicians is actually whether this illness should be investigated from
a medical perspective or from a social/public health/ psychological viewpoint. The approx
2,500 peer-reviewed papers of the last 25 years that overwhelmingly support a medical
approach have not resulted in anything meaningful like clinical treatment trials. Why? We
have treatments for many diseases of unknown etiology. The xmrv "horse" just happens to
be one in the public eye that might influence policy.
Wanting quick fixes and explanations is a bias that should be questioned as you say. It is
also rather transparent and easy to discuss. Various groups have challenged the original
paper of the National Cancer Council, Cleveland Clinic and the WPI. And done so formally
in published commentaries. What I don't see is the other side of the coin.
One example:- the (inappropriate) name "CFS" was coined by the CDC in the 1980's to
classify outbreaks of a "flu from which no one recovered". In 2010 the CDC conducts a
study. The case definition it uses excludes those after whom it itself named the disease.
(Because of the presence of neurological abnormalities). What is going on? Why has the
CDC altered it's research definition so dramatically? Why is this definition so at odds with
clinical experience? Is this warranted or not? I do not see scientific commentary discussing
this.
Another: it is a matter of public record that the CDC was at the receiving end of
congressional investigations for mis-appropriating money earmarked for CFS research.
Money was allegedly diverted for "more worthy research". Many of the same people are
involved in the recent research. Is not the onus on the CDC to keep this issue in the open?
The potential confirmation bias of those producing the Sep 2009 paper has been formally
discussed. But none of those 3 labs have been charged with finds mis-appropriation.
Where are the questions to the CDC from the scientific community requesting affirmation
that past history has not biased the recent study?

http://angrybychoice.fieldofscience.com/2010/07/confirmation-bias.html

What Im Reading to My Son

Page 3 of 6

Angry by Choice: Confirmation Bias

8/9/12 4:03 PM

Anyway, enough of this. Thanks again for your previous response.

August 10, 2010 8:04 PM

Akshat Rathi said...

Damn right! Confirmation bias affects our everyday thinking in the lab. But even in if we are
able to over come it (which is very hard), it can play bigger roles in the process ahead. For
example, in our current peer-review system, if you want to get a paper selected you
suggest a few names which are people who do similar sort of work as you or who you think
will buy your arguments. When the paper arrives at these reviewers desk they will suffer
from a confirmation bias because the paper they are reading is helping them support their
own hypothesis. Thus, it might happen that the reviewer might overlook some details and
allow the publication of a paper which did not deserve to be published.

What Ive Read (2012)

Robot Visions by Isaac Asimov

August 22, 2010 5:23 PM

The Killing Moon by N.K. Jemison

Post a Comment

Men at Arms by Terry Pratchett

The Brothers Karamazov by Fyodor Dostoevsky

Links to this post

The Fellowship of the Ring by J.R.R. Tolkien

11/22/63 by Stephen King

Create a Link

God and the Folly of Faith by Victor J. Stenger

Newer Post

Home

Subscribe to: Post Comments (Atom)

Older Post

The Girl Who Kicked the Hornet's Nest by Stieg Larsson

The Girl Who Played with Fire by Stieg Larsson
Sourcery by Terry Pratchett
Theodore Rex by Edmund Morris
Interesting Times by Terry Pratchett

Favorite Youtube videos

Mr Deity 1-1
EC #1

Categories
??? (4)

Blog Archive

Animals (11)
Atheist (28)
Bananaman (4)
Beer (1)
Book Club (2)
Book Review (5)
Books (7)

! 2011 (63)
" 2010 (40)
! December (1)
! November (7)

Communication (28)

! October (1)

Creationists (4)

! September (1)

Critical Thinking (1)

! August (3)

Debate (15)

" July (8)

Dumbass (3)
Education (7)
Eukaryotic Microbiology
(8)
Evolution (28)
Fun (37)
Grants (1)
journal club (6)
LabRat Rocks (1)

http://angrybychoice.fieldofscience.com/2010/07/confirmation-bias.html

! 2012 (24)

Why we pursue the

Truth
Educating at the High
School-College
Interface
Poop Transplants, Is
Your Microbiota an
Organ Syst...
HIV: Basic Research
IS the Force behind

Page 4 of 6

Angry by Choice: Confirmation Bias

8/9/12 4:03 PM

Translatio...

Logic (7)
Music (8)

RC revisited

Mycology (6)
NIH (1)

Grants and Rants and

the Lotto

Offensive BS (18)

Confirmation Bias

Opinionated Bastard (11)

Sometimes things
come out
right...mostly

Origin of Species Book

Club (4)
Peer Reviewed Research
Blogging (19)

! June (2)

Peronsal (1)

! May (1)

Politics (59)

! April (3)

Questions (2)

! March (4)

Quiz (7)

! February (3)

RANT (2)

! January (6)

Religion (46)

! 2009 (52)

Religion poisons
everything (1)

! 2008 (57)

Science (88)

! 2007 (21)

Science Policy (5)

Scientific Method (19)
Scientists (16)
Teaching (32)
Who we are (3)
writing (3)
YAY (3)

Followers (2)

Follow this blog

Field of Science

Latest From The Network...

Field of Science
Blogs:
A is for Aspirin
Angry by Choice
C6-H12-O6
Catalogue of Organisms
Chinleana
Doc Madhattan
Epiphenom
Games with Words
Genomics, Evolution, and Pseudoscience
History of Geology
Inkfish
Labs
Memoirs of a Defective Brain
Moss Plants and More
Pleiotropy
Plektix

Powered by Y! Pipes

Protein Evolution and Other Musings

http://angrybychoice.fieldofscience.com/2010/07/confirmation-bias.html

Page 5 of 6

Angry by Choice: Confirmation Bias

RRResearch
Rule of 6ix
Skeptic Wonder

8/9/12 4:03 PM

about | contact | archives & alumni

subscribe: feed - email
follow: twitter - facebook - badge

The Allotrope
The Astronomist
The Biology Files
The Culture of Chemistry
The Curious Wavefunction
The Phytophactor
The View from a Microbiologist
Variety of Life

http://angrybychoice.fieldofscience.com/2010/07/confirmation-bias.html

Page 6 of 6