IE 332 Book Report S.

Kami Üçkardeş – Dilan Derin

BOOK REPORT
Superforecasting: The Art and Science of Prediction
Authors: Philip E. Tetlock and Dan Gardner

INTRODUCTION:

Superforecasting The Art & Science of Prediction, is a book which teach us how
to be a good forecaster with real life examples, and approaches the events from a
different (but interesting) perspective. Each chapter describes different features’ of
superforecasters with striking examples and astonishes the reader. In first chapter the
author mentions about predictability and unpredictability. Also tells us there is no (weak)
relationship between intelligence, numeracy knowledge and being a good forecaster. The
point is how to think. In second chapter, we get information about our thinking system (1
& 2). And teaches us doubts’ importance in prediction. Chapter 3, the author talks about
importance of usage numeric probability rather than maybe, likely etc, clearly defined
terms and timelines. If we want to be a good forecaster we must satisfy these basic
requirements. In fourth chapter we see how luck and skill play role in forecasting and
observe that how an ordinary people beat experts in very serious game. In Chapter 5, we
sight, how to approach questions, and a path: Fermi-Ize-> Outside View-> Inside View-
>Synthesis. In sixth chapter Philip Tetlock mentions about 2 different ways to approach
the events: Fate and Probabilistic thinking. And shows the relationship between thinking
way and forecasting accuracy. In seventh chapter, updating the prediction is emphasized.
A superforecaster should update their prediction frequently and for not going wrong a
way, updating should be done bit by bit. In chapter eight, they emphasize that forecasts
should always be in perpetual beta. A superforecaster should always analysis and
improve their forecast. In chapter nine, they talk about the advantages and disadvantages
of a team for forecasters. In chapter ten, they talk about what should be the qualities of a
leader in the team. In chapter eleven, they analyzed how biases affects the forecasts. In
chapter twelve, they look at how human qualities of a forecaster can affect the forecast.

Chapter 1: An Optimistic Skeptic

A researcher (author) gathered a group of experts -academicians, pundits etc- to

make thousands of predictions about many areas -economics, wars etc.- Time passed and
checked the accuracy of predictions; and found that the average expert did as well as
random guessing. And called them dart-throwing chimpanzees. Nowadays every people
call themselves forecaster and these guys are everywhere; on Tv’s, newspapers’, radios’
etc. “There is one undeniable talent that these talking heads have is their skill at telling a
compelling story with conviction.” The author also realized that the actual message of
research was garbled in the constant retelling and the subtleties were lost entirely; the
message becomes “All expert forecasters are useless”. Now there are 2 different
IE 332 Book Report S. Kami Üçkardeş – Dilan Derin

approaches to events (forecasting); Debunkers; who are called forecasting as a fools’

errand and the second one defenders of experts and their forecasts. The author thinks that
it is important to stake out a position between these two; the future is predictable at least
in some situations and open-minded, hardworking person can cultivate the requisite
skills.
The author called himself “optimistic skeptic” in the previous part. In this chapter
he explained the reason. The things that no one else considers today have unexpected
consequences. (Arabian Spring) This is a big reason for the “skeptic” half of my
“optimistic skeptic” stance.”. We live in a world where the actions of one nearly
powerless man can have ripple effects around the world.
More often forecasts are done and then …. nothing. They have no idea how good
their forecasts are in the short, medium, or long-term—and no idea how good their
forecasts could become. That’s because the forecast-measure-revise procedure operates
only within the rarefied confines of high-tech forecasting. Forecast, measure, and revise.
Repeat. It’s a never-ending process of incremental improvement that explains why
weather forecasts are good and slowly getting better. They make large numbers of
forecasts and routinely check their accuracy. With these analyses, meteorologists can
sharpen their understanding of how weather works and tweak their models. Bill Gates
also has a quote about this “You can achieve incredible progress if you set a clear goal
and find a measure that will drive progress toward that goal...” There is a stagnation;
forecasting’s’ only goal is not to foresee the future it is also meant to entertain,
sometimes forecasts are used to advance political agendas etc. Which makes it difficult to
even start working toward measurement and progress. This is the reason why author
called himself “optimistic”. Forecasting is in every area of our lives. The only thing we
should do is setting a clear goal and get serious about measuring. You don’t need to be
an expert in any field to make good forecasts. Superforecasting requires minimum levels
of intelligence, numeracy, and knowledge of the world. The point is how to think.
Superforecasting demands thinking that is open-minded, careful and self-critical. Another
requirement of being a superforecaster is able to blend the computer-based forecasting
and subjective judgment.

Chapter 2: Illusion of Knowledge

Another way to become superforecaster is to approach the events with suspicion.

In 1956 Archie Cochrane (Physician) went to doctor to show his armpit. The specialist
discovered a lump in the Archie’s’ armpit, and said we had to take it out, it was a basal
cell carcinoma. The surgery was booked. After surgery doctor said, “I have done my best,
but I may not have saved your life” before waiting pathologists’ report. (He died at 1988)
That’s the problem Cochrane did not doubt the specialist and specialist did not doubt his
own judgment. The absence of doubt that made medicine unscientific and caused it to
stagnate for so long. Consider Galen, the second-century physician, he said: “All who
drink of this treatment recover in a short time, except those whom it does not help, who
all die.” Which was obvious, therefore, that it fails only in incurable cases. People were
doing just the thing without analyzing. So, “cargo cult” science has the outward form of
IE 332 Book Report S. Kami Üçkardeş – Dilan Derin

science but the lacks what makes it truly scientific. Feynman (Physician) observed,
“Doubt is not a fearful thing but a thing of very great value”.
In describing how we think and decide, modern psychologists often deploy a
dual-system model; system 1 and system 2. The numbering of the two systems is not
arbitrary. System 1 comes first. It is fast and constantly running in the background.
“System 1 is designed to jump conclusion from little evidence” which Kahneman said.
Kahneman also gave a memorable label to System 1: WYSIATI (What You See Is All
There Is). On the other hand, System 2 is charged with interrogating the System 1’s
answer. Another mistake which is done by forecasters is trying to solve a question which
requests system 2 with system 1. Formally, it is called attribute substation. The message
is not “System 2 is better than System 1”. System 1 is fast, automatic and effortless.
Makes it very good at pattern matching. The question is when should we trust it?

Chapter 3: Keeping Score

Consider a forecast Steve Ballmer made in 2007 when he was CEO of Microsoft:
“There’s no chance that the iPhone is going to get any significant market share. No
chance.”. This prediction was noted “The Ten Worst Tech Predictions of All Time” in
2013. But, how can we say that Steve Ballmer’s’ forecast was wrong? If we parse
forecast carefully the key term was there is “significant market share”. What qualifies
significant? And which market was Ballmer talking about? And market for what? There
were too many unanswered questions. Ballmer’s’ forecast is ambiguous. It looks wrong,
it feels wrong. But is it wrong beyond all reasonable doubt? The author says, “Judging
forecasts is much harder than often supposed, a lesson I learned the hard way—from
extensive and exasperating experience.”
Forecast without a time frame is absurd. This problem alone renders many
everyday forecasts untestable. Similarly, forecasts often rely on implicit understandings
of key terms rather than explicit definitions—like “significant market share” in Steve
Ballmer’s forecast. This sort of vague verbiage is more the rule than the exception. And it
too renders forecasts untestable. Forecasters use words like maybe, probably, likely, fair
chance etc. during making forecast, instead of numeric probabilities. These also make
prediction untestable. Sherman Kent (Academician) has legendary quote about this
situation: “I’d rather be a bookie than a goddamn poet”.
Shortly, we need to have forecasting questions with clearly defined terms and
timelines. We need to have lots of forecasts with numbers, and the math to calculate
scores (Brier). We must have squeezed out as much ambiguity as appears humanly
possible.

Chapter 4: Superforecasters

In October 2002 Intelligence Community(IC) reported that Iraq had continued its
weapons of mass destruction(WMD) programme and the threat was growing. The US
invaded Iraq in 2003 after this analysis. United States turned Iraq upside down looking
for WMD but found nothing. Which was one of the worst intelligence failure in modern
history. In 2006 the Intelligence Advanced Research Projects Activity (IARPA) was
IE 332 Book Report S. Kami Üçkardeş – Dilan Derin

created. to make the intelligence community smarter and more effective. They started
working on a project which was called Good Judgment Project (GJP). In the first year,
several thousand volunteers signed up, after first test this number decreased to 3200 and
these 3200-people started forecasting. Time passed. And the result was shocking. A few
hundred-ordinary people and some simple math can not only compete with professionals
supported by a multibillion-dollar apparatus but also beat them.
Most things in life involve skill and luck, in varying proportions. Regression to
the mean is an indispensable tool for testing the role of luck in performance: Mauboussin
(Managing director and head of Global Financial Strategies at Credit Suisse) notes that
slow regression is consistent with activities dominated by skill, while faster regression is
more associated with chance. Hopefully, we can conclude that the superforecasters were
not just lucky. Mostly, their results reflected skill.

Chapter 5: Supersmart?

Sandford Sillman was one of the greatest -actually the greatest- superforecaster
with a Brier score of 0,19. (In GJP). Sandy has a bachelor of arts degree, with a double
major in math and physics, from Brown University, plus a master of science degree from
MIT’s technology and policy program, along with a second master’s degree, in applied
mathematics, from Harvard—and a Ph.D. in applied physics from Harvard. What a mind
it is. Can we conclude that superforecasters are better simply because they are more
knowledgeable and intelligent than others? According to tests results; Regular forecasters
scored higher on intelligence and knowledge tests than about 70% of the total population.
This number was 80% for superforecasters. As we can see there is no big difference
between those numbers. So, intelligence and knowledge help but they add little beyond a
certain threshold.
How to approach questions? Common path is as follows: Fermi-Ize-> Outside
View-> Inside View->Synthesis. If you do not have any idea about the question, divide
the problem to sub-problems. Fermi-Ize; This method makes forecasting easier because it
reduces the number of random guesses and prevents bait and switch (If we face with a
difficult problem we automatically switch that with an easier one) which changes the
structure of the problem. Outside view; Before focusing on details of the problem, look at
the big picture. (If question asks a number of pianos in Roseland, first look the number of
pianos in Chicago etc.) If we start forecasting with outside view, we will begin with an
anchor (the number we start forecasting) which will be meaningful. And a better anchor
is a distinct advantage. Inside view; Extremely dangerous step of forecasting, if you
aimlessly examine one tree, then another, and another, you will quickly become lost in
the forest. It should be targeted and purposeful: it is an investigation, not an amble.
Firstly, we must construct a hypothesis, then start looking for evidence pro and con. This
looks like a detective work because it is methodical, slow and demanding. But it works
far better than wandering aimlessly in a forest. Another condition of being a successful
forecaster is to be able to approach events from different perspectives. How to obtain new
perspectives? Method 1: Wisdom of crowd (what other forecasters think about your
approach, what is their starting point). Method 2: Tweak wording (Imagine a question
like “Will the South African government grant the Dalai Lama a visa within six months?
IE 332 Book Report S. Kami Üçkardeş – Dilan Derin

“and replace grant with deny.) With this method, you can get opposite sides’ thought,
perspective.
For superforecasters, beliefs are the hypothesis to be tested, not treasures to be
guarded.

Chapter 6: Superquants?

Human beings have coped with uncertainty for as long as we have been
recognizably human. Both 0% and 100% weigh far more heavily in our mind than other
percentages. Because of this we always ignore small chances and use the two-setting dial
(Either true or false) as much as possible. This is the proof that why people who are so
sure about themselves are more popular on media. (Tv. Etc.)-In reality, they have no
ability except storytelling, talking heads-. Most people would identify science with
certainty. But that was a very nineteenth-century view of science. One of twentieth-
century science’s great accomplishments has been to show that uncertainty is an
ineradicable element of reality. “Uncertainty is real,” Byers (Canadian mathematician &
philosopher) writes. If nothing is certain, it follows that two settings mental dials totally
useless. The only setting that remains is “maybe” (The third one)-the one setting people
intuitively want to avoid-.
Let’s examine the relationship of forecasters with “certainty”. The average
forecaster who sticks with tens is less accurate than who uses five and still less accurate
than who uses one. Most people never attempt to be more precise. Which is a serious
mistake. As the legendary investor Charlie Munger sagely observed, “If you don’t get
this elementary, but mildly unnatural, mathematics of elementary probability into your
repertoire, then you go through a long life like a one-legged man in an ass-kicking
contest.”
We can approach the events we encounter in life in 2 different ways: Probabilistic
thinking and meant-to-happen thinking (fate). Experiments have shown that fate thinking
rate goes down from adult Americans to undergrad students and from undergrad students
to forecasters. There is also a relationship between thinking way and forecasting
accuracy. As the fate thinking rate decreases the accuracy increases.

Chapter 7: Supernewsjunkies?

The other difference of superforecasters from regular forecasters is ability to

update their predictions when new information comes up. A forecast is based on the
available information but when this information changes, the forecast should be changed
for improving accuracy. Superforecasters should not stay behind of these changes, they
need to monitor changes even daily if necessary.
When a new information comes, a superforecaster should be cautious about not
underreacting and not overreacting. People tend to underreacting and the reason of it
mostly because of the “belief perseverance”; if the new information upset their belief,
they can be really stubborn about not accepting the new information. People’s
overreacting can be because of the “dilution effect”; the new information can be
IE 332 Book Report S. Kami Üçkardeş – Dilan Derin

irrelevant for our prediction and this can weaken our confidence. Solution for both
underreacting and overreacting is commitment and finding middle passage.
A superforecaster should update their prediction little by little. But like Bayes
theorem says, new prediction should be based on both past prediction and the new
information.

Chapter 8: Perpetual Beta

The other difference of superforecasters is the ability of having a “growth

mindset”; they know that they can improve themselves with their abilities and hard
working. Failure is not the end for them, they are always in a cycle: try, fail, analyze,
adjust and try again.
Making forecasts is essential for learning forecasting like any other things that we
want to learn. Books guide us for getting right lessons from our experience but getting the
right feedback is really essential for it. Unfortunately, getting an effective feedback is
hard for a forecaster. An ambiguous language can make the prediction hard to judge.
Getting feedback later, because of the circumstances, also is not good. You’ll not enough
time to fix your mistakes or you may forget details of your prediction. These will lead
you not to get right lessons from the feedback. A superforecaster should be determined to
keep the cycle as long as it takes. Forecasts should always be in perpetual beta.
Forecasters should always analysis and improve their forecast.

Chapter 9: Superteams

Building team from forecasters can be really effective. People tend to make
forecast with discussing other people even when they are not work in a team. Discussing
with other people can sharpen judgement and you can do things that you cannot do alone.
In teams, people share information between them; this lead them see other perspectives
for the issue. It is important for improving the accuracy.
But the danger is they can create illusions when they encounter with an
uncomfortable situation like in the Bay of Pigs disaster. Teams with collective
unconsciousness, tend to agree with each other. Since everyone agrees, they assume they
are in the right path. But if every team member is conscious about this and questioning
themselves if they’re in this delusion, the danger will be minimized. When
superforecasters encounter with an agreement, they question themselves and search for
the reasons why it is accurate. Tetlock’s research shows that in one year, teams were 23%
more accurate than individual forecasters. Individuals who are educated one year for
becoming a forecaster are placed in a team and after one year their accuracy is increased
by 50%.

Chapter 10: The Leader’s Dilemma

IE 332 Book Report S. Kami Üçkardeş – Dilan Derin

Leaders have a crucial role in a team for avoiding the dangers of being a team.
They must lead the team because of that they need to be confident, decisive and have a
vision. But forecasting is fill with uncertainties. How can a leader lead the team in this
ocean of uncertainties?
In the 19th century, the Prussian army used uncertainty as a way to understand the
reality. All the circumstances can change very fast and because of that soldiers must
adapt these changing circumstances even without the orders of the commanders. They
need to be flexible. The commanders told the soldiers the goals, not step by step
instructions to achieve this goal. They communicated effectively and encouraged the
soldiers making decisions without waiting for the orders. The leader of the
superforecasters should also not give orders but the information, goals; leader should give
new perspectives. Uncertainty is important, superforecasters should doubt the ideas from
themselves and their teams’ but the uncertainty shouldn’t be about their own abilities.

Chapter 11: Are They Really So Super?

The superforecasters are good because they work hard for their prediction, they
value the details and think every important aspect carefully. They use System 2 to catch
the mistakes that are from the System 1. But sustaining all this work for a long time is
hard even for a superforecaster. They are humans after all, there’ll be a moment where
they choose their biases for choosing the easier method. Like in Müller-Lyer Illusion;
even if you check with a ruler to see that the lines are identical, without the ruler again
you’ll see them in different lengths. But minimizing and resisting the biases is possible.
“Nassim Taleb’s Black Swan”: Let’s assume that we are in Europe four centuries
ago where all the swans are white. When a person asks you that think about a weird
swan, you’ll think an ugly swan whether a large size or different shape. But you won’t
think a swan with a different color, all the swans you think will be white. When an
Australian boat comes with a black swan, you’ll be stunned. Yes, some situations can be
unpredictable but consequences can be, at least you’ll be prepared for such situations.

Chapter 12: What’s Next?

Sometimes forecasters put ahead of their interests while they are forecasting. It
happens because as a human we tend to do that. Maybe you’re in politics and don’t want
to admit a truth or maybe you do this unconsciously. But for your forecast, accuracy will
fall inevitably. For controlling these obstacles, some countries introduced evidence-based
policies. Good forecasting effects many aspects of life. It can help us to see if an event is
success or failure. Even we can predict there will be a war or not. Accurate forecasting
will always be wanted.
For a superforecaster, asking the right questions is also really important. But there
is not one big question to answer. For finding accurate answer, one should ask many
questions. With enough information and considering that there are no certainties in life,
one can become a superforecaster.
IE 332 Book Report S. Kami Üçkardeş – Dilan Derin

TAKE-AWAYS:

Consciously or subconsciously, even you’re not a forecaster, we all encounter

with a forecast in our everyday life. As authors say in the beginning “We are all
forecasters”. This book is not only for people who is a forecaster, even if you don’t know
anything about forecasting you can read this book. It is an informative and entertaining
book. They went to many details for forecasting, they told many interesting stories. They
even looked at worst forecasts in the history. It is also a guide for good forecasting and
you can use it in your life even if you’re not expert. It gave us a new perspectives that
actually experts in the television are actually story tellers rather than good forecasters.

*To be a skeptic and do not forget that that even small things lead great changes. As we
see in first chapter a Tunisian paddler changed the system of whole Arabic peninsula. We
need to think too much even in making small decisions.
*We have to measure our forecast after making it, and repeating following cycle:
Forecast, measure, revise. Without measure and revise forecasting means nothing.
Weather forecasters are good and getting better because they apply this cycle perfectly.
One of the important conditions of being a good forecaster is measuring and revising
forecast. We have to do it if we want to be one.
*We should not accept anything as it is in life (cargo cult) and try to understand it. We
must make the suspicion a part of our lives and approach the events with it. Do not forget
the Feynman’s’ speech, “Doubt is not a fearful thing but a thing of very great value”.
*If we want to make our prediction testable and understandable by everyone, we have to
make it with clearly defined terms and timelines, and also numeric probabilities than
maybe, likely etc. Because as Kent said “I’d rather be bookie than a goddamn poet.”
*IARPA’s Good Judgment Project proves that, there is really weak relation between
intelligence, numeracy knowledge and being a good forecaster. And shows that ordinary
people beats experts in a very serious game. The point is how to think.
*If we do not have any idea about a subject or a question, we can follow this path: Fermi-
Ize-> Outside View-> Inside View->Synthesis, and come up with an idea or an answer.
The first thing we have to do is looking at the big picture instead of a details of the
problem. If we do this we will begin with a good anchor. And a better anchor is distinct
advantage.
*If we want to be a good forecaster we have to know that, our beliefs are hypothesis to be
tested, not treasures to be guarded. In daily life we have to open to different opinions and
take them into account. The most important way to succeed is to be open to innovations
and different opinions.
* A good forecaster should update their prediction based on a new information that
comes. But he or she should be careful about not overreacting or underreacting the
information.