Fron%ers

of
Computa%onal Journalism
Columbia Journalism School

Week 13: Tracking Eects
December 12, 2014

Evolu%on of Media Eects Theory

1930s: "Hypodermic needle" or "magic bullet" theory. Whatever
is communicated is taken up wholly.

1950s: "Two-step" theory. Media inuences "opinion leaders"
who then inuence individuals through interpersonal contact.

1970s: Agenda-seSng. Media doesn't tell people what to think,
but what to think about.

2000s: Online data makes rst mass informa%on diusion
studies possible.

Media and vo%ng study, 1940 Elec%on

"One of the surprising results of The People's Choice study was
that the impact of media exposure to campaign messages was
rather negligible in terms of conversion changing voters from
an inten%on to vote for one candidate to an inten%on to vote for
the other. A more signicant eect of media exposure was
reinforcement assuring voters that their current inten%on was
worthwhile and correct."

-- Glenn G. Sparks, Media Eects Research, A Basic Overview

Two-step model: Decatur Study, 1955

Surveyed 800 women in Decatur,
Illinois, to determine how they made
purchasing choices (study funded by
an adver%ser.) Found personal
contact much more inuen%al than
media messages.

Solidied idea of "opinion leader"
who is inuenced by media, and then
inuences others personally.

Agenda-SeSng Theory
"In choosing and displaying news, editors, newsroom sta, and
broadcasters play an important part in shaping poli%cal reality.
Readers learn not only about a given issue, but also how much
importance to a_ach to that issue from the amount of
informa%on in a news story and its posi%on. In reec%ng what
candidates are saying during a campaign, the mass media may
well determine the important issues - that is, the media may set
the "agenda" of the campaign."

-- McCombs and Shaw, The Agenda-SeBng FuncDon of Mass
Media,1972

Agenda-SeSng Theory

Points out that the media is how we perceive all of the world
beyond our personal experience.

Argues that, therefore, media denes the issues that we think
about. Media produces salience.

1968 Chapel Hill elec%on study showed high correla%on (r>0.9)
between news content and surveys asking undecided voters
what their personal "issues" were.

Agenda-SeSng Research Methodology

(1968)
1) Ask 100 undecided voters:

"What are you most concerned about these days? That is, regardless of
what poli%cians say, what are the two or three main things which you
think the government should concentrate on doing something about?"

2) Content analysis of media sources.

"For the Chapel Hill community almost all the mass media poli%cal
informa%on was provided by the following sources: Durham Morning
Herald, Durham Sun, Raleigh News and Observer, Raleigh Times, New
York Times, Time, Newsweek, and NBC and CBS evening news
broadcasts."

The metrics newsrooms have tradi%onally used tended to be

fairly imprecise: Did a law change? Did the bad guy go to jail?
Were dangers revealed? Were lives saved? Or least signicant of
all, did it win an award?
But the math changes in the digital environment. We are awash
in metrics, and we have the ability to engage with readers at
scale in ways that would have been impossible (or impossibly
expensive) in an analog world.
The problem now is guring out which data to pay a_en%on to
and which to ignore. It is about seSng up frameworks for
tes%ng, analysis and interpreta%on that are both scalable and
replicable.

-- Aron Pilhofer (New York Times senior newsroom dev),
Finding The Right Metric For News

"Eect"

If journalism has an eect, it must change belief
or behavior.

In principle, this should be detectable.

Key idea: Output vs. Outcome

To measure eects, you must measure
something outside of your newsroom.

Coun%ng the number of stories published is
"output." Coun%ng the number of people who
believed or acted aier viewing the story is
"outcome."

Output

Newsroom

Stories

Government

Informa%on
and Services

Outcome

be_er life for user

Industry

Products

Why is measuring journalism

eects hard?
Widely varying expecta%ons for dierent
stories.
Dierent types of eects merely "informed"
vs. took ac%on
Many causes for change in user, impossible to
isolate journalism.
Long %me lags, indirect eects.

Dierent Types of Journalism

Entertainment
Simple informa%on
Buying decisions
Drawing a_en%on to wrongdoing
Explaining something complex

Voter Misinforma%on Survey, 2010

from MisinformaDon and the 2010 ElecDon: A Study of the US Electorate,

WorldPublicOpinion.org and Knowledge Networks

Voter Misinforma%on Survey, 2010

Ideas from other elds: academic publishing

Public Library of Science "ar%cle-level metrics"

Ideas from other elds: social science research

From Donovan and Hanney, The "Payback Framework" Explained

Mixed-method eects tracking

Build a database which tracks, for each story:

1. Response: all tweets, status updates, comments,

blog posts, links...
2. Claims of eect. Who said that this story
changed something, and what they said.
3. Story metadata. Publica%on date, author, URL,
category, etc.

AFAIK, no one has done this yet.