Firefly Communications White Paper

High definition
PR evaluation
A refreshing approach to measuring and
evaluating PR performance from Firefly
Communications.

Executive Summary
PR has always had an issue proving its worth. For years it has relied on the much-maligned
Advertising Value Equivalence (AVE) methodology, comparing the volume area of media
coverage to the equivalent value in paid-for advertising space.
Thankfully, with the advent of digital channels and the availability of free analytics tools such
as Google Analytics, we can say goodbye to AVE.
PR is no longer just about reputation management. It has a real business role to play. This
can be anything from contributing to sales or securing charity donations, to reducing the
strain on a call centre.
With this maturity comes a demand to move away from purely output-based metrics, such
as coverage volume, to looking at measuring business outcomes and impact.
Firefly advocates an evaluation model based on AMECs 7 Barcelona principles which puts an
emphasis on clearly defined goals that are business focused and take social media into
account.
Recognising that every organisations strategic goals are different, this method for
measuring and evaluating PR performance can also be fine-tuned to individual
organisational requirements.
With this in mind, Firefly has created an evaluation toolkit that can be scaled according to
scope and budget. It is based on four high definition streams of activity:

Business Goals - via Firefly Web Analytics

Media Impact - via Firefly Media Analytics

Social Media Effect - via Firefly Social Analytics

Search Contribution - via Firefly Search Analytics

Each part of Fireflys evaluation offering has foundation and advanced settings and is
suitable for business-to-business or consumer campaigns; SMEs and large corporates; as
well as business, charities and public sector organisations alike.

Contents
Executive Summary ................................................................................................................... 1
Why counting cuttings needs to be retired............................................................................... 3
1.

The attribution problem ................................................................................................ 4

2.

The business benefit of PR ............................................................................................ 4

3.

The social media conundrum ........................................................................................ 4

4.

Where PR and search collide ......................................................................................... 5

Firefly Communications principles of evaluation ..................................................................... 5

A focus on outcomes and business results................................................................................ 6
Setting objectives ...................................................................................................................... 6
Fireflys approach to evaluation: PR in high definition ............................................................. 7
1.

Business Goals ............................................................................................................... 7

2.

Media impact................................................................................................................. 8

3.

Social Media Effect ........................................................................................................ 9

4.

Search Contribution..................................................................................................... 10

Evaluation investment - how much should I spend? .............................................................. 11

What next? .............................................................................................................................. 11

Why counting cuttings needs to be retired

The world of PR has always had a problem with measurement and evaluation. Unlike other
marketing disciplines, such as advertising and direct mail, PR (at least defined in terms of
media relations) has been hard to track and measure.
We want to be able to measure brand awareness or reputation, but consumers and
business-buyers arent self-aware enough to understand the impact of PR. We resort to
monitoring things we can get a handle on, like media coverage, because inherently
communications objectives are virtually impossible to measure in a truly meaningful way.
A big issue is what can be referred to as the attribution problem i.e. what direct or
indirect effect has PR had on a business outcome?
Previously this was addressed through the (rightly) maligned AVE methodology. Namely
counting press cuttings and then calculating the advertising value equivalent i.e. what
it would cost if the space was bought as advertising media. This figure was then multiplied
three or more times to represent the assumption that editorial content is more credible and
powerful than paid-for advertising, due to the nature of third party endorsement.
Most savvy PR professionals had reservations about AVE years ago. Thankfully it is now
widely dismissed as being an inaccurate and often misleading way of evaluating success.
So what has replaced AVE as an evaluation tool?
For many agencies and in-house teams, we havent moved beyond a system of counting
cuttings. Some PRs add an extra layer to this by comparing competitor cuttings volumes to
create a share of voice figure, while others have created their own scoring methodologies,
which incorporate factors such as sentiment. One of the most well regarded examples of
this is Microsofts PRIME score, which is used across all of Microsofts markets.
Unfortunately, there is no industry-wide consensus on an effective scoring methodology
with any empirical foundation.
There are four main issues that need to be addressed:

1. The attribution problem - how can the direct and indirect impact of PR be singled
out as a contributing factor, particularly when compared to call to action activities
such as advertising?
2. A requirement for PR to move beyond just creating potential reach and visibility, but
having an observable impact on business outcomes such as sales.
3. The rise of social media organisations reputations need to be managed outside of
traditional media. Understanding where traditional media and social media intersect
and the relative dynamics is still a work in progress for the industry.
4. Can PR influence online visibility, particularly via search engines? If so, can it be
proved?

1. The attribution problem

PR consultant and friend of Firefly Andrew Smith likes to compare PRs attribution problem
to a football team. Imagine if the players were paid solely on how many goals they scored. If
only the scorer gets the credit, what about the assists from the other players who set up
the opportunity.
Under digital marketings last click attribution methodology, the scorer gets all the credit.
That hardly seems fair. However, its exactly what marketers do when they attribute success
to the marketing activity which last touched the customer.
In todays multi-channel world, its even harder to assess the impact of different marketing
activities, because the journey isnt linear. As McKinsey stated the funnel is dead instead
there is the case for considering a multi-faced customer decision journey.
As consumers and business service buyers, we are constantly exposed to a whole range of
channels, including display advertising, PPC, newsletters, video, social media, articles in the
press and so on (though as PR consultants, wed always argue the case that public relations
has the power to really influence customers).
The challenge is being able to establish the conversion flightpath and the role of PR in
determining an observable effect on the outcome.

2. The business benefit of PR

Business owners and CEOs are putting marketers under increasing pressure to ensure that
PR does more than promote the organisations reputation - it has to contribute towards
business objectives such as sales too.
This concept usually makes many PRs shift around awkwardly in their seats. PR doesnt
work that way, they say.
That doesnt wash any more.
PR has to visibly demonstrate its value. In a commercial setting, this usually means
contributing to sales in some way. And this doesnt necessarily mean a direct link - i.e. I saw
your piece in The Times and have decided to buy your widget. As outlined above, many
channels and touchpoints will contribute to any decision being made. Indeed it would be
rare for a single marketing channel on its own to directly sway a decision; the challenge is to
show both the direct and indirect contribution of PR. This is because PR is largely an
assistive medium.
Theres no one size fits all solution to this, but the use of web analytics is usually a part of
the evaluation methodology.

3. The social media conundrum

It is widely accepted that social media plays an influential role in an organisations
reputation. Whether its responding to criticism about the organisations practices via
Twitter or Facebook, or promoting content so that it is widely discussed and shared online,
organisations know that social media-based commentary is increasingly important to
understand and engage with.

There are a number of factors for PRs to consider when evaluating the impact on reputation.
These include the posts, tweets and shares on social networks themselves. How many are
there? Whos posting them? Are they influential? Where are their posts appearing? How
many people are seeing them? Whats the sentiment? What is the short, medium and long
term impact on reputation?
Then there is the impact and comparison with mainstream media. So, is the media coverage
driving social media conversation, or is it the other way around? Or a combination of the
two? What is more damaging, a negative tweet from an influential Twitter user or a negative
mention in a newspaper?
Until recently, the recording, measurement and evaluation of media and social media
coverage has happened in isolation. This has been partly a technical issue (service providers
simply havent been able to integrate these two) and partly an issue of measuring
readership and circulation in different ways.

4. Where PR and search collide

With every iteration of Googles search engine algorithm, PR seems to have an increasingly
important role to play. The SEO industry has had to deal with accusations of black hat
practices and respond to these algorithm changes by providing services that reflect Googles
ambition to serve up the most unique, relevant and timely results for a users search.
Googles recent Hummingbird, Panda and Penguin updates seek to improve user experience
in a number of ways for example, by making search results much more contextual,
penalising so-called thin or weak content websites and fighting copyright infringements.
These directly impact how a website is created, managed and promoted.
Were not saying a PR agency can replace an SEO agency (quite the opposite). But if the role
of PR is to provide quality content and a means of gaining bona fide high trust, high
authority inbound links, then PR is more important to SEO than ever.
Evaluating the impact of PR through media coverage alone misses this benefit out entirely.
While PRs cant (or shouldnt) promise links in media articles, if evaluation includes an SEO
element, then the campaign is more likely to be formulated with search in mind. For
example, this might mean creating supplementary content, hosted on the organisations
website, that a PR consultant can offer to the publication for linking.

Firefly Communications principles of evaluation

AMEC, the international association for measurement and evaluation of communications,
published a set of principles in 2010, which we believe should form the bedrock of any
organisations PR measurement and evaluation offering. There are seven Barcelona
Principles:
1.
2.
3.
4.
5.

Importance of Goal Setting and Measurement

Measuring the Effect on Outcomes is Preferred to Measuring Outputs
The Effect on Business Results Can and Should Be Measured Where Possible
Media Measurement Requires Quantity and Quality
AVEs are not the Value of Public Relations

6.
7.

Social Media Can and Should be Measured

Transparency and Replicability are Paramount to Sound Measurement

A focus on outcomes and business results

Public relations practitioners have tended to focus on output-based measurement and
evaluation counting cuttings being the most common approach. And theres nothing
wrong with counting cuttings per se (as long as the cuttings are in the right places and
include the right messages). But we advocate the AMEC approach and working towards
measuring outcomes and ultimately business results.
The diagram below demonstrates different aspects of the framework which can be
measured and evaluated. The questions remain: which ones are right, and how should they
be implemented?

Source: AMEC

Setting objectives
In line with the first Barcelona principle, the objective setting exercise is key. We understand
that communications objectives are inherently impossible to evaluate, but that doesnt
mean to say we cant work on meaningful metrics. These need to be discussed openly
between agency and client neither party is best placed to set them alone. We advocate
using the SMART model and define objectives that are measurable, achievable, relevant and
time-sensitive.

Fireflys approach to evaluation: PR in high definition

Firefly recognises that there isnt a one size fits all approach to evaluation. Just as every
organisations business goals can vary, so will its communications goals - and therefore the
method of evaluating its communications activities.
Fundamentally, evaluation of PR activities should be tied to answer the question: what is the
organisation trying to achieve? Once you know that, and have a PR strategy to meet those
goals, its a question of picking the appropriate tools to measure the impact of the strategy
and tactics.
To meet the need to offer a method of evaluation that suits all organisations and budgets,
Firefly has developed a menu-based approach. This is based on four streams of activity:
1. Business Goals - Firefly Web Analytics
2. Media Impact - Firefly Media Analytics
3. Social Media Effect - Firefly Social Analytics
4. Search Contribution - Firefly Search Analytics
To suit the different budgets and types of insight required, each of these evaluation streams
have two levels: foundation and advanced. These are explained below.

1. Business Goals
The Holy Grail for PRs has always been to answer the question, what effect does PR have
on my business? Its a false assumption to say that PR cannot contribute to business goals
(why bother otherwise?) Previously PR has arguably been more about visibility and
awareness and concentrated on measuring the volume of coverage, but ignored what
happens next. If a customer or prospect ventures on to an organisations website, PRs
havent been able to prove whether this was due to what they had been doing - either
directly or indirectly.
One way of evaluating the impact of PR on an organisations business goals is by looking at
the effect on website performance. This assumes, of course, that a website is vital to an
organisations business goals (which isnt true 100% of the time).
Google Analytics, the ubiquitous web analytics package, provides the ability to track much
more than traffic volume. In fact, to use Google Analytics just to monitor the number of hits
a site gets is, at best, just a small fraction of the picture and, at worst, can be extremely
misleading. Indeed, Google Analytics can track much more including offline data in diverse
areas from retail sales to interaction with exhibition stands.
Rather than just look at web traffic, Firefly works with organisations to set Goals that are
relevant to the business objectives. For example, this could mean sign-ups to a new service
or getting prospective customers to download a white paper. Or a combination of Goals.
Google Analytics becomes more powerful still when you start to create Campaigns that
enable web traffic to be attributed to particular activities and audiences for example, a
blog post announcing a new product, or distribution of an e-newsletter about the same
announcement. This is achieved through creating bespoke trackable URLs that can be tied to
individual activities.

One of the limitations of basic Google Analytics usage is looking at the time on a particular
page. How do you know if someone has read something? They may have simply had the
page open and been doing something else. This is where Event Tracking helps.
Lets consider some events that might be useful for marketers and PRs to track:

How many visitors to your site clicked on a promotional video?

Who requested a demo or trial?

How many people clicked on a Download this white paper button?

Who scrolled to the bottom of the page (suggesting that it has been read)?

Is the whole of an infographic being viewed?

Which pictures are being downloaded from the media centre and is this after seeing
the press release?

Google Analytics also provides powerful insight into who has visited and interacted with
your site. If you are targeting 18 - 24 year old men with an interest in sport, then getting one
million visits from 55+ women interested in tapestry is a Pyrrhic victory!
Google Analytics can also be used to work towards assessing business results. For example,
by using campaign URLs, you can see which activities generated sales. And to avoid the
attribution problem, Google Analytics has some attribution modelling tools to ensure that
its not the last click that gets all the credit.
Fireflys Foundation Analytics involves setting up and monitoring Google Analytics Goals and
Campaign URLs, all available on a bespoke dashboard, which can be shared via PDF. Well
run weekly reports to give you a good topline summary against pre-agreed performance
criteria.
Advanced Analytics provide a more in-depth analysis of website performance, tied closely
to business goals. In addition to the Foundation Analytics activities, it involves up-front site
analysis and, where appropriate, Event Tracking, Attribution Analysis and on-going support.
One-off reports can also be run to investigate areas of interest for example, looking at the
geography and demographics of users that responded to a particular campaign.

2. Media impact
There are two main reasons to evaluate media impact: firstly, to get a benchmark for share
of voice in the market; and secondly, to identify whether messages are getting through and
out to market. Insights into these two areas should be used to govern the next campaign,
rather than a basis for fee negotiation!
Firefly starts by working with clients to understand the audience(s) in detail, then to build a
highly targeted media list. Only media coverage in these publications should be evaluated.
Fireflys Foundation Media Analytics service meets the basic needs to record and analyse
coverage in key publications, setting benchmarks based on best industry practice. Here we
analyse performance against the media hit list, looking at areas such as readership, type and
sentiment. Coverage is shared via hyperlink or PDF, via our cuttings agency.

With Advanced Media Analytics Firefly offers a more comprehensive service, recording key
messages, sentiment, prominence (e.g. whether the company is featured in the headline or
there is a photo) and spokesperson performance.
A bespoke Advanced Media Performance service is also available to suit an organisations
individual requirements (e.g. tracking individual product lines or reputational drivers).
These services are available either on a single company basis or for competitor tracking.
With multi-company competitor tracking, Firefly only evaluates the coverage (rather than
capturing and sharing it), to keep costs down. Typically, companies will evaluate between
one and five competitors. Reports can be broken down by country or region (e.g. APAC,
Europe).

3. Social Media Effect

For years, social media has been monitored in a silo, with no integration with mainstream
media monitoring. The consensus was its just different. Thats true to an extent. Its
impossible to apply the same metrics to compare readership of an online article, for
example, and how many people have read a tweet.
However, PR practitioners already know that topics discussed in social channels are
inextricably linked to those being reported in the media. Therefore, reporting these
separately is short-sighted.
Social media also has relevance beyond the marketing and PR departments. For many
organisations, particularly those that are consumer facing, its a customer service issue too.
Overall, were looking to establish two things: the impact of your activities and the wider
conversation about your brand.
Utilising social media monitoring tools we can help to:

Identify which sources and commentators carry influence

Work out who is saying what

Identify opportunities for engagement

Positioned to respond quickly to negative comments

Track the impact on your profile and audiences

Measure the impact of your social media activity and how it relates to other PR and
marketing activities

Measure impressions, engagement and engagement rates for tweets and Facebook
updates

Compare sentiment within social media to mainstream media

Evaluate performance of communications and message penetration

Fireflys Foundation Social Analytics service use simple tools such as Klout (as a broad
indicator), Twitter Analytics and Hootsuite to establish social media impact in relation to PR
activities. View/Like scoring can also be added in, if relevant to the communications goals.

Advanced Social Analytics service uses industry-leading tools such as Precise, Radian6 and
Brandwatch to monitor social media activity across multiple channels, including sentiment
analysis and message penetration, identifying opportunities for engagement and issues
resolution.
We never advocate the wholesale use of automated sentiment checking computer
algorithms simply cannot understand all of the nuances of human language so advocate
human-based interpretation and analysis.
As part of Social Analytics, we are also able to integrate Social Media Effect with Media
Impact to provide the full picture.
Clearly, volumes vary between B2B and B2C communications and within industries, so this is
very much a case of picking the right tools for the job.

4. Search Contribution
PR and digital communications make a clear contribution to a brands online visibility,
whether this is through a search engine or via social media activities. Through Google
Analytics we are able to track the effect of campaigns as part of our Business Goals offering,
but knowing what effect activities have had on search results is also important.
To make the most of evaluating the search contribution of PR, we recommend fully
integrating public relations and search campaigns. This will ensure that the PR strategy is
working towards the same goals as the search strategy - for example, by targeting the same
audiences and having consistent keywords.
Fireflys Foundation Search Analytics looks at the number of inbound links that can be
attributed to PR. It provides a simple snapshot of how PR may be helping improve an
organisations search results against keywords.
Advanced Search Analytics is deployed when PR and search is fully integrated. Our search
partners at The Search Agency provide a complete diagnostics package, which includes an
on-site architectural report to identify key issues to resolve any on-going performance
problems. Like the other advanced evaluation toolkits here, we aim to tie the metrics to
business outcomes, rather than outputs.
Firefly is also able to monitor and evaluate performance against search results, working with
pre-defined keywords (also compared to a competitor shortlist). Search Analytics is always
best when combined with Google Analytics.

10

Evaluation investment - how much should I spend?

So, weve presented our options for evaluating PR. The next question on most peoples lips
is, how much does it cost?
Investment in evaluation can be a thorny issue. Often, particularly with smaller campaigns,
clients perceive high levels of spending on measurement and evaluation to be onerous - its
seen as being at the expense of campaign delivery.
Our answer to the question is to start by looking at key elements that relate to campaign
activities that are easily measured. For example, theres little point in measuring the number
of inbound links if there isnt any activity designed to increase their number.
While competitor tracking goes down well in the boardroom, its not always the most
important when it comes to using evaluation techniques to decide how to take the PR
campaign forward.
Our overall advice is concentrate on a few key KPIs that blend output, outcomes and ideally
business results. Spend no less than 10% of your budget on evaluation and use third party
data where possible (an alarming number of PR firms mark their own homework).

What next?
There is no silver bullet to solve the measurement and evaluation issues that have plagued
the PR industry for decades. However, with the advent of digital communications and
powerful and freely available analytics packages, we can get far more clarity about the
effectiveness of PR than in the past. Like all marketing disciplines, the attribution issue
hasnt been completely solved, but thankfully the evidence is growing about how PR can
contribute to a business bottom line, or a charity or government bodys organisational
goals.
If you like what youve heard from us, then why not get in touch for a free consultation to
understand how Firefly can help measure the impact of your PR and marketing
communications activities and how they may help you reach your strategic goals.

Please give us a squeeze at any of the following:

Email - hello@fireflycomms.com
Twitter - @Firefly_comms
Phone - +44 (0)203 861 3600

11