How reliable are the consumers?

Comparison of sensory profiles

from consumers and experts
WORCH Thierry(1)
L Sbastien((2))
PUNTER Pieter(1)

(1)

OPP Product Research

(2) AgroCampus
A C
O
Ouestt

Project 8013
July 2008

mailto: thierry@opp.nl

Senior project manager Pieter Punter

Project manager Thierry Worch

introduction
in the sensory theory:
experts
p
p
panels are used for the p
products description
p
consumers should only be used for the hedonic task
they lack two essentials qualities for profiling (consensus
and reproducibility)
there are strong halo effects (Earthy, MacFie & Hedderley,
1997)
in the sensory practice:
consumers are sometimes used for both tasks
it has been proven that consumers
consumers description show the
required qualities (consensus and reproducibility) (Husson, Le
Dien, Pags, 2001)

8013

problematic

How reliable are the consumers?

8013

presentation of the studies

products:
twelve luxurious women perfumes
(Gazano, Ballay, Eladan & Sieffermann, 2005)

8013

Angell
A
(Eau de Parfum)

LInstant
LI
t t
(Eau de Parfum)

Cinma
(Eau de Parfum)

JAdore
(Eau de Toilette)

Pleasures
(Eau de Parfum)

JAdore
(Eau de Parfum)

Aromatics Elixir
(Eau de Parfum)

Pure Poison
(Eau de Parfum)

Lolita Lempicka
(Eau de Parfum)

Shalimar
(Eau de Toilette)

Chanel N5
N5
(Eau de Parfum)

Coco Mademoiselle
(Eau de Parfum)
5

presentation of the studies

expert panel (Agrocampus Rennes)
twelve persons (11 students and 1 teacher) from the Chantal Le
Cozic school (esthetics and cosmetic school)
focus group per group of six, with two animators
generation of a list of twelve attributes
Vanille, Notes Florales, Agrume, Bois, Vert, Epic, Capiteux,
Fruit, Fracheur Marine, Gourmand, Oriental, Enveloppant

training session for the most difficult ones

the twelve products were tested two times in two one-hour
sessions

8013

presentation of the studies

consumer panel (OP&P Product Research, Utrecht)
103 nave Dutch consumers living in the Utrecht area

the same twelve perfumes were rated on 21 attributes

odour intensity, freshness, jasmine, rose, camomile, fresh lemon,
vanilla, mandarin/orange, anis, sweet fruit/melon, honey, caramel,
spicy
spicy , woody
woody , leather
leather , nutty/almond
nutty/almond , musk
musk , animal
animal , earthy
earthy , incense
incense ,
green

two products (Shalimar and Pure Poison) were duplicated

the fourteen (12+2) products were tasted in two one-hour sessions (seven
products in each session, presentation order was balanced)

8013

presentation route map

the consumer and expert data are compared in three different ways
1.Univariate analysis
analyses of variance
correlations
2.Multivariate
2
Multivariate comparison
construction of the two products spaces (PCA)
comparison of the products
products spaces through GPA and MFA
3.Confidence ellipses
graphical confidence intervals around the products averaged
over the two panels
graphical confidence intervals around the products defined
by the different panels
8013

Performance of the two panels

(univariate analysis)

8013

performance of the panels

usually, the expert panels should have many qualities:
discrimination: panelists should be able to detect and describe
the differences existing between the products
reproducibility: panelists should describe the products in the
same way, when they are repeated
agreement: panelists should give the same description of the
products as the rest off the panel
it can be measured with the correlations (usually, one
panelist
pa
e s is
s co
compared
pa ed to
o the
e mean
ea o
over
e the
e rest
es o
of the
e
panel)

8013

10

expert panel
panel performance
discriminate on 11 out of 12 attributes (Agrume, pvalue=0.08)
reproducible for 11 out of 12 attributes (Notes Florales)

panellist performance (discrimination,

(discrimination reproducibility)
panellists 1, 3 and 12 are very good
panellists 8, 9 and 10 are not good in discrimination
(discriminate the products on less than 6 out of 12 attributes)
panellist
lli 9 iis also
l not good
d iin reproducibility
d ibili ((reproducible
d ibl on
only 3 out of 12 attributes. Notes Florales, Agrume and
Enveloppant)
8013

11

expert panel (correlations)

distribution of the correlations (correlation between expert i and the
mean over the ((n-1)) others))

8013

12

consumer panel
discrimination (on the twelve original products)
the consumers discriminate the products on all attributes except
camomile (pvalue = 0.62)
NB: the consumers discriminate on Citrus (pvalue < 0.001)

reproducibility (on the two duplicated products only)

consumers are reproducible on all attributes except one
(woody)

8013

13

consumer panel (reproducibility)

Shalimar
Shalimar 2

8013

14

consumer panel (correlations)

distribution of the correlations (correlation between a consumer i and
the mean over the ((n-1)) others))

8013

15

conclusions on the panel performance

expert panel
discriminates between the products
are reproducible
high correlations

consumer panel
discriminates between the products
shows reproducibility
reproducibilitys
s qualities
lower but still positive correlations (consumers are untrained)

Both panels show the same qualities

8013

16

Products spaces
(multivariate analysis)

8013

17

methodology
products spaces
the products profiles (averaged over the panellists or consumers)
are computed.
Principal Components Analysis is then run on these product x
attribute matrices
comparison of the two products
products spaces (expert and consumer) is a
multi-table problem
comparison through the Procrustean analysis
comparison through Multiple Factor Analysis (MFA)
(
)
comparison through the confidence ellipses technique

8013

18

expert panel
g p
3,5
3,0

AromaticsElixir

25
2,5

Shalimar

2,0
1,5

Chaneln5
1

JAdore_EP
0,5

PurePoison

JAdore_ET

0,0

CocoMelle

-0,5
-1,0

Bois
Epic

LInstant
Notes.florales

-1,5

Cinema

Dimension 2 (21.87 %)

Dimen
nsion 2 (21.87 %
%)

Pleasures
1,0

Angel

-2,0
-2,5
-3,0

LolitaLempicka

-3,5
-4,0

Oriental

Vert

Capiteux

Agrume
Fraicheur.marine
0

Enveloppant

Fruit
-4,5
-4

-3

-2

-1

Dimension 1 (64.22 %)
Vanille

Gourmand

-1
-1

Dimension 1 (64.22 %)

8013

19

consumer panel
6,0
5,5
5,0
4,5
4,0

LolitaLempicka

3,5

3,0
2,5

Angel

Cinema

LInstant

vanilla

15
1,5

honey

1,0
0,5

JAdore_EP

caramel

0,0
-0,5 JAdore_ET

camomille

Pleasures

-1,0

anis

CocoMelle
PurePoison

citrus

Chaneln5

-1,5

Shalimar

Dimension
n 2 (17.97 %)

Dimen
nsion 2 (17.97 %
%)

2,0

-2,0
-2,5
-3,0
-3,5
-4,0

AromaticsElixir

-4,5
-5,0
-5,5

nutty

sweet_fruit

freshness
green
jasmin
rose

animal
musk
incense
leather
woody

fresh_lemon

earthy

-6,0
-6,5
-7,0
70

i t
intensity
it
-5

-2.5

2.5

spicy
i

7.5

Dimension 1 (68.29 %)

-1
1
-1

Dimension 1 (68.29 %)

8013

20

Multivariate comparison of the

two panels (GPA and MFA)

8013

21

expert vs consumer: Procrustes analysis

0.4

GPA consensus space

0.3

(coefficient of similarity: 0.93)

A
Angel
l

0.1

Cinema
LInstant

JAdore_ET CocoMelle
JAdore_EP
PurePoison
Pleasures

-0.1

0.0

Dim
m2

0.2
0

LolitaLempicka

Chaneln5

Shalimar

-0.3

--0.2

AromaticsElixir

-0.2
02

00
0.0

02
0.2

04
0.4

Dim 1
8013

22

expert vs consumer: Multiple Factor Analysis

MFA partial points
representation
2

experts
consommateurs

(RV coefficient: 0.87)

LolitaLempicka

Cinema
LInstant

JAdore_EP
JAdore_ETCocoMelle
PurePoison
Pleasures
Chaneln5

Shalimar

-1

Dim 2 (19.35 %)

Angel

-2

AromaticsElixir

-2
I di id
8013

-1
lf

Dim 1 (64.06 %)
23

expert vs consumer: Multiple Factor Analysis

g

MFA variables
representation

expert

Gourmand
vanilla

consumer

honey
Vanille
caramel

(RV coefficient: 0.87)

anis

Dimension 2 (19.35 %)
D

camomille
nutty

citrus
Fruit
sweet_fruit

Enveloppant
freshness
0

Fraicheur marine
Fraicheur.marine

animal
musk
woody
leather
incense
earthy
Capiteux

green
jasmin
rose
Agrume
Notes.florales

Vert

f h l
fresh_lemon
intensity

Oriental
spicy
Epic

Bois

-1
-1

Dimension 1 (64.05 %)
8013

24

Comparison through the

confidence
fid
ellipses
lli
ttechnique
h i
(
(Husson,
, L & Pags,
g , 2005))
(L, Pags & Husson, 2008)

8013

25

confidence ellipses
methodology
1.Compute the product profiles (averaged by product over the judges)
2.Create the products space
3R
3.Re-sample
l b
by b
bootstraping
t t i new panels
l
4.For each new panel, compute new products profiles
5.Project as illustrative the products on the original product space
6.Steps 3 to 5 are repeated many times (i.e. 500 times)
7.Confidence ellipses around the products containing 95% of the data are
constructed

principle
if ellipses are superimposed, the products are not significantly different
the size of the ellipses is related to the variability existing around the
products
p

8013

26

confidence ellipses

Confidence ellipses around the products

Angel

Cinema
LInstant

JAdore_EP
JAdore_ET CocoMelle
PurePoison
Pleasures
Chaneln5
Shalimar

-1

Dim 2 (19..39%)

LolitaLempicka

AromaticsElixir

-3

-2

-1

Dim 1 (64.02%)
8013

27

confidence ellipses
as we have two different panels, we can apply this methodology to
both
creation of confidence ellipses around each product seen by
each panel (24 ellipses are created here)
comparison of a given product through the two panels (same
colour)
comparison off the different
ff
products within a panel (same
(
type
of line)

8013

28

confidence ellipses

Confidence ellipses for the partial points

Angel

Cinema
LInstant

CocoMelle
JAdore_EP
JAdore_ET
PurePoison
Pleasures
Shalimar

-1

Chaneln5

AromaticsElixir

--2

Dim 2 (19
9.39%)

LolitaLempicka

cons.
expert

-4

-2

Dim 1 (64.02%)
(64 02%)

8013

29

confidence ellipses
partial points
within a product, the ellipses related to the two panels are
always superimposed (no differences between the panels)

the sizes of the ellipses are equal

the higher amount of consumers compensate the higher
variability due to the lack of training for consumers

8013

30

conclusions
although consumers dont
don t have the habit to describe perfumes
(difficult task), they give the same information as the expert panel (and
its identical to the standard description of the perfumes)
they also have the same qualities (discrimination and reproducibility)
a difference between consumers and experts panel exists in the
variability of the results (more variability for consumers), but this is
compensated by the larger size of the panel (here 103 vs 12)
with co
consumers,
su e s, not
o o
only
y intensity,
e s y, bu
but a
also
so ideal
dea a
and
d hedonic
edo c
questions can be asked in the same time

8013

31

references
Earthy P., MacFie H & Hedderlay D. (1997). Effect of question order on sensory
perception and preference in central locations. Journal of Sensory Studies, vol.12, p215237
Gazano G., Ballay S., Eladan N. & Sieffermann J.M. (2005). Flash Profile and flagrance
research: using the words of the nave consumers to better grasp the perfumes
perfume s universe
universe.
In: ESOMAR Fragrance Research Conference, 15-17 May 2005, New York, NY.
Husson F
F., Le Dien S
S. & Pags JJ. (2001)
(2001). Which value can be granted to sensory
profiles give by consumers? Methodology and results. Food Quality and Preference,
vol.16, p291-296
Husson F., L S.& Pags J. (2005). Confidence ellipses for the sensory profile obtained
by principal component analysis. Food Quality and Preference, vol.16, p245-250
L S., Pags J. & Husson F. (2008). Methodology for the comparison of sensory
profiles provided by several panels: Application to a cross cultural study. Food Quality and
Preference, vol.19, p179-184

8013

32

thank you
special thanks to
Melanie COUSIN
Malle PENVEN
Mathilde PHILIPPE
Marie TOULARHOAT
students from AgroCampus-Rennes, who took care of the whole
expert panel data.

8013

33

Thank you for your attention!