Title: Beyond anxiety and ‘political correctness’: How experimental design trumps ‘gam

ing it’ and gets more deeply into the mind

Howard R. Moskowitz, Ph.D.(corresponding author)
Moskowitz Jacobs Inc.
1025 Westchester Ave.
White Plains, New York, USA
Phone: 914-421-7400
Fax: 914-428-8364
E-mail: mjihrm@sprynet.com
Jacqueline Beckley
Hollis Ashman
The Insight & Understanding Group
Denville, New Jersey 07834 USA
Phone: 973- 328-9107
Fax: 973- 328-9108
E-mail: jackie@theuandigroup.com
hollis@theuandigroup.com
Abstract
The researchers present a research approach which was stimulated, in part, by ge
nomics thinking in the biological sciences, conjoint analysis in marketing, and
psychophysical measurement in experimental psychology. We present the approach i
n terms of different steps, both conceptual and execution, respectively, as a me
thod for measuring truth for highly, emotionally tinged topics. In doing so, we
overcome the continuing issues in both consumer and public opinion research wher
e the respondents’ desire to please the interviewer by seeming to be politically c
orrect (‘PC’) provides erroneous information.

Keywords: conjoint analysis, ideation, concept development, concept-response, se

gmentation
Introduction
One of the continuing issues in consumer and public opinion research is
the respondent’s desire to please the interviewer or be perceived as being ‘PC’ (polit
ically correct). Covert social pressure may well lead to erroneous information,
as the interviewee weighs the answer to be given in light of truth, expected int
erviewer reaction, and the interviewee’s own ideals. How, then, does truth get me
asured, especially when measuring responses for emotionally tinged topics?
During the past 15 years, we have worked extensively with experimental d
esign of ideas (known as conjoint analysis). The respondent is presented with a
set of direct statements constituting a vignette. S/he is then instructed to rat
e the entire vignette as a single paragraph on a single evaluative criterion (e.
g., interest in a product; intention to vote for the political candidate, etc.).
The combinations vary, so that each screen presents the respondent with a compl
etely new combination of elements to be assessed. Furthermore, each respondent e
valuates a unique set of combinations. The same elements are presented, but in n
ew combinations. Every element appears several times (e.g., 2-4), against diffe
rent backgrounds, making it virtually impossible for the respondent to develop a
response set induced by the constancy of a stimulus. Everything changes continu
ally. (Moskowitz and Gofman, 2004; Moskowitz, Porretta and Silcher, 2005)
The analysis of the data is done at the individual respondent level, as
each respondent receives the appropriate number of test vignettes, based upon t
he number of elements and the number of silos or ‘buckets’ into which the elements c
an be placed. The basic experimental design is permuted, so that each respondent
 sees these different combinations. Yet, because each respondent rates combinat
ions constructed by an individual design, it is possible to analyze the data by
individual. It is also possible to construct a wide number of different classifi
cations, based either on the way respondents describe themselves in a parallel c
lassification questionnaire, respond to specific concept elements or the entire
set of concepts, and subsequently, completed utilities (so-called concept respon
se segmentation.)
Rather than presenting the results in detail, we are writing this paper
to explicate a research approach which was stimulated, in part, by genomics thin
king in the biological sciences, conjoint analysis in marketing, and psychophysi
cal measurement in experimental psychology. We present the approach in terms of
different steps, both conceptual and execution, respectively.
Explicating the approach – Steps combining consumer research and psychophysics
Step 1 – Create a structured warehouse for attitudinal data. The approach we used
comes from the belief that a great deal can be learned about people’s attitudes b
y performing multiple studies of a topic, each study focusing on a specific issu
e. Thus, these data on attitudes towards war and terrorism come from a larger se
t of 15 studies on topics dealing with anxiety. Table 1 shows the silos and elem
ents for the two studies discussed here – terrorism and war. One of the goals of
the database was to use the same elements, or quite similar elements, in each st
udy to allow for comparison of the impact of elements. One organizing principle
to keep in mind is that the study of well-structured stimuli teaches alot, espe
cially when the topic being studied is represented by many stimuli. Researchers
will learn more from a collage of stimuli presented in a somewhat superficial m
anner than from one stimulus deeply studied.

Table 1: Silos and elements

Rationale Terrorism War
Silo A – Problem
A1 Media talking about the issue The media talking about potential terror
ism acts… The media talking about potential war…
A2 A threat A bomb threat for a building that is a false alarm…
Working through the Security Council of the United Nations to impose sanctions..
.
A3 More intense description A bomb under your car… Diplomacy that i
sn’t working to stop hostility…
A4 More intense description Bombs blowing up in the middle of a buil
ding… A build up to war that ends up a false alarm…
A5 More intense description Fire raging through a building… Seeing m
y friends or family getting called up to go fight…
A6 More intense description Contamination of the food supply… Bombs bl
owing up in the middle of a city…
A7 More intense description A deadly disease like smallpox or anthra
x let loose...
Friendly Fire killing troops...
A8 More intense description A Computer virus let loose that impacts
your everyday businesses…
A deadly disease like smallpox or anthrax let loose...
A9 More intense description A dirty nuclear bomb set off …
A dirty nuclear bomb set off…
Silo B – Venue
B1 No one affect In a non-populated area… In a non-populated area…
B2 Many affected In a heavily populated area… In a heavily populated a
rea…
B3 Children affected An area crowded with children… An area crowded
with children…
B4 Parents/seniors affected An area crowded with senior citizens…
An area crowded with senior citizens…
B5 Others affect An area filled with tourists… An area crowded with sol
diers…
B6 Warning level 1 When you least expect it… You never expected it to happen
to you or someone close to you...
B7 Warning level 2 During a Yellow alert… During a Yellow alert…
B8 Warning level 3 During an Orange alert… During an Orange alert…
B9 Warning level 4 During a Red alert… During a Red alert…
Silo C - Your emotions & reactions
C1 Alone and helpless You are all alone… and you feel helpless… You thin
k about it when you are all alone…and you feel so helpless
C2 Can t stop thinking about it You think about it, you just can t stop
thinking about it… and you feel uneasy...
When you think about it, you just can t stop...
C3 Get away You d drive any distance to get away from it… You d dr
ive any distance to get away from it…
bC4 Scared You are scared … inside and out You are scared … inside and out
C5 Sensory You experience it all … seeing, smelling, tasting You experience i
t in all your senses…
C6 Overwhelmed All the stress just builds up… you feel overwhelmed
All the stress just builds up… you feel overwhelmed
C7 Memory loss- maximum depression You experience temporary memory loss bec
ause there s just too much to take in... You experience temporary memory
loss because there s just too much to take in...
C8 Family and friends While surrounded by family and friends...
Family and Friends play a big role in your life…
C9 Turning point+B17 At a special moment… in your life At a turning poi
nt in your life...
Silo D - Relief & protection
D1 Highest global authority You trust that God will keep you safe
You trust that God will keep you safe
D2 Next highest global authority You believe that international cooperati
on in the United Nations will keep you safe You believe international cooper
ation in the United Nations will keep you safe
D3 Next highest global authority You think United Nations Forces will kee
p you safe You believe the United Nations Forces will keep you safe
D4 Next highest global authority You believe that Homeland Defense will k
eep you safe You believe the Military will keep you safe
D5 Next highest global authority You believe that the Center for Disease
Control will keep you safe You believe the National Reserves will keep you
safe
D6 Local authority You think that your Local police will keep you safe
You trust your Local police will keep you safe
D7 Local authority You think that your Local hospital will keep you safe
You trust your Local hospital will keep you safe
D8 Media keeps you informed Its important for the Media will keep yo
u informed Its important for the Media to keep you informed
D9 Contact with family and friends You need to contact your friends and fam
ily to make sure they are OK… You need to contact your friends and family to m
ake sure they are OK…
Step 2 –Run multiple studies simultaneously in order to fill the data warehouse. I
n our efforts to create this knowledge base we created 15 studies. Respondents r
eceived e-mail invitations to participate. Those who chose to accept clicked on
an embedded link and were led to the screen shown in Figure 1. They chose a stud
y that interested them or opted to drop out. This self-selection, however, may
generate a somewhat biased group of respondents, since presumably people chose a
survey topic that interested them. With sufficient number of invitations issued
, the studies can fill up quickly. Therefore, the database may be completed in a
s rapidly as a few days.
Figure 1 – The ‘wall’ to which respondents were led. A respondent could choose any of
the studies, and participate in that study, albeit participation per study.

Step 3 – Orient the respondent. Introduce the study by an orientation page (see Fi
gure 2). The orientation does not tell the respondent more than is absolutely ne
cessary.

Figure 2 – Example of the orientation page. This particular study dealt with terro
rism.

Step 4 – Run the actual evaluation of test concepts. Present each respondent with
different combinations of elements, according to a ‘main effects’ experimental desig
n. With four silos, some concepts comprise one element from each silo, whereas o
ther concepts comprise one element from three of the four silos. Figure 3 shows
an example of a test concept. Again, keep in mind that each respondent evaluates
different sets of concepts. It is virtually impossible for a respondent to keep
track of his answers or trace those answers back to the specific element. As th
e interview progresses, the respondent is forced to answer questions at a rapid
pace and, frequently, does not stop to give rational answers. It is important to
make sure that the question is ‘meaningful’, intuitively easy to answer for the top
ic, and anchored at both ends of the scale, in order to reduce confusion and amb
iguity.
Figure 3 – Example of a test concept

Step 5 – Create a model for each respondent. The independent variables in the mode
l are the 36 elements, which are present or absent in the concept. These are so-
called ‘dummy variables’. The dependent variable is either the rating itself or, mor
e frequently, a binary conversation of the variable. For this database, the rati
ngs 1-6 (easy to deal with the issue) were converted to 0, whereas the ratings 7
-9 (hard to deal with the issue) were converted to 100. For each respondent rela
te the presence/absence of the 36 concept elements to the ratings, using the met
hod of ordinary least squares. Each dependent variable generates its own equatio
n, of the form:
Rating = k0 + k1(Element 1) +
k2 (Element 2) .. k36(Element36)
Step 6 – Segment the respondents by the pattern of their 36 ‘utility’ or impact values
. The segmentation used so-called k-means clustering (Systat, 1997). We identifi
ed two meaningful clusters or segments, although there might possibly be more.

Step 7 – Look at the additive constant across the 15 studies in the database. The
additive constant tells us the basic level of the respondents’ inability to deal w
ith the situation. Respondents do not estimate this basic level of inability. Ra
ther, it is deduced from the pattern of responses. Regression analysis partials
out the contributions of the individual elements to a person’s rated inability to
deal with the situation. Once we partial out those contributions, what is left i
s the respondents’ basic inability. Respondents cannot consciously generate a pre
determined level of base anxiety from the pattern of responses. In Figure 4, w
e see that personal situations such as income are far more likely to generate a
basic feeling of ‘I cannot deal with this’ compared to more general problems such as
terrorism. Indeed, terrorism is on par with phobias in terms of the respondents’
basic inability to deal, even though it would be counter-intuitive if this conc
ept was asked in a regular survey!
Figure 4 – Ability to cope with a situation. The bars represent the additive const
ant for the total panel. The additive constant shows the basic level of the resp
ondents’ inability to cope with a situation, in the absence of any messaging.

Step 8 – Look at the ‘knowns’ to establish consistency and validity. One way to estab
lish that the survey is valid is to look at the relationship between the indepen
dent variables and dependent variables. For example, we know that a yellow alert
signifies a less serious threat than an orange alert, which, in turn, signifie
s a less serious threat than the red alert. When these alerts are embedded in a
set of concepts with everything else in the concept also varying, do we see thi
s order of red>orange>yellow in terms of being unable to deal with the issue? T
he answer is yes, although the effect is marginal. Looking at Table 2 (below), w
e see the impact or utility value. A negative number means relief (pushes away f
rom; can’t deal with the situation), whereas a positive number means increasing a
nxiety. The order of colors is exactly what we expect. Yellow is less anxiety-p
rovoking than orange, which is less anxiety-provoking than red. What is just as
important is that the effects are minimal. The dynamics for war are slightly dif
ferent from those of terrorism. A red alert during war is almost irrelevant, whe
reas a yellow alert is a cause of a bit of relief. Both are anxiety-provoking fo
r terrorism, as they should be.
Table 2 – How different alert colors drive ability to deal with the situation (neg
ative numbers) or drive inability to deal with the situation (positive numbers)
War Terrorism
B7 During a Yellow alert… -2 2
B8 During an Orange alert… -1 2
B9 During a Red alert… 1 4
Step 9 – Look at what is most anxiety-provoking versus most anxiety-reducing. Most
of the elements had little effect. However, some interesting repeating patterns
and surprises emerge. Direct personal threats are clearly very anxiety-provokin
g and, more so, for terrorism than for war. But, surprisingly, so is the involve
ment of the United Nations. This is an exceptionally anxiety-provoking statement
, since it is the opposite of the original concept of what the UN was originally
created to do.
In the context of terrorism, a computer virus elicits a marginal response, provi
ding a bit of comic relief, except, perhaps, to a person whose computer is infec
ted with the virus. God, friends, family, and information provided by the media
are all anxiety-reducers. It appears that the basic ability of respondents to de
aling with the concepts of terrorism and war are equal. After that, however, spe
cifics for terrorism generate greater anxiety than those same specifics for war.
(See Table 3).
Table 3: The most impactful elements driving the ability to deal with a situatio
n (negative numbers) or the inability to deal with a situation (positive numbers
).
War War Terrorism
Base Size 124 Base Size 121
Constant 20 Constant 19
A9 A dirty nuclear bomb set off… 16 A3 A bomb under you
r car… 21
A6 Bombs blowing up in the middle of a city… 9 A9
A dirty nuclear bomb set off … 20
A8 A deadly disease like smallpox or anthrax let loose... 9
D2 You believe that international cooperation in the United Nations
will keep you safe 20
D3 You believe the United Nations Forces will keep you safe 8
D3 You think United Nations Forces will keep you safe 19
D2 You believe international cooperation in the United Nations will keep yo
u safe 8 A4 Bombs blowing up in the middle of a buil
ding… 15
A7 A deadly disease like smallpox or anthra
x let loose... 14
D5 You believe that the Center for Disease
Control will keep you safe 12
A6 Contamination of the food supply…
10
D4 You believe that Homeland Defense will k
eep you safe 10
C8 Family and Friends play a big role in your life… -3
C6 You trust that God will keep you safe -3
D9 You need to contact your friends and family to make sure they are OK…
-4 C5 It’s important for the Media will keep you informed
-5
A4 A build up to war that ends up a false alarm… -6 B8
You need to contact your friends and family to make sure they are OK… -6
D1 You trust that God will keep you safe -6
D8 It’s important for the Media to keep you informed -7
10. Segment the respondents by the pattern of their utilities. Segmentation prov
ides a way of delving into mind-sets. Public opinion and market research studies
often segment responses on the basis of patterns of reactions to questions that
ask respondents to ‘define themselves’. Our goal, in this data set, is to generate
segmentation based upon patterns of reactions to the concept elements. Since the
respondents do not tell the interviewer what is important, the importance or dr
iving power of the individual elements is obtained from the pattern of responses
.
We see these patterns when we divide the respondents by the patterns of their ut
ilities;
War (Table 4) – Two segments emerge. The bigger group is the ‘Frightened Ordinary Ci
tizen’, the smaller group is the ‘Local Patriot. The local patriot is less anxiety p
rone (constant = 15) than the frightened ordinary citizen (constant = 23). The
Frightened Ordinary Citizen is afraid of injuries, and is comforted by religion.
The Local Patriot is afraid of involvement of international bodies or even the
government in general, and is comforted by seeing friends become comrades in arm
s.
Table 4: War (total and segments). The table shows the most impactful elements d
riving the ability to deal with a situation (large negative numbers) and the mos
t impactful elements driving the inability to deal with a situation (large posit
ive numbers).
War
Tot FOC LP
Base Size 124 77 47
Constant 20 23 15
‘Frightened Ordinary Citizen’
A dirty nuclear bomb set off… 16 23 5
A deadly disease like smallpox or anthrax let loose... 9 16
-3
Bombs blowing up in the middle of a city… 9 14 0
Friendly Fire killing troops... 5 10 -2
Seeing my friends or family getting called up to go fight… 3 8
-5
A build up to war that ends up a false alarm… -6 -9 -2
Its important for the Media to keep you informed -7 -9 -2
You trust that God will keep you safe -6 -14 7
‘Local Patriot ’
You believe international cooperation in the United Nations will keep you safe
8 -4 28
You believe the United Nations Forces will keep you safe 8 -3
27
You believe the National Reserves will keep you safe 2 -6 14
You trust your Local police will keep you safe 1 -6 13
You trust your Local hospital will keep you safe 3 -3 12
You believe the Military will keep you safe 1 -5 10
During a Red alert… 1 3 -3
A deadly disease like smallpox or anthrax let loose... 9 16
-3
In a non populated area… -3 -3 -3
An area crowded with soldiers… 0 2 -3
Seeing my friends or family getting called up to go fight… 3 8
-5
Terrorism (Table 5) – The same two segments emerge, based upon the pattern of util
ity values. However, the dynamics differ. The Frightened Ordinary Citizens are t
he slightly smaller group (N=57 versus N=67 for the Local Patriot). Furthermore,
the Frightened Ordinary Citizen is less immediately plagued by general anxiety
(additive constant = 9) than is the Local Patriot (additive constant = 27).
However, the dynamics of the elements are similar to what we saw for war, only m
ore pronounced. Both groups are much more anxiety-prone to what bothers them, w
ith impact values of 30 or more for a number of situations. Very little action w
ill reduce the anxiety of the Frightened Ordinary Citizen. The computer virus ag
ain is perhaps a bit of comic relief, when juxtaposed against elements that deal
with dirty nuclear bombs, or bombs under one’s car. In contrast, the Local Patri
ot is anxious about the involvement of the authorities, no matter who they are,
and finds anxiety relief with a personal, and the knowledge that friends and fam
ily are safe.
Table 5: Terrorism (total and segments). The table shows the most impactful elem
ents driving the ability to deal with a situation (large negative numbers) and t
he most impactful elements driving the inability to deal with a situation (large
positive numbers).
Terrorism
Tot FOC LP
Base Size 121 54 67
Constant 19 9 27
‘Frightened Ordinary Citizen’
A dirty nuclear bomb set off … 20 39 5
A bomb under your car… 21 38 8
Bombs blowing up in the middle of a building… 15 31 2
A deadly disease like smallpox or anthrax let loose... 14 28
3
Contamination of the food supply… 10 19 3
An area crowded with children… 6 11 2
In a heavily populated area… 3 10 -3
Fire raging through a building… 4 9 0
A Computer virus let loose that impacts your everyday businesses… -3
-4 -2
‘Local Patriot ’
You think United Nations Forces will keep you safe 19 0 34
You believe that international cooperation in the United Nations will keep you s
afe 20 3 33
You believe that Homeland Defense will keep you safe 10 -3 20
You believe that the Center for Disease Control will keep you safe 12
3 18
You think that your Local police will keep you safe 7 -1 14
You think that your Local hospital will keep you safe 6 -1 12
A bomb under your car… 21 38 8
An area filled with tourists… 1 8 -4
At a turning point in your life... -2 1 -4
In a non-populated area… -1 3 -4
The media talking about potential terrorism acts… -2 2 -5
Family and Friends play a big role in your life… -2 1 -5
You d drive any distance to get away from it… -2 5 -7
Its important for the Media will keep you informed -5 -2 -7
You trust that God will keep you safe -3 5 -10
You need to contact your friends and family to make sure they are OK… -6
-1 -11
Some observations on world-view, method, analytic strategies
Observations on surveys versus experimental design
It is worth pointing out that this study was an experiment to identify h
ow respondents integrate information about different aspects of an anxiety-prov
oking situation, weight the impact of the information, and then come up with a s
ingle answer as dictated by the research instructions. It is also important to k
eep in mind that the approach owes a lot to experimental psychology, with its fo
cus on systematically varying the situation (test concepts), measuring reactions
(ratings), and then searching for a pattern of responses. The critical differen
ce between this type of work and conventional surveys is that surveys are not e
xperiments, but just means by which to elicit the attitudes of the respondents.
Here, the focus is on actual behavior of respondents when they are confronted wi
th a collection of survey issues.
Benefits of experimental design
The world of experimental science always seems to fight with the world o
f observational science. Both disciplines have their place in the pantheon of th
e sciences that make up our world. However, they are different. Conventional sur
vey-taking usually works with the goal of measuring what is, and only occasional
ly works as a measure before or after an experimental intervention. In a sense,
the researcher contributes a great deal in survey research. It is the astute res
earcher who can draw out insights from a set of data, who sees patterns and rela
tions that were not obvious. The greater the number of years that the observatio
nal researcher works, the more likely that the researcher will add greater value
to the research process, much like the experienced clinician adds value.

Perhaps the same can be said for the experimenter, but we are not really
sure.
The experimenter is skilled in setting up experiments. In some rare cases, the e
xperimental design might require a talented experimenter. Yet, for the most part
, most really do not. Experimental science is open, public and, for the most pa
rt, available to all researchers. Perhaps the key benefit for the experimental r
esearcher is that the discipline of experimental design forces the researcher to
create the conditions to discover something. That is far more possible than wha
t confronts the observation researcher. That researcher must wait until somethin
g ‘pops out’, perhaps totally unexpectedly, and then be sufficiently aware of the im
portance of the phenomenon to jump on it, and present it in all its facets.

Acknowledgment
Dr. Howard R. Moskowitz wishes to thank his editorial assistant, Linda Lieberman
, for reviewing and preparing this paper and generally keeping him on production
schedule.
References
Beckley, Jacqueline, and Howard R. Moskowitz. Databasing the Consumer Mind: the
Crave It!, Drink It!, Buy It!, Protect It!, and the Healthy You! Databases. Annu
al Meeting, June 2002, Institute of Food Technologists. Anaheim, CA: Institute o
f Food Technologists, 2002.
Box, G.E.P.., J. Hunter, and S. Hunter. Statistics for Experimenters. New York:
John Wiley, 1978.
Luce, R. D., and J. W. Tukey. "Conjoint Analysis: a New Form of Fundamental Meas
urements." Journal of Mathematical Psychology 1 (1964): 1-36.
Moskowitz, Howard R., Alex Gofman, Rachel Katz, Barbara Itty, Madhu Manchaiah,
and Zhenyu Ma. "Rapid, Inexpensive, Actionable Concept Generation & Optimization
-the Use and Promise of Self-Authoring Conjoint Analysis for the Foodservice Ind
ustry." Foodservice Technology. 2001. 149-168.
Moskowitz, Howard R., and Alex Gofman. United States. Patent Office. System and
Method for Performing Conjoint Analysis. US Patent Pending, US2005/0177398 A1,
August 2005.
Moskowitz, Howard R., Sabastiano Poretta, and Matthias Silcher. Concept Researc
h in Food Product Design and Development. Ames, IA: Blackwell Professional, 2005
.
Stevens, S. S. Psychophysics: an Introduction to Its Perceptual, Neural and Soci
al Concepts. New York, NY: John Wiley, 1975.
Systat. Systat, the System for Statistics. Version 11 ed. Evanston, ILL: Systat,
2007.
Wittink, D. R., M. Vriens, and W. Burhenne. "Commercial Use of Conjoint Analysi
s in Europe: Results and Critical Reflections." International Journal of Researc
h in Marketing 11.S (1994): 41-52.