Você está na página 1de 6

Human error vs design error: accidents often are blamed -

incorrectly - on human error. What appears to be


inattentive or reckless behavior might actually be a
predictable response that product and premises designers
should take into account.
Link/Page Citation
An experienced hospital nurse selects a Lasix vial from a cart containing several medications. After
checking the label three separate times, she fills a syringe and injects the patient, who subsequently dies.
The vial actually contained potassium chloride (KCl), as the label clearly stated. The nurse cannot explain
the lapse.

A teenager goes to a nearby lake, where he walks on a pier. He passes two "No Diving" signs and
reaches a point where "No Diving" is stenciled on the planks at his feet. He dives into the water, strikes
his head on the lake bottom, and suffers spinal cord injuries that leave him paraplegic. He does not
remember seeing the signs.

Accidents like these have two possible causes: human error and design error. The accident may have
occurred because the actor, or "user," (1) was inattentive, careless, or reckless or because the product's
design was unsafe and the user's behavior merely revealed the flaw.

To determine what caused an accident, many people will ask whether the user "could have done
otherwise." If circumstances or poor design limited the user's actions, then he or she is blameless. If the
user failed to prevent the mishap through possible action, then the cause is human error.

People often interpret the "could have done otherwise" criterion literally, as meaning that the user should
always perform the correct action if it is physically possible. An important distinction exists, however,
between possible behavior and likely behavior.

Unconscious processes control most of our everyday behavior, and people typically act without conscious
deliberation or consideration of risk. (2) People often cannot "do otherwise" for mental reasons that are
just as compelling as physical ones. However, product design is sometimes based on an unrealistic,
idealized notion of perfect human behavior. A design that requires possible, but unlikely, behavior is just
as defective as a design that has electrical or mechanical faults.

Taking the examples above, most people would say that the nurse was inattentive and the teenager was
reckless. However, both of these assessments are wrong. Their behavior might be better described as
skilled and adaptive.

Carelessness or skilled behavior?

A large body of scientific research shows that designers can predict user behavior with a reasonable
degree of accuracy. (3) However, faulty usability design is often missed, in part because such flaws are
so pervasive (4) and in part became most people don't have training in human factors, a discipline that
applies scientific knowledge of perception, cognition, and response to product design.

Most people--jurors included--judge behavior based on commonsense "folk psychology" and introspection
("What would I have done?") rather than on scientific analysis. However, common sense often rests on
several well-documented cognitive biases. (5)

One is "fundamental attribution error," (6) the strong tendency to attribute causation to "dispositional"
factors inside the person (such as laziness or carelessness) rather than to environmental circumstances.
Another is "hindsight bias," the mistaken belief that an outcome could have been readily predicted before
the event occurred. (7)

Hindsight bias is strengthened by "counterfactual thinking," the creation of what-if scenarios. People will
usually place causation on the mutable (changeable) aspects of an event. If counterfactual thinking
suggests that the user could have changed the outcome by different behavior, then he or she must be the
cause. (In contrast, few would say that gravity caused a fall. Gravity is immutable.)

These cognitive biases make for a much simpler world. If the nurse's inattention caused the accident, for
example, then the problem could be easily fixed by firing her; if other circumstances were the cause, then
the entire system of dispensing drugs might require review. (8) The biases also make the solution
cheaper because there is then no need to hire additional nursing staff.

In contrast, a scientific analysis examines three sets of factors controlling behavior in the situation:
physical circumstances; general human abilities, limitations, and predispositions; and individual
experiences, expectations, and goals.

The role of physical circumstances is usually obvious. A driver at night might experience dim lighting or
short braking time. If physical constraints prevent correct action, then the user could not have done
otherwise and is blameless.

Behavioral limits are less obvious. All humans are genetically endowed with abilities and limitations that
fall into three categories: perception (input), cognition (processing), and response (output). People can
only attend to so much (perception), remember so much accurately (cognition), or act so quickly
(response).

Human predispositions have evolved to help people cope efficiently with a complex world. One example
is "stimulus-response compatibility," an innate connection between perception and response.

Humans have evolved behavioral predispositions in order to act successfully and efficiently. They are
especially important in stressful situations, when users perceptually narrow their information intake. (9)
However, efficiency and accuracy sometimes trade off; predispositions provide a "quick and dirty answer,"
which is usually adequate but may fail in unusual situations.

For example, a driver who sees another car approaching on a collision course from the left will
automatically steer to the right. The natural tendency to move away from objects heading toward us is
fortunate because the conscious analysis of a projectile's path could cause a fatal delay. In some cases,
however, the rightward steer actually moves the car into the path of the crossing vehicle. Could a driver
do otherwise and steer left to avoid the collision? The answer is "yes" in the literal sense but "no" in a
more realistic, behavioral sense.

In addition, humans have many purely mental predispositions, called "heuristics" and "biases," that also
operate to quickly guide behavior, reduce cognitive complexity, and conserve mental resources.
Fundamental attribution error and hindsight biases are examples, but there are many others that figure
prominently in accidents.

One is "cue generalization," the switching from a difficult, attention-consuming visual cue to a simpler one.
Recognizing a color or shape, for example, is a much simpler perceptual task than reading. Another
powerful predisposition is "confirmation bias," the tendency to seek information that confirms already-held
beliefs and tune out contradictory evidence.

Perhaps the most important predisposition is the creation of expectations based on experience. People
would be virtually paralyzed if they had to stop and consciously analyze every detail of every situation
before acting. Instead, we use experience to develop "schemata" and "scripts"--basic knowledge units
that create expectations about how products work, what behavior is appropriate, and what effect each
behavior will have. This knowledge need not be acquired through perception at the time of product use,
so behavior is faster, smoother, and more efficient.
Users learn to focus attention on the important elements of a situation and ignore its irrelevant or
unvarying aspects. They become "inattentionally blind" (10) to information sources that do not affect the
achievement of their goal. In cognitive psychology, we would say that they have switched from conscious
"controlled" behavior to unconscious "automatic" behavior.

Most routine behavior is automatic. People are seldom consciously aware, for example, of how they steer
their cars or walk down stairs. These are complex tasks that only seem effortless because we are so
skilled at them. Our reliable, highly learned schemata almost eliminate the need to rely on conscious
control.

In fact, this is what we often mean by skill: The user has learned to substitute automatic behavior for the
slow and consciously controlled behavior typical of beginners. Asking a skilled user to explain his or her
automatic behavior is like asking a dropped ball why it fell. It is impossible to verbalize the real causes
and scientific principles at work.

Skilled, automatic behavior may fail under unusual circumstances. Since a person pays minimal attention
to irrelevant and unvarying aspects of routine tasks, he or she may not notice a change when the
situation isn't normal.

For example, the nurse who administered the KCl probably scrutinized labels carefully when she first
began her nursing career. Reading is a relatively difficult task requiring close attention and higher
cognitive processes. After performing the routine repeatedly without incident, the nurse unconsciously
learned that the Lasix vials were always in a particular cart location; the label name, packaging, and
location were redundant selection cues. She started selecting vials according to their location in the cart
(cue generalization), which was simpler and more efficient because it required less mental processing.

On the day of the accident, the positions became scrambled, and the KCl was located in the Lasix's
normal place. The nurse was merely following an unconscious, efficient schema that had always worked.
Her adaptation was skilled behavior and not inattention or carelessness. Skilled users "don't solve
problems and don't make decisions; they do what normally works." (11)

In another case, an infant received a fatal overdose of a drug after a pharmacist checked a reference but
misread an extra zero in the dosage instructions. The pharmacist was a substitute whose previous
dispensing experience was with adults, so she unconsciously expected to see a larger, adult-sized
dosage. Further, she had consulted a second reference book and made exactly the same error. While
seeing may be believing, it is equally true that believing is seeing.

These stories have several morals.

People can sometimes do otherwise in the physical sense, but not in the psychological sense. The nurse,
for example, automatically adapted by shifting from reading text to responding to the simpler location cue.
Once she "correctly" selected the vial by location, she would have engaged in "confirmation bias," a
strong expectation based on heavily reinforced experience that causes a person to miss or misperceive
the highly obvious. Her two additional checks of the vial's label could merely confirm what she believed
was the initial correct selection.

Some may say that this argument is nonsense because the nurse could have read the label and was
merely inattentive. I would answer that she could not do otherwise because she was not aware that she
was not aware.

Behavior should not be disembodied from its context. Every action occurs in a context that includes the
past and the future as well as the present. At the moment of the critical action, the past is reflected in the
user's learned expectations and the future in the user's goals.

Both expectations and goals direct attention and determine what should be relevant and what should be
ignored. The experience of successfully selecting a medication by cart location creates the expectation
that label information can be ignored. A driver whose goal is to turn left at an intersection will look both left
and right, but a driver whose goal is to turn right will look only left and ignore the fight.

Statements by people about their automatic behavior usually have little value. Since the controls of
automatic behavior largely operate outside of awareness, users cannot consciously think about them. The
nurse, for example, could not verbalize the reasons for her behavior.

Bad outcomes do not necessarily imply bad behavior. It is easy to blame the nurse in hindsight--after the
outcome is known--and to say that she could have done otherwise. Of course, she didn't know the
outcome before acting, so the outcome should not be used to judge her behavior. Since most people do
not set out to create accidents, their actions must have seemed reasonable to them at the time. To
understand the accident, the first question should then be "Why did their behavior seem reasonable to
them at the time?"

Circumstances--or, more precisely, the people who create circumstances--often cause the accident.
These people have the opportunity to make the deliberate choice to ensure proper design. The tendency
of people to adapt by reducing mental effort (by, for example, switching from reading to other cues) is
foreseeable. The design of the hospital's drug-administration system failed to take likely human behavior
into account and therefore was unsafe. Safe design must take people for what they are, not what we want
them to be.

Many authorities blame human error for almost all accidents, attributing 80 percent to 90 percent to
human error. (12) However, these studies use statistical analyses that do not consider whether the user's
behavior was reasonable given the relevant design or situation. They focus on the user because he or
she was nearest in space and time to the mishap, while the designer is a shadowy figure far from the
spotlight.

Users may be blamed for misuse as well. Product designers usually aim for a particular functionality--the
car seat is intended to keep the child restrained; the "No Diving" sign is meant to be a warning. User
behavior that is not in accord with the designer's intent may be termed product "misuse"--a subtle but
direct attempt to make the user the responsible party.

Imagine a designer who intended to create a safe toaster. If it had faulty wiring and electrocuted a user,
the manufacturer would doubtless be held responsible because the toaster was poorly designed.
Intention is not the criterion for proper design.

In contrast, consider warning signs. If the teenage diver ignored a warning sign, he probably would be
blamed for his own injuries because of the assumption that the sign fulfilled its intended function.
However, the words on the sign are not intrinsically a warning--they are just words, or perhaps more
accurately, just a bunch of lines and squiggles. Whether they will actually function as a warning depends
on a large number of design variables. (13)

Most products are capable of fulfilling many functions. The user makes his or her own determination
about the product's proper use, a perception partly based on his or her unconscious and automatic
mental processes.

Instructions may play a role in novel situations but are unlikely to be significant when the task is routine. If
intended use and perceived use do not coincide, the designer's viewpoint has no inherent precedence.
The issues are whether the designer created a reasonably usable product, reasonably transmitted the
intended use to the user, had reasonable expectations of likely user behavior, and avoided conveying
unintended or unwanted functions to the user.

The teenage diver's accident shows how intended and perceived function need not be identical. If his
behavior is removed from the context in which it occurred, he probably behaved foolishly, and one might
conclude he was just another reckless teenager.
However, the teenager had lived near the lake his entire life, safely dived many times, and seen many
other people safely dive off the pier over the years, including many on the fateful day. He had never seen
or heard of any diving accidents at the lake or seen the no-diving rule enforced. He had even seen
hometown newspaper photographs of people diving from the pier.

From the teenager's perspective, he had to weigh the signs' credibility against a lifetime of direct
experience. He didn't even "see" the signs on the pier that day. They were as irrelevant as the feeling of
your shoes on the soles of your feet--which you probably hadn't noticed until just now.

In fact, the "No Diving" sign confused real and unreal hazards. The water depth around most of the pier
was sufficient to allow safe diving, which is why no previous accidents had occurred. Unfortunately, the
teenager dived into the one small place where the water was shallow. The signs "cried wolf," obscuring
real risks, overwarning, and destroying their own credibility.

Products may also signal unintended functions. One frequent cause is unintended "affordances," (14)
which are innate connections between a product's appearance and its possible uses. People will judge
proper use simply by looking at the product because "some product characteristics strongly trigger a
particular use action." (15)

For example, in one case, a driver placed papers on his dashboard, causing a windshield reflection that
partly obscured his view. His car then struck and killed a pedestrian walking along a road.

The accident occurred because the driver responded to an unintended function suggested by the
vehicle's design. The dashboard was a flat horizontal surface that could be used as a table. The driver
would never have had the opportunity to make this error if the dashboard had been properly designed,
say, with a curved surface. It was completely foreseeable that some drivers would use the dashboard as
a table.

Final analysis

This scientific approach to human behavior has several implications for attorneys. First, a user who "could
have done otherwise" and prevented an accident is not necessarily at fault. "Could have prevented" does
not necessarily imply "should have prevented."

Second, the invisibility of unconscious mental processes leads to gross overestimation of human error
and to underestimation of design error.

Third, designs that rely on possible but unlikely behavior are just as defective as designs with improper
electrical or mechanical engineering design. They ignore human factors design and rely on unrealistic
assumptions about perfect human behavior. Such cases should be examined to see whether the error
was predictable and whether the design should have been based on a more realistic assessment of likely
behavior.

Last, the defendant's human factors development process should be closely examined. Products are
often developed without proper human factors design or evaluation.

Notes

(1.) Defined generically, a "user" is a person attempting to employ a "product," which is any man-made
artifact, including consumer goods, warnings, steps, and roadways. A "designer" is the person
responsible for the product's form.

(2.) See, e.g., Willem Wagenaar, Risk Taking and Accident Causation, in RISK-TAKING BEHAVIOR 257
(John Yates ed. 1992).
(3.) Specific behavior cannot be predicted perfectly, but it is possible to predict general behavioral
tendencies. If this were not true, then human factors design would be impossible. However, user testing
may be needed to confirm likely behavior--especially if a product is novel, complex, or critical for safety.

(4.) For some everyday examples of faulty design, see Marc Green, Human Error vs. Design Error (2005),
available at www.visualexpert.com/ Resources/humanvsdesignerror.html (last visited May 1, 2006).

(5.) Marc Green, Skewed View: Accident Investigation, OCCUPATIONAL HEALTH & SAFETY CAN.,
June 2003, at 24.

(6.) Lee Ross, The Intuitive Psychologist and His Shortcomings: Distortion in the Attribution Process, in
10 ADVANCES IN EXPERIMENTAL SOC. PSYCHOL. 173 (Leonard Berkowitz ed. 1977).

(7.) See, e.g., Reid Hastie et al., Juror Judgments in Civil Cases: Hindsight Effects on Judgments of
Liability for Punitive Damages, 23 L.& HUM. BEHAV. 597 (1999).

(8.) See, e.g., Marc Green, Nursing Error and Human Nature, 9 J. NURSING L. 37 (2004).

(9.) J. A. Easterbrook, The Effect of Emotion on Cue Utilization and the Organization of Behavior, 66
PSYCHOL. REV., at 183 (1959).

(10.) See generally ARIEN MACK & IRVIN ROCK, INATTENTIONAL BLINDNESS (1998); Marc Green,
Let's Not Blame the Victim ... Just Yet: Inattentional Blindness, OCCUPATIONAL HEALTH & SAFETY
CAN. Jan./Feb. 2002, at 23.

(11.) HERBERT L. DREYFUS & STUART E. DREYFUS, MIND OVER MACHINE 30 (1986).

(12.) See, e.g., What Causes 90 Percent of All Automobile Accidents?, available at www.visualexpert.
com/accidentcause.html (last visited May 1, 2006).

(13.) Marc Green, Why Warnings Fail, OCCUPATIONAL HEALTH & SAFETY, Feb. 2004, at 20.

(14.) JAMES J. GIBSON, THE ECOLOGICAL APPROACH TO VISUAL PERCEPTION (1979).

(15.) Freija Hrosvitha van Duijne, Risk Perception in Product Use (2005) (unpublished Ph.D. dissertation,
Delft University of Technology) (on file with author).

MARC GREEN is principal of Visual Expert Human Factors and adjunct professor of ophthalmology at the
University of West Virginia medical school.
COPYRIGHT 2006 American Association for Justice
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2006, Gale Group. All rights reserved. Gale Group is a Thomson Corporation Company.

Please bookmark with social media, your votes are noticed and appreciated: