by TAYLOR CONE ENG + TECH
Teaching
Credit: sxc.hu
Helicopters
to Fly
A radio-controlled helicopter,
glistening bright red in a baby
blue sky, carries out stunning
maneuvers – rolls, flips, inverted
hovers, and more. The air show-
caliber routine continues for several minutes,
each maneuver executed perfectly and
without mistakes. After the show, you walk
over to the pilot to congratulate him or her
on an outstanding performance only to find
that a computer was in complete control for
the duration of the show.
Sound like a dream? Thanks to recent
developments in machine learning and
artificial intelligence this scenario is no
longer science fiction. Stanford researchers
have developed new algorithms that allow
computers to “learn” Through complex logic
and countless lines of code, some computer
systems have the ability to learn from the
patterns they observe.
Under the direction of Computer Science
Professor Andrew Ng, Ph.D. student
Adam Coates has spent the last five years
developing an “apprenticeship learning”
system that involves a human “teacher”
from which the computer observes and
learns. Coates, along with fellow Ph.D.
students Pieter Abbeel, Timothy Hunter, and
Morgan Quigley, use radio-controlled (RC)
helicopters as the test bed for the project. In
particular, they study the controls necessary
to maintain stable flight as well as perform
Researchers
stunning maneuvers. With several onboard develop learning
sensors, two ground-based cameras and a
few computers doing the brute force data algorithms
analysis, the control system has learned to
for helicopter
execute some impressive aerobatics.
Can Computers Learn aerobatics
The advent of artificial intelligence brought
a number of revolutionary concepts to our
understanding of what computers can do.
One such concept is machine learning,
which involves the design and development
of algorithms that allow computers to learn
from the patterns found in recollected data.
This grants computers abilities never before
utilized and has implications on the areas of
robotics, manufacturing, warfare and search
and rescue among others.
Conventionally, programmers write codes
that explicitly outline every command to
a computer. This is useful and efficient in
most contexts and has had great success
throughout the history of computing.
However, with more complex control
problems, such explicit coding is extremely
inefficient. In these situations, it may make
more sense to turn to machine learning
techniques, where the computer has the
ability to determine the commands by its
own.
volume VIII 29
Credit: Andrew Ng
Snapshots of the flight of the
helicopter during a complicated
maneuver.
The breadth of the
applications of these
learning algorithms
is only limited by
our willingness to
teach our skills to
computers.
30 www.stanfordscientific.org
For example, imagine a scenario as simple as
picking up a pen. Think of all the commands
your brain sends your muscles during this
action. Your brain considers how far away
the pen is, when to open your fingers,
which fingers to use, how tightly to grasp
it, which muscles to employ, how much to
contract these muscles, the speed at which
to execute all of this, and much more. If we
were conscious of every command going
through our brain at a given moment, we
would not be able to focus on anything else.
Similarly, if we are able to minimize the
amount of explicit coding we put into a
program, it will be that much easier to
understand and control. This is exactly what
Coates and his colleagues have in mind:
if they can simply write robust learning
algorithms, their programs will be able to
figure out what to do in specific situations
on their own.
Apprenticeship
Learning
Ng’s research group chose to pioneer a
new type of machine learning for their
autonomous helicopter project. Rather than
using conventional machine learning that
involves the computer running a simulation
and learning from the resulting patterns,
they integrated a human element.
Expert RC helicopter pilot Garett Oku
acts as the team’s teacher – not teaching
the students how to fly, but teaching
the helicopters. Having a human teacher,
particularly such an experienced pilot like
Oku, “made us realize that when you have
this slight advantage of having someone
who can help you with the learning process,
that is a huge boon,” Coates says.
Since humans can already fly RC helicopters
with relative ease, it seems logical to replace
computer-run simulations with human pilot
mentors. As Coates describes, “we set out
to use learning algorithms to try to learn to
fly the helicopter the way a human could
to demonstrate that rather than solving this
using classical control methods you could
use artificial intelligence algorithms to learn
to fly this the way a human does.”
“As soon as we had a human to teach the
computer so the computer knew where
to start, that completely changed the ball
game,” Coates explains. Not only do they
now have a human-controlled system for
the computer to observe, but the responses
– the inputs and outputs – are all from
the real world rather than a simulation.
Combined, these benefits make for a much
more effective machine learning system.
Why Helicopter
Aerobatics?
Although autonomous acrobatic helicopters
are the most tangible result of this project,
it is important to note that in the eyes of
the researchers, helicopters are merely test
beds for the development of their machine
learning algorithms. So if they are just a test
bed, what motivated the research team to
use helicopters?
Mostly, helicopters present a unique design
challenge for they are inherently difficulty to
control.
“If you’re flying in an airplane and you let
go of the controls...the airplane will tend
to keep itself upright without any human
interaction,” says Coates. The same is
certainly not true for helicopters, which tend
to flip over and crash without any human
input. According to Coates, “they’re much
more complicated and the behavior of a
helicopter is just really difficult to predict.”
If stable helicopter flight is already
difficult enough, why make it even more
complicated by pursuing aerobatics? “The
reason we wanted to do aerobatics is
because we can demonstrate these learning
algorithms are able to solve really hard
control problems that until now have been
completely inaccessible,” Coates explains.
“Now we can actually meet the level of a
human pilot, which is brand new.”
This is the most intriguing part of the
project: humans have already solved
this extremely difficult control problem.
Knowing little or nothing about
aerodynamics, a human pilot can fly
aerobatics with a radio-controlled helicopter.
Humans “clearly have this system that’s able
to fly helicopters well beyond anything that
the best computers were able to do, yet as
a control problem it’s still largely unsolved,”
Coates adds.
The Hardware
Even though the project’s focus is software
development, it is nonetheless imperative to
have a robust physical platform with which
to test the algorithms. In fact, the physical
implementation can sometimes cause
the most problems, as Coates describes:
“We were worried that even if the learning
software were good enough to do [what
we wanted], we were concerned that sensor
technology would just not be capable.”
Fortunately, the hardware proved to be very
successful.
As for the airframe itself, it is simply an
off-the-shelf helicopter, as are the engines
and the rest of the propulsion system. The
rest of the hardware – the gyroscopes,
accelerometers and compasses – are all
custom added and stored in a 100-gram
black box on the helicopter.
Inside the black box are three gyroscopes
which sense angular rotation about three
axes, three accelerometers which sense
acceleration along the three linear axes,
and a three-dimensional magnetic compass
which senses the earth’s magnetic field.
Together, these sensors give a relatively
good idea of the orientation of the aircraft
with respect to the earth.
To add position information, the team has
a pair of cameras on the ground as well.
These two cameras effectively act as eyes,
tracking the helicopter in the sky and
generating the data necessary for specific
position knowledge. “All of these sensors
get a coherent estimate in the computer of
where the helicopter is and which way it’s
pointing,” Coates explains.
The Software
Considering how many sensors are
providing the computers with constant
input, it is a veritable challenge interpreting
it all and producing meaningful outputs.
What does the helicopter’s “learning
process” actually consist of?
The first step of the process involves a
human pilot – Oku in this case – conducting
several aerobatic flights with the helicopter
while the computer “observes.” During
this time, the computer examines the
pilot’s every input and the exact result.
These results vary by trial due to weather
conditions as well as changes in the system,
and the computer records these minor
differences among the trials. After enough
trials, the computer can then differentiate
the pilot’s intended trajectories and the
actual resultant trajectories, thereby
understanding the necessary commands
for intended results in a wide variety of
scenarios.
“The basic approach is to have a human fly
the helicopter while the computer observes
how the helicopter behaves,” Coates says.
“From that, the computer tries to learn what
the human was trying to accomplish, the
maneuvers they were trying to fly. Then,
once we put the computer in control of the
helicopter, it has to figure out what actions
to execute in order to repeat what the
human did.”
How well the computer learns and
accurately replicates the maneuvers carried
volume VIII 31
Credit: Andrew Ng
The helicopter learns how to recover out by the human pilot is actually quite
its original position after flying upside- astounding. “We can fly air shows that are
down. as good as those flown by our expert pilot,”
Coates claims. Pretty impressive, considering
the inherent complexity of the system being
used. “These maneuvers represent by far the
most advanced, most difficult maneuvers
flown on any computer controlled
helicopter,” Ng adds.
Firefighting and
Autonomous
Cartography
If the researchers pursued the possibility
of extending the same principles into
full-scale helicopters, these algorithms
would have great implications in tasks such
as firefighting. In fact, a more complex
algorithm would be necessary for such a
scenario, which would require consideration
of the air currents associated with fire.
“The interesting thing about the firefighting
application is you have a lot of nasty air
currents moving around that can affect the
helicopter,” Coates explains. “Excellent pilots
can compensate for this but it would be a
little scary to have a hand-tuned control
system flying because you have no idea
what will happen when it gets hit with
turbulence.”
If the team’s algorithm were able to watch
an expert human pilot fly in such conditions,
“These maneuvers it would learn how to react to unforeseen
circumstances and hopefully be able to do
represent by far the the same when placed in that environment.
most advanced, most “Lately we’ve seen a lot of firefighting in
California with helicopters and we’ve seen
difficult maneuvers how dangerous that can be, and it’d be
nice to get those people out of harm’s way,”
flown on any Coates says.
computer controlled Another useful application is in autonomous
flight and aerial mapping and cooperation
helicopter.” - Andrew between air and ground vehicles. Imagine
Ng there is a ground vehicle that is trying to
reach a goal somewhere else but it cannot
32 www.stanfordscientific.org
see the path to get there. “We could have a
helicopter fly over, build a three-dimensional
map of the area, and send it back to the
robot on the ground,” Coates describes. The
robot on the ground could then figure out
how to get to its goal from that data.
The breadth of the applications of these
learning algorithms is only limited by our
willingness to teach our skills to computers.
This research demonstrates that computers
can learn to carry out extremely difficult
“thought” processes, so it is possible that
nearly computers will eventually be able to
mimic anything humans can accomplish.
The Future of Computer
Learning
While this project has reached (and, in fact,
surpassed) its goals, the field of computer
learning is still young and there is much to
be done. It is not difficult to imagine several
scenarios where machine learning would
be beneficial. Robotics, ground and air
vehicle control, and manufacturing, among
others, are suitable areas for applying
learning algorithms that would be greatly
transformed by such concepts.
However, one important question remains:
how far will we go in granting computers
the ability to think, learn and perhaps
eventually feel? At this point, since we are
far from mimicking human abilities, such
considerations are not paramount, and
research continues unabated. In the future,
as these algorithms become more complex
and gain broader abilities, we may need to
be more cautious with our research. As for
now, we can commend the magnificence of
the work of Ng’s team and eagerly wait for
what is next.
To Learn More
For more information, visit the website of
Credit: sxc.hu
Stanford Autonomous Helicopter at
heli.stanford.edu