u

e
an art anthology

no. 75 (XXXVIII), 2017

Cracow - Vienna
Conrad Rudolph, Amit Roy-Chowdhury,
Ramya Srinivasan and Jeanette Kohl

FACES: Faces, Art, and

Computerized Evaluation Systems
-A Feasibility Study of the
Application of Face Recognition
Technology to Works of Portrait Art

Introduction

On 11 June 1685, James Scott, duke of Monmouth, first duke of

Buccleuch, and illegitimate son of Charles II, returned to Eng­
land from Amsterdam as the self-proclaimed head of the anti­
Catholic party with the intention of seizing the throne from his
uncle, James II, the last Catholic monarch of Britain. Defeated
at the battle of Sedgemoor and, two days later, dragged from
a ditch where he had been hiding, Monmouth was condemned
to death by the king and beheaded on Tower Hill in London,
the executioner requiring at least five strokes to finish the task.
A very unusual painting that may be connected with these
events is held by the National Portrait Gallery in London: a por­
trait, unquestionably, but one of a relatively young man lying in
bed with his head on a pillow, eyes closed, and blanket and
sheet pulled up under the chin, but with what seems to be anoth­
er small piece of white cloth between the sheet and the man's
throat [Fig. 1) .1 Some think that the figure represents Monmouth
-with his head put back in place, the severing perhaps wrapped
by the small piece of white cloth, lying in state, as it were. Others
think that the image does not depict Monmouth. One way or the
other, given the particular associations of the painting with Mon­
mouth and the importance of the events involved, to determine
who this person was would potentially be more than a simple
restoration of the identity of the subject. It would potentially be
 Conrad R udolph, Amit R oy-Chowdhury, Ramya Srinivasan, and Jeanette Kohl

1. Artist unknown, identity of sitter in question, c. 1640s, London, National Portrait Gallery, NPG 1566. Photo: National Portrait Gallery

266
 FACES: Faces, Art, and Computerized Evaluation Systems- A Feasibility Study of the Application of Face Recognition Technology...
2. The FACES anthropometric distances
the restoration of a unique 'document' in the history of a forma­
tive period of a great nation, one that, in its own way, might be
seen as a commentary of sorts on the larger events of which its
depiction may be a part.
The FACES Algorithm and Initial Parameters
Before the nineteenth century, most portraits were, almost by
definition, depictions of people who were important in their own
worlds. But, as a walk through almost any major art museum will
show, a large number of these portraits from before the nine­
teenth century have lost the identities of their subjects through
the fortunes of time. Traditionally, identification of many of these
portraits has been limited to often quite variable personal opin­
ion. FACES (Faces, Art, and Computerized Evaluation Systems)
proposes to establish the initial parameters of the application
of face recognition technology to the probable identification of
works of portrait art- this highly subjective aspect of art history
- while at the same time retaining the human eye as the final
2
arbiter.
267
3. The FACES local features
In the application of this technology to actual (that is, pho­
tographed) human faces, a number of difficulties are inherent in
a real or perceived alteration of appearance of the face through
variations in facial expression, age, facial hair, angle of pose,
lighting, and the very identity of the subject. With portraiture in
sculpture, painting, and drawing, not only do all the problems
that apply to photographed subjects pertain but these works of
art also have their own additional challenges. Most notably, por­
trait art does not provide what might be called a photographic
likeness but rather one that goes through a process of visual
interpretation on the part of the artist.
In the hope of addressing this challenge, the FACES re­
search team was formed in 2011 , seeking and receiving two
grants from the National Endowment for the Humanities, grants
whose two years of support (2012-2014) allowed us to estab­
lish proof of concept of our project. In general, what this meant
was identifying the issues, establishing the basic methodology,
applying the FACES algorithm that is the core of this methodol­
ogy to a particular set of paradigms to establish the initial pa­
rameters of the technology (we use the term paradigm here to
mean a logically chosen body of related images directed toward
 Conrad Rudolph, Amit Roy-Chowdhury, Ramya Srinivasan, and Jeanette Kohl

a particular demonstrative end), and applying the FACES algo­ Practically speaking, this meant that they had to be from a his­
rithm to a body of chosen 'identifications'. torical period marked by its attention to naturalistic represen­
Identifying the issues was and is an ongoing process. tation and, in general, by artists known for such attention. We
The problems already mentioned that arise in traditional face found that portraits from Western Europe, fifteenth to early eight­
recognition applications (facial expression, age, facial hair, angle eenth century, suited our purposes, the images being selected
of pose, and so on) continue to be a challenge, though it does with as much control over variables as possible. The paradigms
seem that they will eventually be resolved. But, aside from this, themselves were conceived of as addressing a broad spectrum
it was immediately apparent that there was a lack of sufficient of issues, there being ultimately around seventy paradigms,
training data, that is, a lack of an established data set of images though many of these addressed identical issues for confirma­
(works of portrait art). Such a data set is used to establish stand­ tion purposes.
ards of probable match or non-match, the larger the data set, In regard to methodology more narrowly speaking, there
the greater the accuracy of the testing, such a lack being some­ are a number of analytical methods used in face recognition
thing that is not the case with most mainstream face recognition practices, all of which are not equally suitable in all contexts. Ini­
applications today. Another issue was the role of the style of the tial research indicated that, given the particular demands of this
individual artist in the creation of a portrait. But the key issue in first general study of the application of face recognition technol­
restoring lost identities to works of portrait art remained the initial ogy to portrait art, two key methods work best: the computation
one: how to test one portrait against another, both of which had of anthropometric distances and of local features. With anthro­
been subject to the subjectivity of artistic interpretation. pometric distances [Fig. 2], a network of very fine distances,
As to establishing the basic methodology, to begin with, giv­ both horizontal and vertical, are measured throughout the face.
en that our objective was to establish the initial parameters of the Through painstaking experimental trial and error, we established
application of face recognition technology to works of portrait a feature set (the most effective body of identifying features) of
art, it was important that the images that made up these para­ eleven possible measurements, such as the width between the
digms that were the basis of our research be visually suitable. eyes and the width of the mouth, in this way quantifying these
3
aspects of the essential structure of a given individual face.
With local features [Fig. 3], the characteristics of the edges of
features such as the corners of the eyes and the corners of the
Portrait Feature Space (PFS) - Match 4
mouth are assessed. Here, we identified a feature set of twen­
-Non-match
ty-two different features or landmark points, as they are called.
••• Standard match deviation
••• Standard non-match deviation These were marked on a generic face mesh that had been 'reg­
10 istered' onto the image being studied and the landmark points
9 were then quantified through the use of Gabor filters (a tech­
til
� 8 nique that analyzes edge gradients of these landmark points at
0
:g eight orientations along five scales, comprising a set with a total
:g 7
iii of forty different readings at each point).
:0 6
.c: Carefully selecting a body of images to form a data set, we
0
1ii 5 used this method ('feature extraction') to analyze several hun­
E
c dred portraits, among which some were known to be 'match'
0 4
e
"0
� images (known to share a common identity) and others known
ca 3
.c:
0
to be 'non-match' (known to not share a common identity).
1ii 2
:; The anthropometric distances and local features of each image
were computed and, when calculated for match images and

0 ����������� non-match, gave a number on a scale from 0 to 1 -the similarity

�������
0.4 0.5 0.6 0.7 0.8 0. 9 1.0 score - analogous to a percentage. The distribution of similarity

Similarity score
4. The FACES portrait feature space (the distribution of match
and non-match scores). Additionally, the similarity score line is
marked with the result of the test between the unknown portrait
sometimes said to be of Monmouth [Fig. 1] and a known
portrait of Monmouth [Fig. 5]
scores (match and non-match scores) that was produced from
this body of portraits is known as the 'portrait feature space'.
On the portrait feature space graph [Fig. 4], which gives the dis­
tribution of similarity scores, the blue curve represents the distri­
bution of match scores (images of known identities), and the red
curve indicates the distribution of non-match scores (images of
unknown identities). The area between the blue dotted lines de­
notes the standard distribution of match scores; that is, the range

268
FACES: Faces, Art, and Computerized Evaluation Systems -A Feasibility Study of the Application of Face Recognition Technology...

5. Studio of Godfrey Kneller, «James Scott, Duke of Monmouth and Buccleuch», 1678, London, National Portrait Gallery,
NPG 5225. Photo: National Portrait Gallery

269
 Conrad R udolph, Amit R oy-Chowdhury, Ramya Srinivasan, and Jeanette
where most of the match score images are found (there could be
match scores that are higher or lower). Conversely, the area be­
tween the red dotted lines represents the standard distribution
for non-match scores (again, there could be non-match scores
that are higher or lower). Because the distribution of scores is
based upon a specific body of images, it will vary as the body
of images varies.
With the portrait feature space established, let me contin­
ue to describe our methodology as if two portraits were being
tested for similarity using the FACES algorithm: one, the test
image, a portrait whose identity is uncertain [Fig. 1], and the
other, the reference image, a portrait known to be of a particu­
lar individual [Fig. 55]. The portrait whose identity is uncertain
happened, in this case, to be by an unknown artist, and under­
went feature extraction following the method described above.
But the portrait whose identity is certain was by a known artist,
Godfrey Kneller, and with a work by a known artist the element
of style enters into in the analytical process. By style, we simply
follow Meyer Schapiro's understanding of style as 'the constant
form[... ] in the art of an individual or group'.6
Anthropometric distances and local features are part of the
body of characteristics that make an individual face individual,
and so they vary from face to face. At the same time, certain of
these features have an 'invariant' quality with individual artists.
That is, while some of them vary and so could contribute to the
distinctive appearance of the sitters depicted by an individual
artist, others tend to not vary and are part of the dynamic of the
style of the individual artist. In order to model- that is, to teach
the computer - the individual style of the artist, information
(the anthropometric distances and local features) is extracted
from a body of portraits of different sitters by the same artist.
The information is then analyzed through a procedure known
as random subspace ensemble classification. This procedure
repeatedly selects random subsets of these extracted features
from the body of portraits, ranking them in order of importance
from 'more invariant' to 'less invariant', potentially including fea­
tures that might not be detectable to the human eye. In this,
random subspace ensemble classification strengthens the anal­
ysis of the information, giving a higher level of confidence in the
results of the analysis by allowing us to select different subsets
of the features in a probabilistic _way and see which one is the
most effective?
With both images having undergone feature extraction -
and with the image that was able to having undergone style
modeling - the test image (the portrait whose identity is un­
known, Fig. 1) was now tested against the reference image (the
portrait whose identity is known, Fig. 5), using a statistical hy­
pothesis test known as a Z-test. This determines whether the
pair is a match or non-match (or if it is a 'no decision') by meas­
uring the deviation of the similarity score from the mean value of
the distribution under consideration in order to obtain a number
270
Kohl
called p-value. This p-value is then subjected to well-known
statistical tests to arrive at the decision. If the similarity score
from the test lies within the standard match distribution (that
is, between the blue dotted lines on the portrait feature space
graph, Fig. 4), the test indicates the probability - not the cer­
tainty- of a match. Similarly, if the similarity score lies within the
standard non-match distribution (between the red dotted lines),
it suggests the probability- not the certainty- of a non-match,
granted that the images that were tested were relatively true like­
nesses and that there were no strongly mitigating factors (such
as too great of an age difference, a face too strongly angled to
provide sufficient information, etc). If the similarity score is more
or less equally distant from the match and non-match distribu­
tions (that is, between the innermost red and blue dotted lines
on the portrait feature space graph), the result is a 'no decision',
meaning that the probability of a match or non-match is not sta­
tistically conclusive.8
The process just described is the FACES algorithm.
As to the results of the two portraits hypothetically being an­
alyzed with the FACES algorithm, when the portrait of uncertain
identity thought by some to represent Monmouth [Fig. 1] was
tested against a portrait known to represent Monmouth[Fig. 5],
we received a similarity score of 0.7204 (we also tested the por­
trait of uncertain identity against another portrait known to be of
Monmouth, Fig. 69, receiving a similarity score of 0.69188). The
similarity score of 0. 7204, represented by the dot on the simi­
larity score line in Fig. 4, is one that is solidly within the match
range (as is the score of 0.69188). And so, does the portrait of
the deceased man represent Monmouth?
In the context of face recognition technology, this question
is more appropriately phrased as: how might the results of this
test be used in an art historical understanding of this portrait?
And the answer to that question is something that will be better
understood after a fuller discussion of FACES.
Another aspect in establishing proof of concept was the
application of the developing FACES algorithm to a particular
set of paradigms in order to establish the initial parameters of
this technology to works of portrait art. While these paradigms
were actually tested without any concern for order, they will
be presented in this paper in the same logical sequence with
which FACES was originally conceived so as to guide the reader
through that same thought process.
The essential unknown in this project is the relation of the
element of artistic creativity or artistic interpretation to the work
of portrait art. Wishing to test the feasibility of the application of
face recognition technology to portraiture in the most straight­
forward way,_ we tested an analogue- an unmediated image of
the subject, not a work of art - against the image of a three­
dimensional work of art that, in this case, physically approaches
the subject in form and size but that nevertheless partakes of the
subjectivity of artistic interpretation. And so we tested a cast of
FACES: Faces, Art, and Computerized Evaluation Systems -A Feasibility Study of the Application of Face Recqgnition Technology...

6. Portrait possibly after William Wissing, «James Scott, Duke of Monmouth and Buccleuch», c. 1683, London,
National Portrait Gallery, NPG 151. Photo: National Portrait Gallery

271
 Conrad R udolph, Amit R oy-Chowdhury, Ramya Sr inivasan, and Jeanette Kohl
7. Possibly by Orsino Benintendi, «Lorenzo de' Medici», death
mask cast (originally gilded), 1492, Florence, Accademia
Toscana di Scienze e Lettere 'La Colombaria', on loan to the
Museo degli Argenti. Photo: Museo degli Argenti
the death mask of a known individual against an identified sculp­
0
tural portrait of the same individual: Lorenzo de' Medici [Fig. 71 ,
Fig. 811]. In this, we received a match score of 0.7114, a rather
strong score and one that indicates that a particular work of
portrait art could in fact carry what might be called an objective
set of facial signifiers (the artistic rendering of anthropometric
distances and local features) while at the same time being sub­
ject to the subjectivity of artistic interpretation - all of this being
dependent, of course, upon the method, ability, and intentions
of the individual artist.
Leaving the relative security of the analogue and work of art
pairing, we tested a paradigm of exclusively three-dimensional
works of art; that is, we tested two works both of which were now
subject to the subjectivity of artistic interpretation. More specifi­
cally, we tested a sculptural portrait of a known individual, Pope
272
Urban VIII, against another sculptural portrait of the same indi­
vidual [Fig. 912, Fig. 1013], both around the same stage of the
individual's life and both depicted by the same artist, Gian Lor­
enzo Bernini. In other words, we proceeded with as much con­
trol over variables as possible, in the process obtaining a very
strong match score of 0. 7572. Although the similarity between
these two portraits might seem 'self-evident', to quote one 'sup­
portive critic' of FACES, the point is not that such a test can
tell us what we already know but that the technology can work,
that it can provide an objective, quantified basis for visual rela­
tionships between two works of art - and this is something that
is best done with images whose resemblance is not already in
question.
Incrementally, we broadened our tests- too involved to fully
detail here - introducing a similarly controlled but wide-ranging
series of systematically chosen variations extending from more
controlled paradigms to less controlled (that is, more challeng­
ing) ones. These included the same stage of an individual's life
but by different artists (for example, Cardinal Scipione Borghese
by both Bernini and Giuliano Finelli; strong match score of
0.7324)14, different stages of an individual's life by the same art­
ist (Urban VIII, although only ten years apart, by Bernini; very
solid match score of 0.7003)15, and different stages of an indi­
vidual's life by different artists (Pope Alexander VII, once more
only ten years apart, by Bernini and Melchiorre Cafa; a solid
match score of 0.6989)16 - all in three-dimensional imagery.
Again, these tests do more than confirm what might seem to
be self-evident. They demonstrate that this technology has the
ability - despite the basic similarity of most human faces - to
recognize the same subject even when rendered by different art­
ists, each no doubt very conscious of his own personal style. Put
another way, they demonstrate that a particular body of measur­
able features may at times be retained by different artists in their
depictions of the same sitter. And they show that the technology
can accept at least moderate age difference - not only in works
by the same artist but, which is more challenging, by different
artists.17 Together, these tests tentatively attest to the general ac­
ceptance by this technology of the normal variables that occur
in the genre of portraiture and suggest a strong foundation for
further development.
Expanding our examination, we tested two-dimensional
imagery (painting and drawing), first simply considering two
two-dimensional images of the same subject by the same artist
(self-portraits by Bernini; 0.7342, a strong match score)18, and
then mixing media by testing a number of sculpture vs. painting
(that is, three-dimensional vs. two-dimensional) paradigms (for
example, a sculpted bust and a painting of Urban VIII, both by
Bernini; 0.7104, a solid match score).19 In this, we employed
a systematic series of distinctions similar to those already men­
tioned (same artists, different artists, same ages, different ages,
and so on). The testing of two-dimensional imagery against
 FACES: Faces, Art, and Computerized Evaluation Systems -A Feasibility Study of the Application of Face Recognition Technology. . .

8. Probably after a model by

Andrea del Verrocchio and
Orsino Benintendi, «Lorenzo
de' Medici», probably 1478-1521,
Washington, National Gallery
of Art. Photo: National Gallery
of Art

two-dimensional imagery is straightforward enough: images
of the same medium tested against each other. But the testing
of three-dimensional against two-dimensional imagery raised
other questions, at least initially. For while photography ren­
ders a three-dimensional subject into a two-dimensional image
automatically, through technology - whether a photographed
human face or a photographed work of sculpture- painting and
drawing undertake the same process of transformation of the
three-dimensional into the two through the thought process of
the artist. Something that might potentially, at least, create chal­
lenges for facial recognition technology- and something that we
had to be aware of- our tests found that this was not an issue.

273
 Conrad Rudolph, Amit Roy-Chowdhury, Ramya Srinivasan, and Jeanette Kohl

9. Gian Lorenzo Bernini, «Pope Urban VIII», c. 1632, Rome, Galleria Nazionale d'Arte Antica, Palazzo Barberini. Photo: Art Resource

274
FACES: Faces, Art, and Computerized Evaluation Systems- A Feasibility Study of the Application of Face Recognition Technology...

10. Gian Lorenzo Bernini, «Pope Urban VIII», c. 1632, Ottawa, National Gallery of Canada. Photo: National Gallery of Canada

275
 Conrad Rudolph, Amit Roy-Chowdhury, Ramya Srinivasan, and Jeanette Kohl

11. Francesco Laurana, «Battista Sforza», c. 1474, Florence, Museo Nazionale del Bargello. Photo: Art Resource

276
 FACES: Faces, Art, and Computerized Evaluation Systems -A Feasibility Study of the Application of Face Recognition Technology...

Identifications with FACES

The final aspect that concerned us in establish­

ing proof of concept was the application of the
developing FACES algorithm to a series of inter­
esting and sometimes important 'identifications'.
In this, we do not claim that we are providing the
incontestable identity of the sitter in question,
but only that the similarity score of a given pair of
portraits does or does not fall within the distribu­
tion of match or non-match scores of the portrait
feature space. Let us run through a number of
our identification paradigms, sometimes merely
as examples of the potential of FACES, some­
times drawing attention to various strengths or
weaknesses.
For some time, art historians have felt that
the posthumous bust of Battista Sforza by Fran­
cesco Laurana in the Bargello (c. 1474; Fig. 1120)
was modeled after a death mask cast also by
Laurana now in the Louvre (1472; Fig. 1221). The
similarity score of 0.7116 from our test indicates
the strong probability of a match, regardless of
the obvious challenges to such a comparison.
This suggests that, as was thought, the mask
was taken from Battista, and supports as well
the idea that the cast was quite closely followed
by Laurana as a model - rather than, say, fol-
lowing Piero della Francesca's profile portrait of 12. Francesco Laurana, «Battista Sforza» (?), death mask cast, c. 1472, Paris,
Battista, though it or other images most certainly Musee du Louvre, RF 1171. Photo: Art Resource
may have been referred to by the artist in the
course of his work.22 A score strongly within the
match range of 0.727 was also returned for Botticelli's Portrait of around 1590, possibly shortly after Galileo had been offered
a Lady at the Window (c. 1475; widely thought to be a render­ a chair in mathematics at the University of Pisa [Fig. 1526]. When
ing of Smeralda Brandini; Fig. 1323) and Verrocchio's Lady with tested against a chronological spectrum of eight other known
Flowers (c. 1475; Fig. 1424), the two portraits sometimes being portraits of Galileo, the results neatly broke down into consist­
suggested by some to represent the same sitter, thus lending ently decreasing similarity scores: first, within the match range
objective support to this position despite the two distinctly dif­ for the chronologically three closest likenesses (1601-c. 1612);
ferent personas conveyed in the images. Since neither portrait then, within the 'no decision' range for the next two (1624); and,
has a certain identity, the probable match is relative rather than finally, within the non-match range for the final three (1635-
absolute, but a probable match nonetheless. Perhaps of a little 1640).27 While age remains a challenge for FACES and requires
less prominence but of no less interest is the portrait by Michiel more research (this is also the case with photographed human
van Musscher of his family, where a similarity score of 0.708 faces), age differences of around ten years or so have not been
was given between a person who is obviously the artist's wife - too much of an obstacle. If the test image here does in fact date
thought by some to depict the artist's first wife, Eva Visscher, and from around 1590 and does depict Galileo, this body of portraits
by others his second wife - and a known portrait of Eva by him, would seem to represent something of an exception in the age
an identification that can affect the immediate understanding of question in that strong similarity scores were returned for por­
the painting.25 traits with an approximately twenty-four year age difference.28
In a somewhat different vein is the portrait, sent to us by the (A particular body of portraits of Isaac Newton shows an even
Italian astronomer Paolo Molaro, of what he believes may be broader spectrum, although it is not so consistent in its scores
the earliest known likeness of Galileo Galilei, painted perhaps as the Galileo paradigm).29

277
C onrad R ud olph, Amit R oy-Chowdhury, Ramya Srinivasan, and Jeanette Kohl

13. Botticelli, «Portrait of a Lady

at the Window (Smeralda
Brandini?)», c. 1470-1475,
London, Victoria and Albert
Museum. Photo: Victoria and
Albert Museum

278
FACES: Faces, Art, and Computerized Evaluation Systems -A Feasibility Study of the Application of Face Recognition Technology...

14. Verrocchio, «Lady with Flowers», c. 1470-1475, Florence, Museo Nazionale del Bargello. Photo: Art Resource

279
 Conrad Rudolph, Amit Roy-Chowdhury, Ramya Srinivasan, and Jeanette Kohl

15. Artist unknown, «Galileo Galilei» {?),c. 1590, Trieste, private collection

280
 FACES: Faces, Art, and Computerized Evaluation Systems- A Feasibility Study of the Application of Face Recognition Technology...
16. Nicholas Hilliard, «Young Man Among Roses», c. 1587, London,
Victoria and Albert Museum, P.163-191 0. Photo: Victoria and
Albert Museum
One paradigm that traverses some of the potentially
complex terrain that one might encounter in the use of this
technology is that of Nicholas Hilliard's Young Man Among Ros­
es (c. 1588; Fig. 16), said by Roy Strong to be 'perhaps the
most famous miniature ever painted', and generally thought to
281
30
represent Robert Devereux, second earl of Essex. Essex's like­
ness has survived in a fairly reasonable number of paintings and
miniatures. One challenge in identifying the sitter of Young Man
Among Roses as Essex is that Essex grew a beard around 1596,
not only obscuring his facial features in portraiture with regard to
traditional identification but also making it more difficult for the
effective use of face recognition technology. Among the body of
portraits of Essex, curator Elena Greer feels that the most useful
one for comparison with Young Man Among Roses is the depic­
tion of Essex by William Segar of around 1590, which is around
the same date as Young Man Among Roses and before Essex
31
grew his beard. Indeed, when Young Man Among Roses was
tested against the Segar portrait, the score that was returned
was a rather strong 0.72. Furthermore, when tested against
the Hilliard miniature at the Metropolitan Museum in New York
(1588), believed to probably represent Essex, the result was
32
a score of 0.7. And when tested against a Hilliard miniature
at the National Galleries of Scotland in Edinburgh, thought by
some to depict Essex, the return was a score of 0.705, with this
Edinburgh miniature also receiving a score against the Segar
33
portrait of 0.71. Scores in the non-match range were returned
for all tests of Young Man Among Roses against the bearded por­
trayals of Essex, as well as for one miniature of a clean-shaven
figure generally thought to be Essex of c. 1587 attributed to the
34
studio of Hilliard. Generally speaking, while a non-match can
be the result of a number of factors aside from that of different
identities -such as issues of facial hair, poor likeness, age differ­
ence, angle view, and so on, just as with photographed human
faces - a match is unlikely to be coincidental. And when there
is a pattern of match and non-match images as we received
for this paradigm - where the beard of the sitter could easily
account for all of the non-match portraits but one - the weight
of the matches tends to suggest that the miniature thought by
some to represent a beardless or largely beardless Essex from
the studio of Hilliard is probably best understood as simply
a less true (or less conforming) likeness than the other beard­
less or largely beardless depictions, an exception that does not
tend to discredit the match scores. These tests, therefore, pro­
vide a body of quantitative evidence derived from the images
themselves that strongly supports the traditional supposition
that Young Man Among Roses represents Essex.
The results of FACES are only as dependable as the images
tested, and some paradigms gave seemingly undependable re­
sults. In a paradigm in which we tested the only known 'portrait'
35
of Anne Boleyn - the 'Moost Happi' medal - the results were
such as to suggest that the test image itself did not provide suf­
ficient portrait data for a reliable test.
A variation on this same situation at least seems to be the
case with the tangled web of portraits that different proponents
have claimed at one time or another portray William Shakespeare
36
(1564-1616). Perhaps the leading authority on the portraits of
 Conrad R udolph, Amit Roy-Chowdhury, Ramya Sr inivasan, and Jeanette Kohl

17. Cast of the sculpture, probably by

Garrat Johnson (Gheerart Janssen)
the Younger, of «William Shakespeare»,
c. 1620, Stratford, Holy Trinity Church,
cast NPG 1735. Photo: National Portrait Gallery

18. Martin Droeshout, «William Shakespeare»

(Droeshout engraving, First Folio,
first state, published 1623).
Photo: Folger Shakespeare Library

Shakespeare, Tarnya Cooper, believes that only two extant por­
traits provide us with a reasonable sense of Shakespeare's like­
ness, the Stratford bust (c. 1620, probably by Garrat Johnson
37
the Younger; Fig. 17 ) and the Droeshout engraving (published
38
1623, by Martin Droeshout; Fig. 1 8 ) Neither of these depic­
. of claimed Shakespeare portraits, a process we have seen is
tions, however, is an accomplished work of portrait art, some­
thing that does not contribute to reliable test results with FACES.
At the same time, while the Droeshout engraving was said by
Ben Johnson to represent the face of Shakespeare well, the
context of this statement- a partially cliche-based verse dedica­
tion as opposed to some less contrived discourse - suggests
that this 'opinion' need not necessarily be taken as an objective
avowal that the engraving is a true likeness as understood in
39
this study. T he concern here is not whether either of these por­
traits generally looked like Shakespeare to his contemporaries
but whether their creators were capable of rendering a sufficient
body of anthropomorphic distances and local features consist­
ent enough to allow the images to return similarity scores that
are able to support a clear match or non-match- and, beyond
that, to return such scores from any of the rest of the main body
possible in principle from earlier, 'self-evident' paradigms.
It is Cooper's opinion that the Stratford bust and Droeshout
engraving- both posthumous to Shakespeare - were based on
40
a now lost previous portrait for which Shakespeare sat. It is
also her opinion that the so-called Chandos portrait is the most
likely of all the claimed Shakespeare likenesses to have been
made in the author's lifetime (c. 1600-1610, attributed to John
41
Taylor; Fig. 19 ). But when we look at the FACES test returns
for this body of portraits, we find a welter of ambiguous 'no de­
cision' associations between the many portraits. T he problem
arises in part from both the features and the quality of many

282
 FACES: Faces, Art, and Computerized Evaluation Systems- A Feasibility Study of the Application of Face Recognition Technology...

19. Attributed to John Taylor, «William Shakespeare»(?) (Chandos portrait}, c. 1600-1610, London, National Portrait Gallery, NPG 1.
Photo: National Portrait Gallery

283
 Conrad R udolph, Amit R oy-Chowdhury, Ramya Sr inivasan, and Jeanette Kohl

20. Artist unknown, identity of sitter in question, c. 1570, London, National Portrait Gallery, NPG 96. Photo: National Portrait Gallery

284
 FACES: Faces, Art, and Computerized Evaluation Systems- A Feasibility Study of the Application of Face Recognition Technology...
of the portraits: beards and mustaches that limit information,
copies or possible copies - even copies of copies - that may
contribute to a degradation of information, different levels of ac­
complishment on the part of the various artists that potentially
limit or misrepresent information, and overpainting and cleaning
that also potentially limit or misrepresent information. But there
is more to the Gordian knot of Shakespeare portraits than this.
There is also apparently another, fundamental difficulty inherent
in this body of images. For, from the very beginning, as it were,
we see that while the Stratford bust is loosely associated through
scores in the 'no decision' range - that is, scores that could
be said to be not non-matches - with the Droeshout engrav­
ing (0.663) and with the (heavily overpainted) Chandos portrait
(0.681), it is also similarly loosely associated with the Janssen
portrait (0.643).
42 The Droeshout engraving has the same loose
'no decision' associations, but it is additionally loosely associ­
ated with the so-called Cobbe portrait of Shakespeare (0.654,
'no decision' range).43 At the same time, the Chandos portrait
follows this same pattern of initial, loose associations but now
with an increasing number of additional loose, 'no decision'
associations.44 Of all the portraits in this paradigm (in which
scores in the 'no decision' range far outnumbered those in the
non-match range), the only two to receive a score in the match
range (0.692} were the Janssen portrait and a portrait of Thom­
as Overbury (1581-1613) by Marcus Gheeraerts the Younger
(c. 1610).45 If it is true, as is generally believed, that the Strat­
ford bust and Droeshout engraving offer reasonable likenesses
of the historical figure of Shakespeare, one unavoidable if not
very satisfying conclusion seems to be - aside from the other
problems of beards, condition, quality, and so on-that the intro­
duction of the very similar physiognomy of at least one different
individual (possibly Thomas Overbury via the Janssen portrait,
as has been previously suggested46) into the body of claimed
Shakespeare portraits is compounding the lack of clarity of this
already challenging body of images.47 Even so, the large num­
ber of scores in the 'no decision' range - which indicate that
there are some similarities but not enough to meet generally
recognized standards - suggests that, as this technology ad­
vances, future research on this and perhaps a broader body of
portraits may very well contribute to a more secure sense of the
likeness of the greatest writer in the English language.
Finally, the relatively recent exhibition at the National Portrait
Gallery in London, Imagined Lives: Portraits of Unknown People,
in which a number of portraits whose identities have been con­
tested over the years were revisited, suggested to us that we
look at two of the more interesting images.48
The first was the portrait of a woman thought at one time to
represent Mary Queen of Scots, at another to depict her mother,
Mary of Lorain, and currently simply to be unidentified (c. 1570;
Fig. 2049). When tested against eight other portraits known to be
of Mary - including six by Frangois Clouet and one by Hilliard
285
- the results were almost startling, showing four scores in the
match range (three Clouets and the Hilliard) and four in the 'no
decision' range, some of the latter being quite close to the match
range (including three Clouets)- all this despite the differences
in style in the representation of the face.50 (The score in the non­
match range was with her tomb sculpture in Westminster Abbey.)
The second portrait was that thought by some to be of
Monmouth with which this paper opened and which the exhibi­
tion described as of an unknown man [Fig. 1]. As mentioned,
when tested against two portraits known to represent Monmouth
[Figs 5, 6], it received scores in the match range of 0.7204 and
0.6918 (it also received a score in the 'no decision' range of
0.6772 when tested against a relatively unaccomplished mezzo­
tint copy of an earlier portrait51). While these are strong scores,
the eyes of the deceased man are closed and the face is at an
angle, both of which reduce the amount of information currently
computable by the FACES algorithm.52
And so, does this portrait represent Monmouth -do any of
the portraits that received scores supporting a match represent
the individuals against whose known portraits they were tested?
FACES: Concluding Remarks
A great portrait is more than the sum of its parts and certainly
cannot be reduced to a discrete series of objective factors that
are readily subject to the 'superior' discrimination of the com­
puter. However, setting aside what in this case might rightly or
wrongly be called 'extraneous' elements- elements sometimes
associated with the portraiture of a particular artist, including the
general composition, setting, pose, facial expression, lighting,
skin treatment, hair, drapery, finish, and so on - setting aside
these elements, the face remains. Certainly more elusive than
the other elements, the face can nevertheless still be quantified
in a way not so dissimilar to the traditional canons employed
by artists in the depiction of the human being from the ancient
Egyptians to the Classical Greeks to the artists of the Renais­
sance. With this understanding, at the end of two years of re­
search, we have established proof of concept of the application
of face recognition technology to works of portrait art. This in­
cludes identifying the issues, establishing the basic methodolo­
gy, applying the FACES algorithm (the core of this methodology)
to a carefully chosen set of paradigms in order to determine the
initial parameters of the technology, and applying the FACES
algorithm to a body of selected 'identifications'. In these first
steps, there is no more reason to expect completion or perfec­
tion than there is with any other new digitally based technology,
all of which typically go through long periods of development.
As with seemingly everything digital, this is a work in progress,
a procedure that will certainly be refined in probably every area,
including the most challenging.
 Conrad Rudolph, Amit Roy-Chowdhury, Ramya Srinivasan, and Jeanette Kohl
The FACES algorithm does not operate like the make-be­
lieve computer processes a person sometimes sees in popular
media. Nor is it for every portrait. As with photographed human
faces, works of portrait art are subject to variations in facial ex­
pression, age, facial hair, angle of pose, lighting, and the very
identity of the images tested.
However, unlike photographed human faces, works of por­
trait art are also subject to the subjectivity of artistic interpreta­
tion, whether simply from issues of style or from something more
pronounced, such as a consciously more flattering rendering of
the sitter (queens, in particular, appear to be a productive area
of future research in this regard53). But there are also other fac­
tors involved, including artistic quality, with renderings of less
mimetic accomplishment affecting the readings of portraits no
less in original images than in copies, which sometimes, though
not always, are susceptible to distinct degradation of detail.54
Post-production interventions (restoration, overpainting, exces­
sive cleaning, repairs, etc.) are a constant and serious concern.
And it may even be that, on occasion, the use of different media
- terracotta as opposed to marble, chalk in contrast to oil, for ex­
ample- which human perception reads as more or less identical
are read more precisely by these tests (apparently because of
the sometimes greater exactitude of particular media in detailing
local features), this also being an area that might benefit from
future research.55 Finally, the human element in the processing
of these tests may also sometimes result in very minor variations
in scores due to extremely slight differences in the registering of
points of measurement on the facial meshes.
In all this, we recognize that the style of the individual artist
is not a simple issue, being complicated by any number of fac­
tors from the work of contemporary assistants to interventions of
later restorers, and future work in this area will undoubtedly take
the subject further. But, at this point, the test results suggest that
local features carry a distinctly greater invariant quality than do
anthropomorphic distances. It would seem that these particular
patterns of invariant features for individual artists are part of the
dynamic of the style of an individual artist. In our work on this,
which amounts to only the first steps in approaching a very com­
plex subject, we emphatically do not claim to identify the style
of individual artists. But we are able to identify certain patterns
that appear in the particular bodies of portraits that we tested,
patterns that could be of great interest to scholars who would
like to know more about the different ways- now tending toward
'invariants', as it were, now 'variants' - in which these artists ap­
proached their various subjects [cf. Fig. 21].
The use of the FACES algorithm should not be thought of as
limited to facial recognition in the sense of identification alone.
We have already mentioned using FACES to study difference
in the depiction of various bodies of sitters, queens being one
among a number of potential subjects. And we have suggest­
ed that FACES might sometimes be of help in getting a better
286
understanding of the artistic process in cases where casts of
one kind or another were used. But the FACES algorithm could
also be applied to a study of adherence or non-adherence to
widely recognized artistic canons, formal or informal, as well as
to the identification of variations in the practice of an individual
artist: over time, with different subjects, with different genres, af­
ter exposure to external influences, and so on. It certainly might
eventually be possible to more confidently determine probable
bodies of work of anonymous artists - including in the so-called
minor arts- or to identify probable unrecognized works of known
artists. And it might be possible to study difference in larger bod­
ies of works- art historical 'big data'?- for instance, a particular
body of art from Italy during a particular time frame examined
in relation to a corresponding particular body from Northern
Europe during a corresponding time frame. The FACES team
wanted, but was unable for reasons of time, to move out of the
Western sphere: for example, to take up an unusually large body
of portraits that appear to depict the same Moche noble through­
out the course of his life (Andes, c. 100-800 AD).56 The basic
concept of FACES could even be applied in an altered form to
ornament of all kinds and especially to certain minor variations in
medieval (at least) architectural detail - by which I do not mean,
for example, different types of moldings but rather more or less
the same molding rendered in slightly different ways according
to the particular practice of the individual mason - something
that potentially might reveal a wealth of information about build­
ing processes, building history, and architectural identities in
buildings that are sometimes national treasures. Perhaps the ini­
tial impetus to FACES might be one of its greatest challenges:
working out as much as possible the identities of particular bod­
ies of miniature portraits of uncertain identity from the sixteenth
century on, portraits that were often given as part of social and
political networks and that might help recover lost networks at
different levels of society, from the local to the national to the
international. These people, many of them forgotten now, were
important in their own time. By restoring their identities we would
be doing more than just contributing to these portraits as works
of art, we would be restoring historical 'documents' that can tell
us more about our own past - just as much, if handled right,
as any historical document, the value of which is unquestioned.
The application of the FACES algorithm should be made
a part of standard curatorial practice where suitable, whether
during initial acquisition or later study. Results based on agreed
upon standards should be made available online: while an anal­
ogy between face recognition and fingerprints is imperfect,
it should be pointed out that, with such technologies, their effec­
tiveness increases exponentially with the size and organization
of the data set. The use of technology in the study of works of art
- X-rays, infrared reflectography, radio spectroscopy, chemical
analysis, and so on - is not going away. Indeed, it is becoming
an ever more firmly established part of the curatorial endeavor.
 FACES: Faces, Art, and Computerized Evaluation Systems- A Feasibility Study of the Application of Face Recognition Technology...

Characteristic anthropometric distances Characteristic local features

in decreasing order of importance: in decreasing order of importance:

Algardi: 4,8,3,7,2,11 Algardi: 8,5,18,13,6,1,2,3,14,11,19,4,17,16

Bernini: 1,10,7,5,8,6,1 Bernini: 2,3,11,8,12,6,4,20,10,15,14,1,18,17
Clouet: 5,11,2,7 Clouet: 6,3,2,19,22,14,11,9,5,8
Holbein: 2,8,11,3 Holbein: 11,8,5,7,17,22,16,19,4,1
Kneller: 2,1,8,9,10,5,4,3,6 Kneller: 5,20,2,13,4,1,9,17,10,16,18,11,19,6
Van Mierevelt: 4,6,11,7,3,1,8,9 Van Mierevelt: 2,5,8,14,21,17,9,11,13,21
Van Musscher: 9,7,3,11,2 Van Musscher: 8,15,5,3,4,1,9,11,21

List of anthropometric distances List of local features

1: distance between forehead tips 1: forehead tip (left)

2: distance between forehead center and chin bottom 2: forehead tip (right)
3: distance between nose top and bottom 3: forehead center
4: distance between points on temples 4: chin bottom
5: distance between chin ear corners 5: nose top
6: distance between points on chin 6: nose bottom
7: distance between pupils 7,8: points on temple (left, right)
8: distance between cheekbones 9,10: chin ear corners (left, right)
9: distance between mouth corners 11,12: points on chin (left, right)
10: width of nose 13,14: cheekbones (left, right)
11: distance between forehead center and nose bottom 15,16: mouth corners (left, right)
17,18: pupils (left, right)
19,20: left eye corners (left, right)
21,22: right eye corners (left, right)

21. List of FACES artists' characteristics in regard to anthropometric distances and local features

None of this technology has taken the place of the eye, knowl­
edge, or experience of the conservation scientist, curator, or art
historian. Rather, it has simply drawn attention to previously un­
noticeable or unconfirmable information by contributing catego­
ries of objective data to be used in conjunction with larger bodies
of information to which the researcher already has access.
And so, does the portrait of the deceased man represent
Monmouth? While recognizing that the angle of the face and
the closed eyes limit measurable information, the similarity score
returned for this portrait does lend new and undeniable support
to previous opinion that the figure depicts the young, self-pro­
claimed anti-Catholic rebel, in the process contributing to an un­
derstanding of this portrait as a unique image that would seem
to have served an unusual purpose at a dramatic and transitional
period in British history, the whole story told- to contemporaries,
at least-through Monmouth's pose, perhaps the small piece of
cloth at his throat, and certainly the expression on his face.
What could be more complex than the human face -the in­
dex of the mind, the window of the soul, and, at times, an impen­
etrable mask? Computers can extract features in a quantitative
way that, generally speaking, are usually only intuitively sensed
with human vision, thus providing new categories of information
that were not truly extant before for the art historian or curator to
take into consideration in their research. T hey can, sometimes,
provide an objectifying factor in otherwise often very subjective
cases. While we are insistent that this technology does not prove
the identity of its subjects - either absolutely or relatively - its
results, when probable, match what is known with a given un­
known. Typically, this is something that is unlikely to be acciden­
tal, and so the results tend to be believable. It is in this sense
that this technology, when effectively used, can not only help
solve old and vexing art historical problems but also - through
the possible restoration of lost identities-help deepen our view
of the history of a given time and place.

287
 Conrad Rudolph, Amit Roy-Chowdhury, Ramya Srinivasan, and Jeanette Kohl
FACES was conceived of at the University of California, Riverside, in the
spring of 2011, the team being composed of Conrad Rudolph (project
director). Amit Roy-Chowdhury (expert on face recognition technology),
Ramya Srinivasan (Graduate Student Researcher), and Jeanette Kohl
(an authority on portraiture and the face in art). The FACES project is
a collaboration of the humanities (art history) and the sciences (com­
puter science). This article presents FACES from the point of view of
the humanities, that is, how this technology generally works, what the
parameters of its application to portrait art are at this time, what its ad­
vantages are, and so on. For the computer science basis of the study,
see R. Srinivasan, C. Rudolph, and A. Roy-Chowdhury, 'Computerized
Face Recognition in Renaissance Portrait Art', Signal Processing Maga­
zine, 32:4, July 2015, pp. 85--94. These two papers are meant to operate
as a pair.
For their much appreciated advice, we would like to thank Lucy
Churchill, Tarnya Cooper, Anne-Lise Desmas, Christopher Donnan,
Lee Durkee, Robert Gerhardt, Bryce Kretschman, Alison Luchs, Paolo
Molaro, Beth Morrison, Stella Nair, Steven Ostrow, and especially the
National Endowment for the Humanities for the two grants that made
this project possible.
London, National Portrait Gallery, NPG 1566.Death, Passion and Poli­
tics: VanDyck's Portraits of Venetia Stanley and GeorgeDigby, ed. by
A. Sumner, London, 1995, cat. 48; National Portrait Gallery, Imagined
Lives: Portraits of Unknown People, London, 2011, p. 92.
2 To the best of our knowledge, FACES is the first systematic study of
the application of face recognition technology to works of portrait art,
although there has been a wide array of studies of art employing com­
puter analysis. For a review of the literature in general, see D. G. Stork,
'Computer Vision and Computer Graphics Analysis of Paintings and
Drawings: An Introduction to the Literature', LNCS [Lecture Notes in
Computer Science). 5702, 2009, pp. 9-24. For a study using face rec­
ognition technology in particular, C. W. Tyler, W. Smith, and D. G. Stork,
'In search of Leonardo: Computer-based Facial Image Analysis of
Renaissance Artworks for Identifying Leonardo as Subject', in Human
Vision and Electronic Imaging, ed. by B. E. Rogowitz et a/. 17, SPIE Pro­
ceedings, 8291, 2012. For an attempt at the application of this technol­
ogy to style, see L. Shamir and J. A. Tarakhovsky, 'Computer Analysis of
Art', Journal on Computing and Cultural Heritage, 5, 2012, article 7. For
a discussion of pre-digital face recognition studies, H. Hammerschmidt­
Hummel, The True Face of William Shakespeare: The Poet'sDeath Mask
and Likenesses from Three Periods of His Life, London, 2006, pp. 47-65.
For some recent work on portraiture that has been useful in this pa­
per, M. Painton, Hanging the Head: Portraiture and Social Formation in
Eighteenth-century England, New Haven, 1993; R. Brilliant, Portraiture,
London, 1990; The Image of the Individual: Portraits in the Renaissance,
ed. by N. Mann and Luke Syson, London, 1998; E. Pommier, Theories
du portrait:De Ia Renaissance aux Lumieres, Paris, 1998; Portrat, ed. by
R. Preimesberger et at., Berlin, 1999; P. Zitzlsperger, Gianlorenzo Bernini.
Die Papst- und Herrscherportrats. Zum Verhiiltnis von Bildnis und Macht,
Munich, 2002; M. Porter, Windows of the Soul: The Art of Physiognomy in
European Culture 1470-1780, Oxford, 2005; P. Rubin, Images and Iden­
tity in Fifteenth-Century Florence, New Haven, 2007; K. Christiansen and
S. Weppelmann, The Renaissance Portrait: From Donatello to Bellini,
New York, 2011 ; Similitude. Ahnlichkeit in der Kunst des Mittelalters und
der Friihen Neuzeit, ed. by J. Kohl et a/., Munich, 2012; H. Belting, Faces:
Eine Geschichte des Gesichts, Munich, 2013.
3 T he distances between forehead tips, between forehead center and
chin bottom, between nose top and bottom, between points on temples,
288
between ear corners at chin, between points on chin, between irises, be­
tween cheekbones, between mouth corners, between forehead center
and nose bottom, and the width of nose.
4 Left forehead tip, right forehead tip, forehead center, chin bottom, nose
top, nose bottom, left point on temple, right point on temple, left chin
ear corner, right chin ear corner, left point on chin, right point on chin,
left cheekbone, right cheekbone, left mouth corner, right mouth corner,
left iris, right iris, left corner of left eye, right corner of left eye, left corner
of right eye, and right corner of right eye.
5 James Scott,Duke of Monmouth and Buccleuch, studio of Godfrey Knel­
ler, 1678; London, National Portrait Gallery, NPG 5225.
6 M. Schapiro, 'Style', in Aesthetics Today, ed. by M. Philipson, Cleveland,
1961, p. 81. For a study of the quantification of style in the drawings of
Pieter Bruegel the Elder, see J. M. Hughes, D. J. Graham, and D. N. Rock­
more, 'Quantification of Artistic Style through Sparse Coding Analysis in
the Drawings of Pieter Bruegel the Elder', PNAS [Proceedings of the
National Academy of Sciences of the United States of America). 2010,
107(4). DOI:10.1073/pnas.0910530107.
7 For a more detailed discussion of this, seeR. Srinivasan, C.Rudolph, and
A. Roy-Chowdhury, 'Computerized Face Recognition in Renaissance
Portrait Art', Signal Processing Magazine, 32:4, July 2015, pp. 85-94.
For an attempt to train computers to recognize schools of art, see Shamir
and Tarakhovsky, 'Computer Analysis of Art'.
8 For a more detailed discussion of this, see Srinivasan, Rudolph, and
Roy-Chowdhury, 'Computerized Face Recognition'.
9 James Scott, Duke of Monmouth and Buccleuch, possibly after William
Wissing, c. 1683; London, National Portrait Gallery, NPG 151.
10 Lorenzo de' Medici, stucco death mask cast (originally gilded), possibly
by Orsino Benintendi, 1492; Florence, Accademia Toscana di Scienze e
Lettere 'La Colombaria', on loan to the Museo degli Argenti. K. Lange­
1h 1h Centuries, 3 vols, Florence,
dijk, The Portraits of the Medici: 15 -18
1981-1987, cat. 74:25; Christiansen and Weppelmann, The Renaissance
Portrait, cat. 56.
11 Lorenzo de' Medici, bust probably after a model by Andrea del Ver­
rocchio and Orsino Benintendi, probably 1478-1521; Washington, Na­
tional Gallery of Art. Langedijk, The Portraits of the Medici, cat. 74:28;
A. Luchs, 'Lorenzo from Life?: Renaissance Portrait Busts of Lorenzo
de' Medici', The Sculpture Journal, 4, 2000, pp. 6-23. As pointed out by
Luchs, /estoration should be taken into account in any future analysis,
something beyond the scope of this initial testing; on the restoration,
see G. Bitossi, M. Mauro, and D. Rossi, 'La maschera funebre di Lorenzo
il Magnifico: Caratterizzazione chimico-fisica della stucco originale e dei
successivi restauri', in Lo stucco: Cultura, tecnologia, conoscenza, ed.
by G. Biscontin and G. Driussi, Venice, 2001, pp. 519-526.
12 Pope Urban VIII, by Bernini, c. 1632; Rome, Galleria Nazionale d'Arte An­
tica, Palazzo Barberini. Zitzlsperger, Gianlorenzo Bernini.Die Papst- und
Herrscherportrats, pp. 80-87, cat. 12; Bernini and the Birth of Baroque
Portrait Sculpture, ed. by A. Bacchi, C. Hess, and J. Montagu, with the
assistance of A.-L. Desmas, Los Angeles, 2008, p. 288, list no. A18a.
13 Pope Urban VIII, by Bernini, c. 1632; Ottawa, National Gallery of Cana­
da. Zitzlsperger, Gianlorenzo Bernini.Die Papst- und Herrscherportrats,
pp. 80-87, cat. 8; Bernini and the Birth of Baroque Portrait Sculpture,
cat. 2.5.
14 Cardinal Scipione Borghese, by Bernini, 1632; Rome, Galleria Borghese,
CCLXVI; Bernini and the Birth of Baroque Portrait Sculpture, cat. 4.1.
 FACES: Faces, Art, and Computerized Evaluation Systems -A Feasibility Study of the Application of Face Recognition Technology...
Cardinal Scipione Borghese, by Giuliano Finelli, 1631-1632; New York,
Metropolitan Museum of Art; Bernini and the Birth of Baroque Portrait
Sculpture, cat. 5.4.
15 The Galleria Nazionale d'Arte Antica Urban VIII, see note 12. Pope Urban
VIII, by Bernini, 1640-1644; Spoleto, Cathedral; Bernini and the Birth of
Baroque Portrait Sculpture, p. 289, list no. A19.
16 Pope Alexander VII, by Bernini, 1657; private collection; Bernini and the
Birth of Baroque Portrait Sculpture, cat. 6.6. Pope Alexander VII, by Mel­
chiorre Cafa, 1667; Ariccia, Palazzo Chigi; Bernini and the Birth of Ba­
roque Portrait Sculpture, pp. 263-265.
17 For more on aging, see note 29.
18 Bernini, self-portrait, c. 1623; Rome, Galleria Borghese; Bernini and the
Birth of Baroque Portrait Sculpture, cat. 1.1. Bernini, self-portrait, c. 1625;
Oxford, Ashmolean Museum; Bernini and the Birth of Baroque Portrait
Sculpture, cat. 3.1.
19 The Ottawa Urban VIII, by Bernini; see note 13. Pope Urban VIII, by
Bernini, c. 1631-1632; Rome, Palazzo Barberini; Bernini and the Birth of
Baroque Portrait Sculpture, cat. 2.4.
20 Battista Sforza, by Francesco Laurana, c. 1474; Museo Nazionale del
Bargello, Florence. C. Damianaki, The Female Portrait Busts of Fran­
cesco Laurana, Rome, 2000, pp. 55-60.
21 Battista Sforza(?), by Francesco Laurana, c. 1472; Musee du Louvre, RF
1171. On the cast and its association with the bust of Battista Sforza, see
Damianaki, The Female Portrait Busts, pp. 61-63.
22 The Piero della Francesca portrait of Battista and the cast received
a similarity score of 0.664. While this is in the 'no decision' range, it is
a fairly high score considering that the della Francesca portrait is a pro­
file, something that FACES is not yet able to deal with fully.
The portrait busts of Brunelleschi (d. 1446) by Andrea di Lazzaro
Cavlacanti, called Buggiano (Florence, Museo del opera del Duomo),
and Giovanni Bandini (Florence, Duomo) received a strong score of
0.713 when tested against each other. However, Buggiano's 1447-1448
portrait also got a score (0.67) in the 'no decision' range when tested
against the 1446 death mask cast of Brunelleschi (Florence, Duomo),
while Bandini's late sixteenth-century rendering (sometimes said to be
idealized) obtained a score (0.702) solidly in the match range. This sug­
gests the possibility that Bandini consulted the cast over a century after
Brunelleschi's death.
Somewhat differently, the portrait of Lorenzo de' Medici of around
1553 from the workshop of Bronzino (Florence, Galleria degli Uffizi;
Langedijk, The Portraits of the Medici, cat. 74:6; Bronzino: Artist and
Poet at the Court of the Medici, ed. by C. Falciani and A. Natali, Flor­
ence, 2010, cat. 11-17), which looks as if it could have been a rendering
of the previously mentioned National Gallery of Art bust (as has been
suggested, Luchs, 'Lorenzo from Life?', p. 9; probably 1478-1521;
Fig. 8), received a strong match score of 0.7226 against the latter, tend­
ing to uphold this view. And Vasari's portrait of Lorenzo of 1533-1534
(Florence, Galleria degli Uffizi; Langedijk, The Portraits of the Medici,
cat. 74:10) - said by the artist to have been modeled on an earlier por­
trait he felt closely resembled Lorenzo - returned a score in the match
range of 0.7228 when tested against the later portrait by Bronzino, sug­
gesting that Vasari turned to a likeness that shared a basic body of facial
features with the other likenesses of Lorenzo. (The portrait of Lorenzo
by Vasari got a score (0.6747) in the 'no decision' range against the
National Gallery bust, such a score probably resulting from the extreme
angle pose - profile - of the Vasari portrait.)
289
23 Portrait of a Lady at the Window, Smeralda Brandini (?), by Botticelli,
c. 1470-1475; London, Victoria and Albert Museum. Christiansen and
Weppelmann, The Renaissance Portrait, cat. 14.
24 Lady with Flowers, by Verrocchio, c. 1470-1475; Florence, Museo
Nazionale del Bargello. A. Butterfield, The Sculptures of Andrea del
Verrocchio, New Haven, 1997, cat. 15. The Lady with Flowers is also
sometimes thought to represent Ginevra de' Benci (by Leonardo da
Vinci, c. 1474-1478; Washington National Gallery), a test that resulted
in a 'no decision' score of 0.649. On this portrait, see Butterfield The
Sculptures of Andrea del Verrocchio, pp. 90-94.
25 Family of the Artist, by Michiel van Musscher, 1694-1701; Antwerp, Royal
Museum of Fine Arts; lnv. 739. Eva Visscher, by Van Musscher, c. 1685;
Amsterdam, Rijksmuseum, SK-A-4233. The woman in the painting of
the family received two scores in the 'no decision' range (0.652 and
0.657) when tested against two other known portraits of Eva Visscher.
It is possible that the 'no decision' range score in the first pair (Interior
with Eva Visscher, by Van Musscher, c. 1683; Netherlands, private col­
lection) is because of the profile in the second portrait (this same private­
ly owned portrait received scores in the non-match range against the
Amsterdam and the Cracow likenesses, probably for the same reason;
0.63 and 0.602). In the second portrait of the second score in the 'no de­
cision' range (Eva Visscher, by Van Musscher, c. 1690; Cracow, Wawel
Royal Castle, lnv. 1126), the figure is painted with noticeably larger eyes.
26 Galileo Ga/ilei (?), artist unknown, c. 1590; Trieste, private collection.
27 Galileo Galilei (0.724, match range), engraving by Giuseppe Calendi
after a lost painting of 1601 by Santi di Tito and published 1793 in Clem­
ente de Nelli, Vita e commercia letterario di Galileo Galilei matematico e
filosofo. Galileo Galilei (0.718, match range), by Domenico Robusti (Do­
menico Tintoretto), c. 1604; Greenwich, National Maritime Museum, INV
BHC 2699. Galileo Galilei (0.708, match range), attributed to Filippo di
Nicola Furini (called Lo Sciamerone), c. 1612; Vienna, Kunsthistorisches
Museum, INV GG 7976. Galileo Galilei (0.672, 'no decision' range), by
Ottavio Leoni, 1624; Florence, Biblioteca Marucelliana, lnv. disegni H18.
Galileo Galilei (0.65, 'no decision' range), by Domenico Cresti (called
II Passignano), 1624; Helsinki, private collection. Ga/i/eo Galilei (0.639,
non-match range), by Justus Sustermans (Suttermans), 1635; Florence,
Galleria degli Uffizi, lnv. 1890 n. 745. Gali/eo Galilei (0.628, non-match
range), by Charles Mellin, 1639; private collection. Galileo Galilei (0.59,
non-match range), by Justus Sustermans (Suttermans) (?), 1640; Flor­
ence, Palazzo Pitti, Galleria Palatina lnv. 1912, n. 106.
28 This is said in reference to the Trieste and Vienna portraits.
2 9 To cite a range of results, we received one score of 0.71 in the match
range for a pair of portraits of Isaac Newton with a probable age differ­
ence of up to twenty-nine years (though scores in the broader Newton
paradigm naturally varied from portrait to portrait because, in part, of
artistic quality) (Isaac Newton, by Godfrey Kneller, 1689; Farleigh Wal­
lop, Basingstoke, Hampshire; M. Keynes, The Iconography of Sir Isaac
Newton to 1800, Woodbridge, 2005, cat. ll-1 1saac Newton, by Thomas
.
Murray, 1718; Trinity College, Cambridge; Keynes, The Iconography,
cat. VII). And for portraits by a great master of a subject he knew well,
we tested two self-portraits by Bernini of around forty-two to forty-sev­
en years apart, receiving an understandable score in the 'no decision'
range of 0.684 (Bernini, self-portrait, c. 1623; Rome, Galleria Borghese;
Bernini and the Birth of Baroque Portrait Sculpture, cat. 1.1. Bernini, self­
portrait, c. 1665-1670; Windsdr Castle, Royal Library, RL 5539; Bernini
and the Birth of Baroque Portrait Sculpture, cat. 3.13), while those of
only two years apart got an extremely high score of 0.7342 (Bernini,
 Conrad Rudolph, Amit Roy-Chowdhury, Ramya Srinivasan, and Jeanette Kohl
self-portrait, c. 1623; Rome, Galleria Borghese; Bernini and the Birth
of Baroque Portrait Sculpture, cat. 1 .1. Bernini, self-portrait, c. 1625;
Oxford, Ashmolean Museum; Bernini and the Birth of Baroque Portrait
Sculpture, cat. 3.1). The suggested crypto-portrait of Michelangelo in
the figure of Nicodemus in Caravaggio's Entombment received a score
in the 'no decision' range of 0.655 when tested against Daniele da Vol­
terra's portrait of Michelangelo, possibly explained by the difference in
age (The Entombment of Christ, by Caravaggio, 1602-1604; Vatican
Museums. Michelangelo, by Daniele da Volterra, 1548-1553; Haarlem,
Teylers Museum. On this association, see R. Preimesberger, Paragons
and Paragone, Los Angeles, 2011, pp. 1 oo-107).
30 Young Man Among Roses, by Nicholas Hilliard, c. 1587; Victoria and Al­
bert Museum, P.163-1910. R. Strong, Artists of the Tudor Court: The Por­
trait Miniature Rediscovered 152G-1620, London, 1983, p. 9. See also
R. Strong, Tudor and Jacobean Portraits, 2 vols, London, 1969, vol. I,
pp. 15-117; Strong, Artists of the Tudor Court, cat. 263; R. Strong,
The Tudor and Stuart Monarchy: Pageantry, Painting, Iconography, 3 vols,
Woodbridge, 1995-1998, vol. II, p. 219; L. Campbell et a/., Renaissance
Faces: Van Eyck to Titian, London, 2008, cat. 40.
31 Robert Devereux, Second Earl of Essex, by William Segar, 1590; Dublin,
National Gallery of Ireland, NGI.283. Greer, in Campbell, Renaissance
Faces, cat. 40. And see Strong, Artists of the Tudor Court, cat. 218.
32 Robert Devereux, Second Earl of Essex, by Nicholas Hilliard, 1588; New
York, Metropolitan Museum of Art, 35.89.4. G. Reynolds, European Min­
iatures in the Metropolitan Museum of Art, New York, 1996, cat. 8.
33 Possibly Robert Devereux, Second Earl of Essex, by Nicholas Hilliard,
c. 1588-1590; Edinburgh, National Galleries of Scotland, PG 3423.
Strong, The Tudor and Stuart Monarchy, vol. II, pp. 184-185.
34 The portraits with beards are as follows. Robert Devereux (0.57), by
Isaac Oliver, after 1596; London, National Portrait Gallery, NPG 4966.
Robert Devereux (0.61), by Marcus Gheeraerts the Younger, c. 1597;
London, National Portrait Gallery, NPG 4985. Robert Devereux (0.57),
after Marcus Gheeraerts the Younger, c. 1596; London, National Portrait
Gallery, NPG 180. Robert Devereux (0.58), by Isaac Oliver, after 1596;
London, Victoria and Albert Museum, E.1177-1988. Robert Devereux
(0.6), by Marcus Gheeraerts the Younger, 1596-1601; Greenwich, Na­
tional Maritime Museum, BHC2681. Robert Devereux (0.62), by Isaac Ol­
iver, 1596-1598; Royal Collection, RCIN 420933. Robert Devereux (0.61),
artist unknown, c. 1599; Cambridge, Trinity College, TC Oils P 53. Rob­
ert Devereux (0.62), by Marcus Gheeraerts the Younger, c. 1596-1599;
Cambridge, Trinity College; TC Oils P 52. Robert Devereux (0.63), Mar­
cus Gheeraerts the Younger, c. 1596; the Marquess of Tavistock and the
Trustees of Bedford Settled Estates, Woburn Abbey.
Without beard: Robert Devereux (0.63), attributed to the studio of
Nicholas Hilliard, c. 1587; London, National Portrait Gallery, NPG 6241.
35 'The Moost Happi' medal prototype 1534; London, British Museum.
36 In general, see Strong, Tudor and Jacobean Portraits, vol. I, pp. 276-
286; T. Cooper, Searching for Shakespeare, New Haven, 2006.
37 Cooper, Searching for Shakespeare, pp. 47-51. On the Stratford bust,
see Strong, Tudor and Jacobean Portraits, vol. I, pp. 283-286; Cooper,
Searching for Shakespeare, cat. 2.
38 On the Droeshout engraving, Strong, Tudor and Jacobean Portraits,
vol. I, p. 283; Cooper, Searching for Shakespeare, cat. 1. It has been said
by some that the so-called 'Kesselstadt Death Mask' (inscribed 1616;
Darmstadt, Universitats- und Landesbibliothek) is a cast from a death
mask of Shakespeare himself; Hammerschmidt-Hummel, The True
290
Face. We tested six prominent portraits against it said to be of Shake­
speare, receiving scores in the non-match range for five and one in the
'no decision' range. The scores in the non-match range were with the
Stratford bust (0.6322), the Droeshout engraving (0.6289), the Janssen
portrait (0.5931), the Cobbe portrait (0.601), and an X-ray of the Hamp­
ton Court Palace portrait (0.582). The 'no decision' range score (0.6522)
was with the Chandos portrait.
39 For this verse, see Cooper, Searching for Shakespeare, p. 48.
40 Cooper, Searching for Shakespeare, pp. 41-42, 48.
41 Cooper, Searching for Shakespeare, pp. 9, 42. On the Chandos portrait,
see Strong, Tudor and Jacobean Portraits, vol. I, pp. 279-283; Cooper,
Searching for Shakespeare, cat. 3.
42 Janssen portrait, artist unknown, c. 1610; Washington, Folger Shake­
speare Library, FPS 17. On the close association of the Stratford bust,
Droeshout engraving, and Chandos portrait, see Strong, Tudor and Jac­
obean Portraits, vol. I, pp. 81, 284. On the Janssen portrait, see Cooper,
Searching for Shakespeare, pp. 9, 42, cat. 6.
43 Cobbe portrait, artist unknown, c. 1610; Hatchlands Park, Surrey.
44 That is, 'no decision' range scores with the Cobbe portrait (0.662),
the Hampton Court portrait (0.673), and William Herbert, Third Earl of
Pembroke (0.67; by Paul Van Somer, 1617; Royal Collection; RCIN
405870).
45 Thomas Overbury, Oxford, Bodleian Library, LP 74. On the portraiture of
Overbury, see Strong, Tudor and Jacobean Portraits, vol. I, pp. 238-239.
46 Cooper, Searching for Shakespeare, p. 68.
47 See also Strong, Tudor and Jacobean Portraits, vol. I, p. 285, with further
bibliography.
48 3 December 2011 to 8 July 2012. See the accompanying publication,
Imagined Lives.
49 Artist unknown, c. 1570; National Portrait Gallery, London, NPG 96. On
the portraiture of Mary Queen of Scots in general, see Strong, Tudor
and Jacobean Portraits, vol. I, pp. 213-223. On this portrait in particu­
lar, Strong, Tudor and Jacobean Portraits, vol. I, pp. 217-218; Imagined
Lives, p. 92.
50 The scores in match range are as follows. Mary Queen of Scots (0. 7), by
Franc;ois Clouet, c. 1558; Royal Collection, RCIN 401229. Mary Queen
of Scots (0.7112), by Franc;ois Clouet, c. 156G-1561; Royal Collection,
RCIN 403429. Mary Queen of Scots (0.72), by Franc;ois Clouet and
workshop, c. 1555; Wroctaw, Zaktad Narodowy im. Ossolinskich. Mary
Queen of Scots (0.7), by Nicholas Hilliard, 1579; Royal Collection, RCIN
420641.
The 'no decision' scores are as follows. Mary Queen of Scots
(0.6842), by Franc;ois Clouet or Jacques Decourt, c. 1558; Paris, Biblio­
theque nationale de France, Estampes, Res. Na 22. Mary Queen of Scots
(0.6873), by Franc;ois Clouet, c. 1560; Paris, Bibliotheque nationale de
France, Estampes, Res. Na 22. Mary Queen of Scots (0.6743), artist
unknown, c. 156G-1592; London, National Portrait Gallery, NPG 1766.
Mary Stuart (and Franc;ois II of France) (0.69), workshop of Franc;ois
Clouet, c. 1558; Paris, Bibliotheque nationale de France, ms NAL 82.
51 James Scott, by Abraham Blooteling, after Sir Peter Lely, c. 166G-1690;
National Portrait Gallery, London, NPG D19810.
52 For example, the clear visual match between Bernini's bust of Fran­
cesco I d'Este (165G-1651; Galleria Estense, Modena) and the exacting
 FACES: Faces, Art, and Computerized Evaluation Systems- A Feasibility Study of the Application of Face Recognition Technology...
representation of this same bust in the late seventeenth-century painting
attributed to Francesco Stringa (Minneapolis, Minneapolis Institute of
Arts; discussed inS. Ostrow, '(Re)presenting Francesco I d'Este: AnAl­
legoricalStill Life in the Minneapolis Institute ofArts', Artibus et Historiae,
63, 2011, pp. 201-216) received a 'no decision' range score of 0.6824
apparently because of the limiting of facial information resulting from the
rendering in profile.
53 On the draft of a proclamation of 1563 (not published at the time) that,
in part, would have attempted to limit unsuitable images of Queen
Elizabeth, see T. Cooper, The Queen's Visual Presence', in Elizabeth:
The Exhibition at the National Maritime Museum, ed. by S. Doran, Lon­
don, 2003, p. 178. Flattery, of course, was not expected to be limited to
queens; for Mantegna's perceived deficiency in this regard, see Chris­
tiansen and Weppelmann, The Renaissance Portrait, pp. 5-6.
54 This seems to be the cause of the 'no decision' range score with the
Monmouth engraving mentioned above.
291
55 For a paradigm that exemplifies this issue, with two likenesses that ap­
pear to be identical to the naked eye - one of terracotta and the other of
marble - but that receives a non-match score (0.6204), ct. Muzio Fran­
gipane, by Algardi, probably c. 1635, terracotta, Bologna, Pinacoteca
(J. Montagu, Alessandro Algardi, New Haven, 1985, val. II, fig. 150); and
Muzio Frangipane, by Algardi, probably c. 1635, marble; Rome, S. Mar­
cello al Corso, Capella Frangipane (Montagu, val. II, fig. 151). And for
an interesting variation on this problem, the portrait sometimes said to
be of Mary Queen ofScots discussed above (Fig. 20; London, National
Portrait Gallery, NPG 96) received a score in the match range (0.7112)
when tested against a known oil painting of Mary by Clouet (Royal Col­
lection, RCIN 403429) but got a 'no decision' range score (0.6873) with
an otherwise visually 'identical' chalk study by Clouet for the painting
(Paris, Bibliotheque nationale de France, Res. Na 22).
56 On this, see C. B. Donnan, Moche Portraits from Ancient Peru, Austin,
2004, pp. 141-159. Our thanks and regrets to Professor Donnan.