AnalyzingMuseumVisitor sBehavior Patter ns

MassimoZancanaro2,TsviKuflik1,ZviBoger3,DinaGoren
Bar1 andDanGoldwasser1
1

TheUniversityofHaifa,MountCarmel,Haifa,31905,Israel
2
ITCirst,viaSommarive18,38050Povo,Italy
3
BenGurionUniversityoftheNegev,BeerSheva,84105,Israel
zancana@itc.ittsvikak@is.haifa.ac.ilzboger@bgu.ac.il
dgb@univ.haifa.ac.ildgoldwas@cslx.haifa.ac.il

Abstr act. Many studies haveinvestigatedpersonalizedinformationpresenta

tion in the context of mobile museum guides.Inordertoprovidesuchaser
vice,informationaboutmuseumvisitorshastobecollectedandvisitorshave
tobemonitoredandmodelledinanonintrusivemanner.Thiscanbedoneby
usingknownmuseumvisitingstylestoclassifythevisitingstyleofvisitorsas
they start their visit. Past research applied ethnographic observations of the
behaviour of visitors and qualitative analysis (mainly site studies and inter
views with staff) in several museums to define visiting styles. The current
work validates past ethnographic research by applying unsupervised learning
approaches to visitors classification. By providingquantitative empirical evi
denceforaqualitativetheoryweclaimthat,fromthepointofviewofassess
ingthesuitabilityofaqualitativetheoryinagivenscenario,thisapproachis
asvalidasamanualannotationofmuseumvisitingstyles.

1Introduction
Themuseumenvironmentisanattractivearenainwhichtodevelopandexperiment
with ambient intelligence in general and personalized information delivery in par
ticular.Manystudieshaveinvestigatedpersonalizedinformationpresentationinthe
contextofmobilemuseumguides[1].Regardingtheusercharacteristicsthatneedto
bemodelled,mostapproachesfocusonhistoryofinteractionanduserinterests.For
example, the GUIDE system presented in [2]adaptsweblikepresentationsbyadd
inginformationaboutnearbyattractionsthatmightbeinterestingforthevisitorofa
city.TheHIPPIEsystemproposespersonalizedtoursinamuseumbymaintaininga
model of user interests and knowledge [3]. The REAL system [4] adapts route de
scriptionsaccordingtotheactualuserposition,thelimitedtechnicalresourcesofthe
device,andthecognitiveresourcesoftheuser.InthecontextofthePEACHproject
[5] a spreading activation technique applied on a domain knowledgebasewas im
plemented to predict the interest in concepts related to those for which the system
receivedexplicitfeedbackfromtheuser.

Knowledgerelatedfeaturesarenot,however,theonly sourcesofinformationthat
areworthconsideringformodellingamuseumvisitor.Forexample,PetrelliandNot
[6] suggesttakingintoconsiderationwhethertheuserisvisitingthemuseumalone
orwithcompanions,whethersheisafirsttimeorarecurrentvisitor,andsoon.
Behaviouraltraitshavealsobeentakenintoconsideration.Sparacino[7]proposed
categorization of user types into three main categories: (i) the greedy visitor who
wantstoknowandseeasmuchaspossible(ii)theselectivevisitorwhospendstime
onartefactsthatrepresentcertainconceptsonlyandneglectstheothersand(iii)the
busyvisitorwhoprefersstrollingthroughthemuseuminordertogetageneralidea
oftheexhibitionwithoutspendingmuchtimeonanyexhibits. Herapplicationem
ploysBayesiannetworkstomodelboththeuser(interestandstyle)andtheappropri
atenessofthe guidescontent(lengthandorder).
ThesamecategorizationofusertypesisalsousedbyHatalaandWakkary[8]to
getherwithanontologybasedmodelofthe interests. Inboththese papers,the valid
ityofsuchaschemeisjustifiedthroughqualitativeanalysis,mainlysitestudiesand
interviewswithstaffatvariousmuseums.
Inthispaper,wewillfocuson theclassification ofthevisitingstyleproposedby
the ethno methodologists Veron and Levasseur [9]. Starting from ethnographicob
servationsofthebehaviour ofanumberofvisitorsinseveralmuseums,theyargued
that visitors movements may be compared to the behaviour of four typical ani
mals, and they proposed using this strategyas a way of classifying the style of a
visitor.Specifically,they suggeststhattheANTvisitortendstofollowaspecificpath
and spends a lot of timeobservingalmostalltheexhibitstheFISHvisitor mostof
thetimemovesaroundinthecentreoftheroomandusuallyavoidslookingatexhib
its' details the BUTTERFLY visitor does not follow a specific path but rather is
guided by the physical orientation of the exhibits and stops frequently to look for
moreinformationfinally,theGRASSHOPPERvisitorseemstohaveaspecificpref
erenceforsomepreselectedexhibitsandspendsalotoftimeobservingthemwhile
tendingtoignoretheothers.Ofcourse,itmightbeexpectedthatagivenvisitorcan
change her behaviour duringalongvisit,anditisalsopossiblethatthestyleisaf
fectedbythe specific interests.
Thefirstattempttoexploitthisclassificationaspartofausermodelforamobile
guide was in the HIPS project (see mainly[10]) where a Recurrent Artificial Net
work was trained to recognize the visiting style of a visitor given her interaction
history.Thismodelwasthenemployedforselectingandtailoringinformationtothe
visitor [11]. Although most of the ideas tested experimentally in HIPS underwent
userevaluation,theveryideaoftheexistenceofvisitingstyleswastakenforgranted
relyingonthequalitativeanalysisoftheoriginalwork.
ChittarroandIeronutti[12]employedVeronandLevasseur'sclassificationinthe
contextofatoolthatvisualizesusersbehavioursinavirtualenvironment.Theiruse
ofthevisitingstyleswasbasedonqualitativeanalysisand,again,theydidnotcon
tendtheexistence oftheseclasses.
In thispaper,wearetryingtotakeastepbackwewouldliketodiscussameth
odology forvalidatingempirically VeronandLevasseursmodelof visitingstyle.We
used log files of140visitorsexploringafrescoedroomwithamultimediamuseum

guide to provide quantitativebased evidence that museum visitors behavior may

effectively be classified according to Veron and Levasseurs model. We used two
unsupervised learning techniques (Kmeans and AutoAssociative ANN) to cluster
thevisitorsbehaviours.Theclusteringproducedbybothtechniquesmaybeassumed
to characterize Veron and Levasseurs four animals. An agreement analysis con
ductedontheclassificationsschemesdeterminedby clusteringmembershiprevealed
ahighlevelof agreementbetweenthetwotechniques.
This work is intended tocomplement Veron and Levasseur's ethnographic study
byprovidingempiricalevidencefor itaswellastoprovideinformationinaprinci
pledwayfor further researchon user modelling.Ourclaim,asdiscussedinthelast
section, is that this approach may complementif not replacereliability analysis
of observation schemes derivedfromqualitativeresearch suchasVeronandLevas
seurs.

3.DataCollectionandPreparation
Inthecontextofauserstudyofamultimediamobileguide[13],143regularvisitors
to Torre Aquila1 in Trentowere invited to test thesystem. Among the subjects, 61
were males and 82 females. Their age ranged from 20to79years(mean=47,me
dian=50, std.dev=15.9). All were recruited at the entrance of themuseum and re
ceivedafreetickettovisitthecastleasarewardforparticipatinginthedatacollec
tion.
Outofthe143visitlogs,140wereusedforthisstudytheresthadvariouserrors
that prevented their use. The average visit time was 22 minutes, and average time
spent in front an exhibitwas4minuteswithstandarddeviationof70seconds.The
system automatically logged the visitors movements in the space (by means of IR
sensors)andalltheirinteractionwiththemuseumvisitorsguide.
Sinceweareinterestedatthisstagein analyzingthevisitorsbehavior rather
than in predicting the visiting style from the interaction history, we used measures
relatingtotheentirevisitratherthantemporalbasedindices.Themeasuresusedfor
theanalysisweretheaveragetimespentateachposition,thepercentageofexhibits
visited, a numerical representation between 0 and 1 of the order of the visit, and a
combined description of visitors behavior, taking into account interaction and
whether or not visitors viewed complete presentations. Further, four cumulative
measures were defined considering the percentage of the visit for which the visitor
was:(A)interactingwiththeguide(i.e.,askingformoreinformation),butnotreach
ingtheconclusionofthepresentations(B)interactingandreachingconclusions(C)
1

Torre Aquila is a tower at the Buonconsiglio Castle in Trento, Italy where a fresco called
TheCycleoftheMonths,amasterpieceofthegothicperiod,istobefound.Thisfresco,
painted in the Fifteenth Century, covers all fours walls of a room in the tower and illus
tratestheactivitiesofaristocratsandpeasantsthroughouttheyear.Themuseumguideused
to collect visitor data is one of the many prototypes developed in thePEACHprojectfor
moredetailssee[14].

not interacting and not reaching conclusions and (D) not interacting but reaching
conclusions.
Data preprocessing generated 140 7dimensional vectors including the average
time, visit order and completeness, and the percentage of the visit for which the
visitors behavior was according to each of the four types (checking for each and
every position whether the visitor interacted with the system or not and whether
he/sheviewedcompletepresentationsornotandthencalculatingtheratios).

4.AnalysisofMuseumVisitorsBehavior
ThevisitlogsrepresentationwasusedasaninputtoanautoassociativeANNandto
aKmeansclusteringalgorithm,bothofwhichclusteredthedatainto fourclustersin
order to validate the Veron and Levasseur classification and see if their visitors
typesmightbeidentified.
4.1Unsuper visedLear ningwithAutoAssociativeANN
Artificial neuralnetworksareusedtoformdatadrivenmodels.Inordertoperform
unsupervised learning,an autoassociativeANN(AAANN),inwhichthetargets
areidenticaltotheinputs,wasused.IfthetrainedAAANNsucceedsinreplicating
the inputs as outputs, it means that the hidden neurons are encoding the essential
information distilled from the inputs features. In most cases the outputs of the
hiddenneuronsareclosetoeitheroneorzero[15].Thusallexamplesthatgenerate
thesamehiddenneuronsoutputpatternaredeemedtobelongtothesamecluster.
Asexplainedabove,thedataconsistedof140visitsummaryexampleswithseven
visit attributes named: AvT (Average time), A, B, C, D, Order (of the visit) and
Completeness (percentage of frescos visited). The AAANN used was a fully
connected,feedforwardANNoftwohiddenneuronsandsevenoutputneurons,each
having the sigmoidal transfer function, which was presented with the dataset with
the seven input variables and the identical values as targets. The input data were
preprocessedbysubtractingthemeanvalueofeachattributecolumn,anddividingby
the standard deviation of each column. These values were further rescaled to the
0.10.9rangetoserveastheAAANNtargets.ThetrainingwasdonebytheGuter
manBoger set of algorithms that starts with nonrandom connection weights and
employsproprietaryalgorithmsforavoidingentering,andescapingfrom,localmin
ima encountered during the training [16, 17]. The binary pattern of the hidden
neuronswasusedforclustering[18].Theaverageattributevaluesoftheexamplesin
each cluster were divided by the average of the attribute values of the full dataset.
TheresultsareshowninTable 1.

Table1.ANNclusteringresults

Cluster #ofcases AvT

1
53
1.22
2
36
0.93
3
15
0.74
4
36
0.84

A
0.45
0.24
0.77
2.69

B
1.81
0.42
0.18
0.72

C
0.14
0.49
5.68
0.84

D
Order Completeness
0.24 1.00
1.02
1.02
2.97 1.03
1.07 0.99
0.96
0.14 0.97
0.97

The ratios of the attributes that are higher than 1.5 are marked bold, and those
withratiossmallerthan0.5areunderlined.Itcanbeseenthatcluster#1hasahigh
ratioof the B variable, cluster # 2 has a high ratio of variable D, cluster#4hasa
highratioofvariableA,andcluster#3hasahighratioofattributeC.Theattributes
Av T, Order and Completeness apparently do not contribute much to the clusters
formation,althoughitmaybethatcluster1mayhaveasomewhathighermeanAv
T.
Table2.OnewayANOVAontheANNclusters.
ANOVA

AvgTime

BetweenGroups

Sumof
Squares
282415,7
421751,2
704166,9
6,125

136
139
3

3101,112

WithinGroups
Total
BetweenGroups

3,068
9,194
12,549
4,136
16,685
2,759

136
139
3
136
139
3

,023

WithinGroups
Total
BetweenGroups
WithinGroups
Total
BetweenGroups
WithinGroups
Total
BetweenGroups

Order

WithinGroups
Total
BetweenGroups

1,832
4,591
11,865
2,839
14,704
,066

136
139
3
135
138
3

1,955
2,020

136
139

,014

Completeness

WithinGroups
Total
BetweenGroups

,064
1,869
1,933

3
136
139

,021
,014

WithinGroups
Total

df
3

MeanSquare
94138,551

F
30,356

Sig.
,000

90,498

,000

4,183
,030

137,532

,000

,920

68,257

,000

3,955
,021

188,038

,000

,022

1,523

,211

1,551

,204

2,042

,013

Hence Cluster 1 seems to correspondtoan ANTtype(long,orderedandinterac

tive,andgetscompletepresentations)andcluster3correspondstoaFISHtype(short
visit, without getting complete presentations). Cluster 2 corresponds to a
GRASSHOPPER (tends to get more complete presentationsthan BUTTERFLY)and
cluster 4 to a BUTTERFLY (less ordered and does not get complete presentations).
After the clustering, we also used anANOVA[19]withtheclustersidentifiedasa
factorandthecumulativeindexesoutlinedaboveasdependentvariables.Significant

differences were found at p < 0.001 along all thevariablesexceptOrderandCom

pleteness.Table2summarizestheresults.
A Bonferroni [20] posthoc analysis validated the analysis of the ANN results
aboveandshowedthat:
Visitorsincluster1takemoretimethanvisitorsintheotherclusterswhen
visiting the exhibits they are less A than 4 they are moreB than all
the others they are less C than 3 and less D than 2 and 3. Therefore
visitorsinclusters1exhibitthetraitsofthevisitorsstyledefinedasANT
Visitorsincluster2takelesstimethan1butmorethan3theyarelessA
than4theyarelessBthan1and4theyarelessCthan3andmore
D than 1,3 and 4 therefore visitors in cluster 2 may be ascribed to the
visitorsstyledefinedasGRASSHOPER(i.e.,closertoanantthantoafish)
Visitorsincluster3takelesstimethan1and2theyarelessAthan4they
are less B than1and4theyaremoreCthanalltheothersandmore
Dthan1and4butlessthan2thereforetheybelongto FISH
Finally, visitors in cluster 4 take less time than 1theyaremoreAthan
alltheotherstheyarelessBthan1butmorethan2and3theyareless
C than 3 and less D than 2 and 3 theyappear to belong to the style
BUTTERFLY(i.e.,closertoafishthantoanant).
4.2Unsuper visedLear ningwithKmeans
As an alternative way of clustering the cumulative measures, we employed the K
meansalgorithm[21].
In order to reduce the number of variables, we ran a Factor analysis (Principal
Components Analysis with varimax rotation). The results showed that 81% of the
variancecanbeexplainedby4factorswhileOrderandCompletenessshowverylow
correlation with any factor. Table 3 shows the contribution of the cumulative vari
ablesonthefourprincipalfactors.
Table3.ComponentmatrixextractedbythePCA
a
Com ponentMatrix

AverageTime
A
B
C
D
Order
Completeness

1
,824
,346
,918
,541
,405
,197
,209

Component
2
3
,143
,045
,709
,580
,213
,188
,060
,622
,830
,114
,452
,487
,225
,110

ExtractionMethod:PrincipalComponentAnalysis.
a. 4componentsextracted.

4
,190
,046
,031
,354
,266
,265
,838

Weclassifythevisitorsin4clustersusingKmeansanalysisstartingfromthefac
tors.
A oneway ANOVA [19], using the cumulative indexes as dependent variables
andtheclustersdeterminedbyKmeansasafactor,showedthatforallthevariables
therearestatisticaldifferencesexceptforOrderandCompleteness(seeTable 4).
Aposthocanalysisusingthe Bonferroni[20]testshowedthat:
Visitorsincluster1takelesstimethanvisitorsincluster 2andmorethan
visitorsincluster4theyarelessAthan2lessBthan2and3lessC
than4andmoreDthanalltheothersthereforetheymaybeascribedto
thestyleof GRASSHOPPER(closerto anantthanto afish)
Visitors in cluster 2 take more time than all the others are less Athan3
are more B than 1 and 3 less C than 4 and less D than 1 and 4
thereforetheysharemanyofthetraitsof ANT
Visitorsincluster3takelesstimethan2theyarelessAthanalltheothers
moreBthan1andlessthan2lessCthan4andlessDthan1and
4 therefore they resembles visitors belonging to the style of BUTTERFLY
(closerto afishthanto anant)
Finally, visitors in cluster 4 take less time than 1 and 2theyarelessA
than3theyarelessBthan2moreCthanalltheothersandlessD
than1butmorethan2and3thereforetheycanbeclassifiedasFISH.
Table4.OnewayANOVAontheKmeansclusters
ANOVA

AvgTime

Order

Completeness

BetweenGroups
WithinGroups
Total
BetweenGroups
WithinGroups
Total
BetweenGroups
WithinGroups
Total
BetweenGroups
WithinGroups
Total
BetweenGroups
WithinGroups
Total
BetweenGroups
WithinGroups
Total
BetweenGroups
WithinGroups
Total

Sumof
Squares
321775,1
382391,8
704166,9
6,508
2,685
9,194
11,892
4,793
16,685
3,423
1,168
4,591
11,704
3,001
14,704
,084
1,936
2,020
,039
1,895
1,933

df
3
136
139
3
136
139
3
136
139
3
136
139
3
135
138
3
136
139
3
136
139

MeanSquare
107258,353
2811,705

F
38,147

Sig.
,000

2,169
,020

109,868

,000

3,964
,035

112,471

,000

1,141
,009

132,809

,000

3,901
,022

175,501

,000

,028
,014

1,962

,123

,013
,014

,923

,431

4.3Compar isonofthetwoappr oaches

In order to assess to what extent the two clusteringalgorithmsagreeonclassifica
tion of the visitors into the different visitors styles, we used the k statistics [22]
which provides a better estimation of the bare percentage agreement since it takes
intoaccountthepossibility ofchance agreement.
Table 5 shows the confusion matrix. The value of the k statistics in our case is
0.860 with a standard error of 0.035 (p<0.0001 N=140). According to Landis and
Kochscriteria[23],theagreementisverygood(k>0.8).
Table5.ConfusionmatrixfortheclassificationsbasedontheANNandKmeansclustering.
ANNLabels*KmeanLabelsCrosstabulation
Count
A
ANN
Labels

A
B
F
G

Total

50
1
0
2
53

KmeanLabels
B
F
1
0
33
2
2
12
0
36

3
17

G
2
0
1

Total
53
36
15

31
34

36
140

5Discussion,ConclusionsandFutureWork
Qualitativetheoriesfromsociologyandotherdisciplinesareoftenusedasastarting
pointfor buildingcomputationalmodelsofhumanbehaviortobeexploitedinintel
ligentsystems.Usually,ahumanexpertmanuallylabelsanumberofexamples,and
asupervisedlearningapproachisemployedtopredictinarealsituationthe behavior
ofusersaccordingtothetheory,asmodelled(orlearnedbythesystem).Inorderto
test the objectivity oftheobservationscheme,reliabilityanalysisisoftenemployed:
two or more annotators code a number of sequences, and an agreement analysis is
performedbycomputingCohensKappa(orothersimilarindexes)andby lookingat
the confusion matrix. In this paper, we tried to provide quantitative empirical evi
dence for a qualitative theory. We employed two unsupervised learning techniques
for clustering museum visitors' behavior patterns and showed how the clusters ob
tained from them may be explained in the terms ofthetheory.Weclaimthatfrom
the point of view of assessing the suitability of a qualitative theory in a given sce
nario,thisapproachisasvalidasamanualannotationwithreliability analysis.
Furthermore,thelabelsautomaticallyproducedmaythenbeusedbyasupervised
learning approach to predict the classes to which visitors belong, as they enter the
museum.Fromapragmaticpointofview,thisprocedureischeaperandlesserror
pronethan manual annotation, especially when a large corpus of data has to be
annotated.

Futureresearchwillfocusonpredictingvisitorsbehaviortypeusinginformation
collected during the first period of the visit. We intend to evaluate the correlation
betweenthedatarepresentingthecumulativedatarepresentingthewholevisitused
forclusteringinthiswork,withpartialinformationavailableatthebeginningofthe
visit of the same visitors. The results may allow us to use the clustering results as
labelsforpredictionvisitingstylewithpartialdata.
Additional future research willtrytocorrelatethecurrentclusteringresultswith
othervisitorstypesdefinition(suchasSparacinos[7]andothers).

References
1. Baus,J.andKray,C.:ASurveyofMobileGuides.WorkshoponMobileGuides.Mobile
HumanComputerInteraction'03(2003)
2. Cheverst K., Davies, N., Mitchell, K., Friday, A. and Efstratiou, C.: Developing a Con
textaware Electronic Tourist Guide: Some Issues and Experiences. The CHI 2000 Con
ferenceonHumanfactorsinComputingSystems,TheHague,Netherlands(2000)1724
3. Oppermann, R. and Specht,M.:AContextSensitiveNomadicExhibitionGuide.Inpro
ceedingsofHandheldandUbiquitousComputing:SecondInternationalSymposium,HUC
2000,Bristol,UK,September(2000)127142
4. Baus J., Krger A., Wahlster W. A resourceadaptive mobile navigation system. In Pro
ceedingsofthe7thinternationalconferenceonIntelligentUserInterfaces.SanFrancisco,
CA.(2002)
5. Kuflik,T.,Callaway,GorenBar,Rocchi,C.,Stock,O.,andZancanaro,M.:NonIntrusive
UserModellingforaMultimediaMuseumVisitorsGuideSystem.InProceedingsofUM
2005,TenthInternationalConferenceonUserModelling,Edinburgh,July(2005)
6. PetrelliD.andNot,E.'UserCentredDesignofFlexibleHypermediaforaMobileGuide:
ReflectionsontheHyperAudioExperience'.UserModelingandUserAdaptedInteraction:
The Journal of Personalization Research 15(34): (2005) 303338
http://dx.doi.org/10.1007/s1125700588161
7. Sparacino, F.: The Museum Wearable: RealTimeSensorDrivenUnderstandingofVisi
torsInterestsforPersonalizedVisuallyAugmentedMuseumExperiences.Museumsand
theWeb,Boston,Massachusetts(2002)
8. Hatala M. and Wakkary R.: OntologyBased User Modeling in an Augmented Audio
Reality System for Museums. User Modeling and UserAdapted Interaction. 15 (2005)
339380
9. VeronE.andLevasseurM.:Ethnographiedel'exposition,Paris,BibliothquePublique
d'Information,CentreGeorgesPompidou(1983)
10.MartiP.,RizzoA.,PetroniL.,TozziG.,andDiligentiM.:AdaptingtheMuseum:ANon
intrusiveUserModelingApproach.In:ProceedingsofUserModelingConferenceUM99
(1999)
11.NotE.,PetrelliD.,SariniM.,StockO.,StrapparavaC,andZancanaroM.:Hypernaviga
tioninthePhysicalSpace:AdaptingPresentationtotheUserandtotheSituationalCon
text.TheNewReviewofHypermediaandMultimedia,4(1998)3345
12.Chittaro L. Ieronutti L.: A Visual Tool for Tracing Users Behavior in Virtual Environ
ments.ProceedingsoftheWorkingConferenceonAdvancedVisualInterfaces,Gallipoli,
Italy(2004)4047

13.GorenBar D., Graziola I., Pianesi F., Zancanaro M. and Rocchi C.: Innovative Ap
proachesforEvaluatingAdaptiveMobileMuseumGuides.In:StockO.andZancanaroM.
(eds):PEACH:IntelligentInterfacesforMuseumVisits.CognitiveTechnologies.Series,
Springer,Berlin(2006)
14.StockO.andZancanaroM.:PEACH:IntelligentInterfacesforMuseumVisits.Cognitive
TechnologiesSeries,Springer,Berlin(2006)
15.BogerZ.andGuterman,H.:KnowledgeExtractionfromArtificialNeuralNetworksMod
els. ProceedingsoftheIEEEInternationalConferenceonSystemsManandCybernetics,
SMC'97,Orlando,Florida,Oct.(1997)30303035
16.Guterman, H.: Application of Principal Component Analysis to the Design of Neural
Networks.Neural,ParallelandScientificComputing2(1994)4354
17.Boger, Z.:WhoisAfraidoftheBigBadANN?Proc.Intl.JointConf.NeuralNetworks,
Honolulu(2002)20002005
18.BogerZ.:FindingPatientsClusters'AttributesbyAutoassociativeANNModeling.Proc.
Int.JointConf.NeuralNetworks,Portland,OR,(2003)26432648
19.Rutherford,A.:IntroducingAnovaandAncova:AGLMApproach,SAGE(2001)
20.Bonferroni, C. E.:Teoria statistica delle classi e calcolo delle probabilit,Pubblicazioni
delRIstitutoSuperiorediScienzeEconomicheeCommercialidiFirenze8(1936)362
21.MacQueen, J.: Some Methods for Classification and Analysis of Multivariate Observa
tions.In:Proceedingsofthe5thBerkeleySymposium(1967)281297
22.Cohen, Jacob (1960). A coefficient of agreement for nominal scales. Educational and
PsychologicalMeasurement,20,3746.
23.LandisJR,KochGG.1977.Themeasurementofobserveragreementforcategoricaldata.
Biometrics33:159174.