uslng Lnsemble MeLhods 1o lmprove

Accuracy ln Medlcal uaLa

Supervlsor ur Sald 8ahaLl Cuchanl
8evlewer ur Clle Callmo
Lxamlner ur Anders !ohnson
8ehzad Cskool
MasLer SLudenL ln uppsala unlverslLy
lall 2011
CuLllnes
Introduct|on
rob|em Descr|pt|on
1echn|ca| So|ut|on ( Methodo|ogy)
Lmp|r|ca| Lva|uat|on
Conc|us|on
lnLroducLlon
WhaL ls 8elkl?
8elkl's PlsLory
8elkl's AppllcLlons
lnLroducLlon
Chakras(rooL) 8ody SysLem
lnLroducLlon
Pow 1o erform
a 8elkl
ALLunemenL
lnLroducLlon
24channel sysLem Lo reglsLer braln slgnals
Samples' SLrucLure
19 volunLeers 12 operaLed by 8elkl pracLlLloner and 7
operaLed by a regular person
Samples have Laken ln 20 seconds wlndows
Lach sample conslsLs lnformaLlon of 19 separaLe
channels of LLC for each volunLeer
Class 1 1wo mlnuLes for 8esL (6 LLC samples)
Class 2 Seven mlnuLes for 8elkl or lacebo (21 LLC
samples
Samples sLored ln LexL flles ln Lhe followlng order
12 8elkl
7 lacebo
roblem uescrlpLlon
Samp|e' Crder
Io|unteer 1 |nformat|on for channe| 1 (6 samp|e for rest + 21 samp|e for ke|k| or |acebo)
Io|unteer 1 |nfo for channe| 2 (6+21)

Io|unteer 1 |nfo for channe| 19 (6+21)
Io|unteer 2 |nfo for channe| 1( 6+21)
Io|unteer 2 |nfo for channe| 2 (6+21)

Io|unteer 2 |nfo for channe| 19 (6+21)

Io|unteer 19 |nfo for channe| 1( 6+21)
Io|unteer 19 |nfo for channe| 2 (6+21)

Io|unteer 19 |nfo for channe| 19 (6+21)
Later the order above changed to prepare d|fferent features' order as |nput to SIM and
Lnsemb|es
roblem uescrlpLlon
ro[ecL Cvervlew MeLrlcs
1he Alm of ro[ecL ls Lo classlfy 8elkl/8esL and 8elkl/lacebo daLa whlch ls
Laken from Sahar Chobadl's MasLer Lhesls pro[ecL
We assumed LhaL hls work ls perfecL and all daLa she gaLhered are flne Lo
apply ln Lnsemble meLhods
Metr|cs
CorrecLlon ClasslflcaLlon 8aLe (1 Lrror 8aLe)
8unnlng 1lme (Seconds)
roblem uescrlpLlon
ro[ecL SLrucLure
reparlng
lnpuL vecLors
lmplemenLlng a
Slngle SvM
1o classlfy Samples
Applylng Lnsemble
MeLhods on
Samples
SLudylng effecL
of ulfferenL
kernels and
arameLers
roblem uescrlpLlon
lnpuL vecLor leaLures
1lme uomaln (4)
Mean varlance Skewness kurLosls
lrequency uomaln (3)
uelLa 1eLa Alpha1 Alpha2 8eLa
1lmelrequency (13)
A3AvL u3AvL u4AvL u3AvLA3CW u3 CW u4
CW u3 CWA3 vA8 u3 vA8 u4 vA8 u3 vA8
nonLlnear (3)
lracLal Lyapanov AnLropl
1echnlcal SoluLlon (MeLhodology)
Classlflers
SvM
Lnsemble Learnlng MeLhods
8agglng
8oosLlng (Ada8oosL)
Plerarchlcal ClasslflcaLlon
AdapLlve Comblner
MeLhods for AggregaLlng SupporL vecLor Machlnes
Sum 8ule
voLlng
Lnsemb|e Learn|ng Methods
Lnsemble learnlng ls an ldea LhaL uLlllzes mulLlple
classlflers and comblnes Lhelr declslons
1here ls no unlque caLegorlzaLlon for ensemble learnlng
A llsL of elghLeen classlfler comblnaLlon schemes
covered by !aln uuln and Mao (2000)
lour ensemble classlfler lnLroduced by WlLLen and
lrank (2000) bagglng boosLlng sLacklng and error
correcLlng ouLpuL codes
Seven meLhods of mulLlple classlflers proposed by
Alpaydln (2004) voLlng errorcorrecLlng ouLpuL codes
bagglng boosLlng mlxLures of experLs sLacked
generallzaLlon and cascadlng (24)
8agglng
8agglng ls a meLaalgorlLhm LhaL can be vlewed as a speclal case of
averaglng whlch was lnlLlally dellberaLed for classlflcaLlon and ls Lyplcally
used wlLh declslon Lree models alLhough lL can be applled Lo any varleLy
of model for classlflcaLlon or regresslon
8agglng uses sampllng and replacemenL meLhod whlch ls a Lechnlque
LhaL explolLs several verslons of a Leachlng seL by uslng Lhe booLsLrap
ulfferenL models are Lralned by uslng Lhese daLa seLs ln case of regresslon
Lhe ouLpuL of Lhe models are comblned by averaglng and ln case of
classlflcaLlon by voLlng Lo produce a slngle ouLpuL lL ls lmporLanL Lo
conslder LhaL 8agglng ls merely successful when uslng unsLable nonllnear
models (24) SvMs wldely are used Lo lmplemenL 8agglng meLhod 1o
reallze 8agglng each slngle SvM ls Lralned separaLely vla booLsLrap
meLhod by uslng selecLed Lralnlng samples aL random and Lhelr ouLpuLs
are comblned uslng an approprlaLe aggregaLlon meLhod (12)
unsLable le a small change ln Lhe Lralnlng seL can cause a ma[or change
ln Lhe model" (24)
Ada8oosL
Ada8oosL (adapLlve boosLlng) ls Lhe mosL convenLlonal
boosLlng algorlLhm 1he number of learners can be
arblLrary and also Lralnlng seLs can be small because lL
ls posslble Lo use Lhe same Lralnlng seLs several Llmes
(24) AlLhough each SvM ls Lralned uslng dlfferenL
Lralnlng samples Lhe selecLlon scheme of Lralnlng
samples ln Lhe Ada8oosL Lechnlque ls raLher dlverse
from Lhe 8agglng Lechnlque (12) Slmllar Lo 8agglng
each SvM ls Lralned by dlsslmllar Lralnlng samples as
well 8uL Lhe selecLlon meLhod of Lralnlng samples ln
Lhe Ada8oosL meLhod ls falrly dlverse from Lhe bagglng
Lechnlque
AggregaLlon MeLhods
1wo Lype of aggregaLlon meLhod exlsLs llnear and
nonllnear comblnaLlon Lechnlques
Ma[orlLy voLlng and doublelayer hlerarchlcal
meLhods are Lwo nonllnear Lechnlques LhaL can
be used Lo comblne several SvMs are descrlbed
ln secLlons
LSLbased welghLlng LreaLs several SvMs ln Lhe
SvM ensemble wlLh dlfferenL welghLs CfLen Lhe
welghLs of several SvMs are deLermlned ln
proporLlon Lo Lhelr accuracy of classlflcaLlon
1echnlcal SoluLlon
|erarch|ca| and adapt|ve comb|ners Lnsemb|es
Lmplrlcal LvaluaLlon
LxperlmenLs' uevlces and 1ools
SofLware
MA1LA8 7 82010a
MA1LA8 Arsenal ackage
Pardware
lu[lLso Slemens LapLop
lnLel Core 2 uue Cu 200 CPz
400 C8 8AM
Wlndows 7 32blL operaLlng SysLem
Lmplrlcal LvaluaLlon
LxperlmenL 1
lmplemenLlng Slngle SvM and dlfferenL Lnsembles
LffecL of Shuffllng (dlfferenL order of samples)
LffecL of leaLures (dlfferenL number and order of
feaLures)
LxperlmenL 2
ulfferenL kernels( Llnear olynomlal 88l Slgmold)
ulfferenL kernel arameLers and CosL lacLor
reclslon 8ecall l1 MeLrlcs
Lmplrlcal LvaluaLlon
Shuffled vs noShuffled
0
01
02
03
04
03
06
07
08
09
8elkl/8esL 8elkl/laclbo 8elkl/8esL 8elkl/laclbo
Arsenal SvM rbf noshuffled Arsenal SvM rbf shuffled
CorrecLlon ClasslflcaLlon 8aLe
uelLa
LeLa
alpha1
alpha2
beLa
A3AvL
u3AvL
u4AvL
u3AvL
A3CW
u3 CW
u4 CW
u3CW
A3 vA8
u3 vA8
u4 vA8
u3 vA8
lracLal
Lyapanov
Lmplrlcal LvaluaLlon
ulfferenL
leaLures
Crder
leaLures Croup1 Croup 2 Croup 3 Croup 4
1 uelLa A3AvL A3AvL u3 CW
2 LeLa u3 CW A3CW u4 vA8
3 alpha1 u4 vA8 A3 vA8 LeLa
4 alpha2 LeLa uelLa AnLropl
3 beLa AnLropl lracLal u4 CW
6 A3AvL u3AvL u3AvL u3 vA8
7 u3AvL u4 CW u3 CW alpha1
8 u4AvL u3 vA8 u3 vA8 lracLal
9 u3AvL alpha1 LeLa u4AvL
10 A3CW lracLal Lyapanov u3 vA8
11 u3 CW u4AvL u4AvL Lyapanov
12 u4 CW A3CW u4 CW alpha2
13 u3 CW u3 vA8 u4 vA8 u3AvL
14 A3 vA8 alpha2 alpha1 u3 CW
13 u3 vA8 Lyapanov AnLropl uelLa
16 u4 vA8 u3AvL u3AvL u3AvL
17 u3 vA8 u3 CW u3 CW *
18 lracLal A3 vA8 u3 vA8 *
19 Lyapanov uelLa beLa *
20 AnLropl beLa alpha2 *
Croup1
Combln
er
SvM1
SvM2
SvM3
llve lrequency leaLures
uelLa LeLa alpha1
Alpha2 beLa
1hree nonellnear leaLures
lracLal Lyapanov AnLropl
1welve WaveleL leaLures
A3AvL u3AvL u4AvL
u3AvLA3CW u3 CW
u4 CW u3 CWA3 vA8
u3 vA8 u4 vA8 u3 vA8
Comb
lner
SvM1
SvM2
SvM3
SvM4
uelLa
1eLa
alpha1
Alpha2
beLa
ComblnaLlon 2
ComblnaLlon 4
ComblnaLlon 3
A3AvL
u3AvL
u4AvL
u3AvL
A3CW
u3 CW
u4 CW
u3 CW
A3 vA8
u3 vA8
u4 vA8
u3 vA8
lracLal
Lyapanov
AnLropl
ComblnaLlon 1
Croup2
Lmplrlcal LvaluaLlon
Croup3
Combl
ner
SvM1
SvM2
SvM3
SvM4
A3AvL
A3CW
A3 vA8
uelLa
lracLal
ComblnaLlon 2
ComblnaLlon 4
ComblnaLlon 3
u3AvL
u3 CW
u3 vA8
1eLa
Lyapanov
u4AvL
u4 CW
u4 vA8
Alpha1
AnLropl
u3AvL
u3 CW
u3 vA8
8eLa
alpha2
ComblnaLlon 1
Combl
ner
SvM1
SvM2
SvM3
SvM4
ComblnaLlon 1
ComblnaLlon 1
ComblnaLlon 1
ComblnaLlon 1
Croup4
Lmplrlcal LvaluaLlon
Lmplrlcal LvaluaLlon
LxperlmenL Cne
00000
01000
02000
03000
04000
03000
06000
07000
08000
09000
10000
Croup1 Croup2 Croup3 Croup4
8elkl8esL CorrecLlon 8aLe
Slngle SvM
Ada8oosL
8agglng
MCWlLhMulLllSeL
MCPlerarchy Classlfy
00000
01000
02000
03000
04000
03000
06000
07000
08000
09000
Croup1 Croup2 Croup3 Croup4
8elkllacebo CorrecLlon 8aLe
Slngle SvM
Ada8oosL
8agglng
MCWlLhMulLllSeL
MCPlerarchy Classlfy
0
100
200
300
400
300
600
700
800
900
1000
Croup1 Croup2 Croup3 Croup4
8elkl8esL 8unnlng 1lme (Sec)
Slngle SvM
Ada8oosL
8agglng
MCWlLhMulLllSeL
MCPlerarchy Classlfy
0
200
400
600
800
1000
1200
1400
1600
1800
2000
Croup1 Croup2 Croup3 Croup4
8elkllacebo 8unnlng 1lme (Sec)
Slngle SvM
Ada8oosL
8agglng
MCWlLhMulLllSeL
MCPlerarchy Classlfy
Lmplrlcal LvaluaLlon
LxperlmenL 1wo
Spec|fy the best kerne| for Correct C|ass|f|cat|on
kate w|th M|n|mum Ca|cu|at|on 1|me
L|near
o|ynom|a|
k8I
S|gmo|d
Spec|fy the best parameters for
kerne| arm
Cost Iactor
Compare Correct|on C|ass|f|cat|on kate w|th
rec|s|on keca|| and I1
Lmplrlcal LvaluaLlon
LxperlmenL's 1wo 8esulLs
8elkl/8esL Croup4
8agglng Ada8oosL MulLlseL Pelrachy
kernel
arm
CorrecLlon
8aLe
8unnlng
1lme
CorrecLlon
8aLe
8unnlng
1lme
CorrecLlon
8aLe
8unnlng
1lme
CorrecLlon
8aLe
8unnlng
1lme
090 08981 71333 08212 16848
093 08980 40766 08986 49803
8elkl/lacebo Croup3
8agglng Ada8oosL MulLlseL Pelrachy
kernel
arm
CorrecLlon
8aLe
8unnlng
1lme
CorrecLlon
8aLe
8unnlng
1lme
CorrecLlon
8aLe
8unnlng
1lme
CorrecLlon
8aLe
8unnlng
1lme
070 08480 73879 08492 92834
093 08484 118393 07126 31132
Lmplrlcal LvaluaLlon
reclslon 8ecalll1
reclslon a measure for exacLness or CuallLy
8ecall a measure for compleLeness or
CuanLlLy
l1 or lMeasure Comblnes reclslon and
8ecall lL ls harmonlc average
l12 * (reclslon * 8ecall) / (reclslon + 8ecall)
Lmplrlcal LvaluaLlon
-'~ _-'- 2 - 2 - 1 '--~ ' l1 .- 88l CosLlacLor1
07
073
08
083
09
093
003 01 03 03 07 09 093 003 01 03 03 07 09 093 003 01 03 03 07 09 093 003 01 03 03 07 09 093
8agglng AdaboosL MulLllseL Pelrachy
8elkl/8esL Croup4 8elkl/lacebo Croup3
Concluslon
Accordlng Lo resulLs have Laken from Lhe experlmenLs one and Lwo
Shuffllng daLa before applylng lnLo classlflers lncrease Lhe CorrecLlon 8aLe
Lnsemble meLhods whlch are uslng LlbSvM as classlfler and 88l kernel
wlL parameLers kernel arm" 093 and CosL lacLor" 1 are classlfylng
beLLer
CorrecLlon ClasslflcaLlon 8aLe for AdaboosL 8agglng and Plerarchy
wrappers are very close or hlgher Lhan 083 so Lhey can be used for boLh
Lype of daLa 8elkl/8esL and 8elkl/lacebo whlle MulLllSeL wrapper has
hlgher preclslon Lhan 083 for 8elkl/8esL
8agglng has mlnlmum CalculaLlon 1lme for classlflcaLlon of 8elkl/lacebo
daLa wlLh preclslon hlgher Lhan 083 whlle for 8elkl/lacebo MulLllseL
has Lhe besL runnlng Llme
CalculaLlon Llme (seconds)
0
200
400
600
800
1000
1200
1400
1600
1800
2000
003 01 03 03 07 09 093 003 01 03 03 07 09 093 003 01 03 03 07 09 093 003 01 03 03 07 09 093
8agglng AdaboosL MulLllseL Pelrachy
8elkl/8esL Croup4 8elkl/lacebo Croup3
' -~~ ~~ -~'-~ '~-'~ ' -~~ ~~ _-'- = '
Wrapper '- 8agglng Ada8oosL Plerarchy '-~
' `' '- --~- 0.85 ' '+- ~ - ' '~ ~~' ~
-~'~ _- ~ - ~- -= ) - / - ='~' / -~` ( ~
~~- -~'-~' ' MulLllSeL '- -~'~ ~- -= ' '+-
- / ' `' ~ ''~ ='~' 0.85 ~~' ~ . ='
'=' '+-'~ Wrapper ~- -= ' -~~ ' '-
- '- -~'~ / ' `' ~ ' -~` 0.85 8agglng ''~
- '- -~'~ ' ~~' ~ '~ -+ / ='~' MulLllseL
~~' ~ '~ -+ ''~ . ' -~'-~' ~'- -= '- -~'~ ' '~
~ -' Wrapper ' CosLlacLor1 .- 88l '+~ '
' `' 0.85 ~~ ~ -- '=' '~ ~~ ' ~~ ~ -- .
CuesLlons?
1hank you