!

Using Linked Data in Learning Analytics LAK 2013 tutorial 09&)A&HC6,CJ,186K*$,>&:&,:CC)(,JC%,1*&%686E,L6&)IH'( ,

!"#$%&'()%*+,-."%(!"#$%&'#()*%+,$%&'#()*%-$*., !/01230/+,45*6,7689*%(8:*8:,;*$*%)&6$+,;1., /".+0(!"%$"%( !1<2,=*(*&%'#,/*6:*%+,>0., 1*2,&"3($4563&#(!"?$&@A86+,?$&@A86-6*:., !B6CD)*$E*,F*$8&,G6(H:A:*+,3#*,45*6,7689*%(8:I+,7B., 72"8*#()&"29",, !1<2,=*(*&%'#,/*6:*%+,>0.,

Example of scientific competitions

What are the evaluation criteria of Robot Wars?

Criteria: ! Damage ! Aggression probabilistic combination of ! Control Item-based method User-based method ! Applause Matrix Factorization
(May be) content-based method

RecSysTEL Evaluation criteria

1. Accuracy 2. Coverage 3. Precision 4. Recall

1. Accuracy 2. Coverage 3. Precision 4. Recall

1. Effectiveness of learning 2. Efficiency of learning 3. Drop out rate 4. Satisfaction

1. Reaction of learner 2. Learning improved 3. Behaviour 4. Results

Combine approach by Drachsler et al. 2008

Kirkpatrick model by Manouselis et al. 2010

3

TEL RecSys::Review study

Conclusions: Half of the systems (11/20) still at design or prototyping stage only 9 systems evaluated through trials with human users.
Manouselis, N., Drachsler, H., Vuorikari, R., Hummel, H. G. K., & Koper, R. (2011). Recommender Systems in Technology Enhanced Learning. In P. B. Kantor, F. Ricci, L. Rokach, & B. Shapira (Eds.), Recommender Systems Handbook (pp. 387-415). Berlin: Springer. 4

The TEL recommender research is a bit like this...

We need to design for each domain an appropriate recommender system that ts the goals and tasks "

TEL recommender experiments lack results The performance transparency and of different research standardization. efforts in recommender They need tohardly be systems are repeatable to comparable. test:

! Validity (Manouselis et al., 2010) ! Verification ! Compare results

Kaptain Kobold! http://www.ickr.com/photos/ kaptainkobold/3203311346/"

Data-driven Research and Learning Analytics"

EATELHendrik Drachsler (a), Katrien Verbert (b)" " (a) CELSTEC, Open University of the Netherlands" (b) Dept. Computer Science, K.U.Leuven, Belgium" "

7!

TEL RecSys::Evaluation/datasets

" Drachsler, H., Bogers, T., Vuorikari, R., Verbert, K., Duval, E., Manouselis, N., Beham, G., Lindstaedt, S., Stern, H., Friedrich, M., & Wolpers, M. (2010). Issues and Considerations regarding Sharable Data Sets for Recommender Systems in Technology Enhanced Learning. Presentation at the 1st Workshop Recommnder Systems in Technology Enhanced Learning (RecSysTEL) in conjunction with 5th European Conference on Technology Enhanced Learning (EC-TEL 2010): Sustaining TEL: From Innovation to Learning and Practice. September, 28, 2010, Barcelona, Spain." 9 "

dataTEL evaluation model 5. Dataset Framework

Formal Datasets Informal
Data C

Data A

Data B

Algorithms: Algoritmen A Algoritmen B Algoritmen C Models: Learner Model A Learner Model B Measured attributes: Attribute A Attribute B Attribute C

Algorithms: Algoritmen D Algoritmen E Models: Learner Model C Learner Model E Measured attributes: Attribute A Attribute B Attribute C
17

Algorithms: Algoritmen B Algoritmen D Models: Learner Model A Learner Model C Measured attributes: Attribute A Attribute B Attribute C

42

10

dataTEL evaluation model 5. Dataset Framework

Formal Datasets Informal

In Data LinkedUp we have Data theB opportunity to apply A Data C a structured approach to develop a community accepted evaluation framework . Algorithms: Algorithms: Algorithms:
Algoritmen A Algoritmen B Algoritmen C

1.! Top-Down by a literature study Models: Models: 2.! Bottom-up by Models: GCM with experts in the field
Learner Model A Learner Model B Learner Model C Learner Model E Learner Model A Learner Model C Measured attributes: Attribute A Attribute B Attribute C Measured attributes: Attribute A Attribute B Attribute C
17

Algoritmen D Algoritmen E

Algoritmen B Algoritmen D

Measured attributes: Attribute A Attribute B Attribute C

42

11

12

)":".0;<"#2(08(2,"(/:*.3*=0#(>%*<"?0%'( (
P1: Initialisation M0-M6: Preparation P2: Establishment and Evaluation

P3: Exit and Sustainability

M7-M18: Competition cycle M18-M24: Finalising

Comp etition

EF proposal

Expert validation

Draft

3x

Revie w of EF

Final release of EF

New versio n

Refin ement of EF

Literature review Group Concept Cognitive Mapping Mapping

Documentation Dissemination Practical experiences and refinement

Hendrik Drachsler

25 February 2013

13

@%03;(A0#+";2(1*;;&#B((
! Group Concept Mapping resembles the Post-it notes problem solving technique and Delphi method ! GCM involves participants in a few simple activities (generating, sorting and rating of ideas) that most people are used to.

GCM is different in two substantial ways: 1. Robust analysis (MDS and HCA) GCM takes up the original participants contribution and then quantitatively aggregate it to show their collective view (as thematic clusters) 2. Visualisation GCM presents the results from the analysis as conceptual maps and other graphical representations (pattern matching and go-zones).

Hendrik Drachsler

25 February 2013

14

@%03;(A0#+";2(1*;;&#B((
brainstorm
! innovations in way network is delivered ! (investigate) corporate/structural alignment ! assist in the development of non-traditional partnerships (Rehab with the Medicine Community) ! expand investigation and knowledge of PSN'S/PSO's ! continue STHCS sponsored forums on public health issues (medicine managed care forum) ! inventory assets of all participating agencies (providers, Venn Diagrams) ! access additional funds for telemedicine expansion ! better utilization of current technological bridge ! continued support by STHCS to member facilities ! expand and encourage utilization of interface programs to strengthen the viability and to improve the health care delivery system (ie teleconference) ! discussion with CCHN

sort
Decide how to manage multiple tasks. 20 Manage resources effectively. 4

Work quickly and effectively under pressure 49

Organize the work when directions are not specific. 39

...organize the issues...

2 1

ic. s. tion ecif ma a sk t sp for nt. le t no f in porta ltip e o r u a e ud is im age m ns 2 4 ultit hat n ctio 3 ire ma am ew 5 to nd id an he 4 ow Sc dec w h k d e r 3 an 5 3 cid wo nd De he ly 2 na 4 et ive tio 4 niz ect 3 ma ga 5 eff for . Or e 2 in 4 tim of tant 5 ge 1 r de 3 na 5 itu impo Ma ult 2 4 a m hat is 1 1 n 3 a w 5 Sc cide 2 4 de 3 1 3 5 2 4 1 3 2 1 5 1 M

Ra

gS tin

e he

t
g ime et

ive e ct eff

ly

a an

Ma

na

ge

rc ou res

es

ec eff

ly. tive

rate
Hendrik Drachsler 25 February 2013 15

@%03;(A0#+";2(1*;;&#B((

Hendrik Drachsler

25 February 2013

16

@%03;(A0#+";2(1*;;&#B((

Hendrik Drachsler

25 February 2013

17

@%03;(A0#+";2(1*;;&#B((

Hendrik Drachsler

25 February 2013

18

@%03;(A0#+";2(1*;;&#B((
! innovations in way network is delivered ! (investigate) corporate/structural alignment ! assist in the development of non-traditional partnerships (Rehab with the Medicine Community) ! expand investigation and knowledge of PSN'S/PSO's ! continue STHCS sponsored forums on public health issues (medicine managed care forum) ! inventory assets of all participating agencies (providers, Venn Diagrams) ! access additional funds for telemedicine expansion ! better utilization of current technological bridge ! continued support by STHCS to member facilities ! expand and encourage utilization of interface programs to strengthen the viability and to improve the health care delivery system (ie teleconference) ! discussion with CCHN

map the issues...

organize
sort
Decide how to manage multiple tasks. 20 Manage resources effectively. 4 Work quickly and effectively under pressure 49

Organize the work when directions are not specific. 39

Ra
4 1 2 3 3

ely . et he effectiv effectively s g S time tin Manage resource 1 5 4 5 4 5 4 2 1 3 2 3 3

Ma 2

na

ge

4 5

d an . ation ks. ecific inform t sp le tas no e of rtant. ltip are mu ltitudimpo ns ge na a muat is ectio ma an dir Sc e wh w to en cid wh de e ho rk cid wo d De e the ely n an niz tio ectiv ga ma eff Or or e inf tim 5 ge e of rtant. na 5 ltitudimpo Ma 4 a muat is 1 an wh 5 Sc cide 4 de 3 3 5 4 3 2

Information Services Community & Consumer Views

Technology

rate

Regionalization

Management Financing

STHCS as model

Hendrik Drachsler

25 February 2013

19

@%03;(A0#+";2(1*;;&#B((
! innovations in way network is delivered ! (investigate) corporate/structural alignment ! assist in the development of non-traditional partnerships (Rehab with the Medicine Community) ! expand investigation and knowledge of PSN'S/PSO's ! continue STHCS sponsored forums on public health issues (medicine managed care forum) ! inventory assets of all participating agencies (providers, Venn Diagrams) ! access additional funds for telemedicine expansion ! better utilization of current technological bridge ! continued support by STHCS to member facilities ! expand and encourage utilization of interface programs to strengthen the viability and to improve the health care delivery system (ie teleconference) ! discussion with CCHN

organize
sort
Decide how to manage multiple tasks. 20 Manage resources effectively. 4

Information Services

Technology

Community & Consumer Views

Work quickly and effectively under pressure 49

Organize the work when directions are not specific. 39

Ra
4 1 2 3 3

ely . et he effectiv effectively s g S time tin Manage resource 1 5 4 5 4 5 4 2 1 3 2 3 3

Ma 2

na

ge

4 5

d an . ation ks. ecific inform t sp le tas no e of rtant. ltip are mu ltitudimpo ns ge na a muat is ectio ma an dir Sc e wh w to en cid wh de e ho rk cid wo d De e the ely n an niz tio ectiv ga ma eff Or or e inf tim 5 ge e of rtant. na 5 ltitudimpo Ma 4 a muat is 1 an wh 5 Sc cide 4 de 3 3 5 4 3 2

Regionalization

rate

map
Community & Consumer Views

Information Services

Technology

Regionalization

Financing
Management Financing STHCS as model

Management

Mission & Ideology

...prioritize the issues...

Hendrik Drachsler 25 February 2013 20

@%03;(A0#+";2(1*;;&#B((

D2.1 Evaluation Criteria and Methods

! Invited 122 external experts ! 56 experts contributed 212 indicators for the evaluation framework ! After cleaning -> 108 indicators remained ! 26 experts sorted on similarity in meaning ! 26 experts rated on priority and applicability

Hendrik Drachsler

25 February 2013

21

Plus Minus Interesting rating

Look at and listen to the presentation of the Evaluation Framework Meanwhilecreate notes on P: M: I: Plus Minus Interesting

Write down everything that comes to your mind, generate as many ideas as possible, do not filter your ideas.

@%03;(A0#+";2(1*;;&#B((
A point map

Hendrik Drachsler

25 February 2013

23

@%03;(A0#+";2(1*;;&#B((
A cluster map 15

Hendrik Drachsler

25 February 2013

24

@%03;(A0#+";2(1*;;&#B((
A cluster map 6

Hendrik Drachsler

25 February 2013

25

@%03;(A0#+";2(1*;;&#B((
Clusters labels

Hendrik Drachsler

25 February 2013

26

@%03;(A0#+";2(1*;;&#B((
Rating Map Priority

Hendrik Drachsler

25 February 2013

27

@%03;(A0#+";2(1*;;&#B((
Rating Map Applicability

Hendrik Drachsler

25 February 2013

28

@%03;(A0#+";2(1*;;&#B((

Hendrik Drachsler

25 February 2013

29

@%03;(A0#+";2(1*;;&#B((

Hendrik Drachsler

25 February 2013

30

WP2: Literature review

WP2: Literature review

1. Literature review of suitable evaluation approaches and criteria 2. Review of comprising initiatives such as LinkedEducation, MULCE, E3FPLE and the SIG dataTEL

WP2: Literature review

Many thanks for your attention!

This silde is available at: http://www.slideshare.com/Drachsler Email: Skype: hendrik.drachsler@ou.nl celstec-hendrik.drachsler

Blogging at: http://www.drachsler.de Twittering at: http://twitter.com/HDrachsler

Hendrik Drachsler

25 February 2013

34