Contents

Introduction............................................................................................................................................3
Deeplearning..........................................................................................................................................3
Assessingdeeplearning..........................................................................................................................5
Adeeplearningassessmentstrategy.....................................................................................................6
Endnotes.................................................................................................................................................9

NewPedagogiesforDeepLearningWhitepaper:
AStrategicApproachtotheAssessmentofDeepLearning

Author:PeterHillandMariaLangworthy
Publishedby:CollaborativeImpactSPC,Seattle,Washington
June2014

FormoreinformationaboutNewPedagogiesforDeepLearningvisitwww.newpedagogies.org.

2014CollaborativeImpact
CreativeCommonsAttributionShareAlike4.0InternationalLicense.

Introduction
NewPedagogiesforDeepLearningisaglobalpartnershipofclustersof100schoolsineachoften
countriesthatarecommittedtomobilizingdeeplearningacrosssystemsthroughcapacitybuilding
fornewpartnerships,pedagogies,measuresandtechnologies.NewPedagogiesforDeepLearningis
led by Collaborative Impact which coordinates a partnership team of worldleading education
specialiststooverseespecificaspectsoftheproject.
OnecomponentofNewPedagogiesforDeepLearningisaresearchanddevelopmentefforttocreate
anewgenerationofinstrumentsandprotocolstoassessdeeplearning.Inthisregard,NewPedagogies
forDeepLearningsharesthesameambitionasseveralotherprojectsnowunderwayindifferentparts
oftheworld.However,itdiffersfrommanyotherinitiativesinthatitispredicatedonthebeliefthat
solutions to the assessment challenge can only emerge in the context of implementing new
pedagogiesinregularclassroomsettings.
Thispaperprovidesabriefdescriptionof,andrationalefor,theassessmentstrategythatwillguide
workwithintheNewPedagogiesforDeepLearningpartnership.Butfirst,afewwordsaboutdeep
learning,whatitisandwhyitisimportant.InformationanddescriptionoftheoverallNewPedagogies
forDeepLearningglobalpartnershipanditsaimscanbefoundatwww.newpedagogies.org.

Deeplearning
Aswelearn,wetypicallyproceedthroughthreeidentifiablephases.Inthefirstphase,weprogress
fromhavinglittleornocapacitytoacquiringdevelopedcapacity.Atthebeginningstages,aparent,
teacherorsomeotherpersonwithcompetencecloselystructuresourlearningandmayprovidedirect
guidanceuntilwearebetterabletomonitoritforourselves,albeitwithongoingfeedbackandsupport
fromthatperson.
In the second phase, we need to achieve mastery. There are no short cuts to mastery; it requires
practiceuntilthenewlearningbecomesinternalizedandautomatized.
Inthethirdphase,iflearningistobeofvalue,weneedtobeabletouseourdevelopedcapacitiesand
applyourknowledgetosolvereallifeproblemsandgeneratenewknowledge.Itisduringthisphase
thatdeeplearningtakesplace.Thisiswhenthelearnerbecomesthedoerandthepointatwhich
learning and living merge. Also, it is during this phase that deep learning competencies and
dispositions, such as real world problem solving, knowledge construction, creative thinking,
collaboration,resilienceandentrepreneurialism,cometothefore.
FullanandLangworthydefinedeeplearningaslearningthatgoesbeyondmasteryofexistingcontent
knowledge; creating and using new knowledge in the world. 1 This conception of deep learning
extendspreviousconceptionsofdeeplearningthatcenterontheconstructionofnewknowledge,to
incorporatethewaysinwhichdigitaltechnologiesaregivingstudentstoolstoapplyanddothings
withthatnewknowledgeintheworldbeyondschoolsandclassrooms.
Formanyyears,schoolsystemsaroundtheglobehavefocusedonensuringhighlevelsofbasicliteracy
andnumeracyforall.Moststillhavesomewaytogobeforethatgoalisfullyachieved,manyhavehit
aceilingintermsofimprovingstudentachievement2andalmostallarestrugglingwithproblemsof
lackofstudentengagement.
3

Meanwhile,intheworldofwork,whileunskilledworkhaslongbeenindecline,manyroutinewhite
collarjobsrequiringhighlevelsofliteracyandnumeracyhavebeen,orareintheprocessofbecoming,
automatedandcomputerized.Wholecategoriesofemploymentaredisappearing,andnotjustjobs
formerly undertaken by the uneducated and unskilled, but increasingly jobs traditionally filled by
collegegraduates.
Atthesametime,opportunitiesforemploymentareconstantlyspringingupinnewfieldsneedingnot
only high levels of education and training, but people who can apply their knowledge to solve
problemsandcreatenewknowledge.Manyofthesejobsgounfilledforthelackofapplicantswith
theskillsemployersarelookingfor.3Themessageforeducatorsisclear:thefocusonbringingallup
tohighlevelsofbasiceducationmustbemaintained,butinadditionthereneedstobeanewfocus
ongoingfurther,ofsecuringstudentengagementandondevelopingdeeplearningcompetencies,not
justforthehighachievers,butforall.
Fosteringdeeplearningpresentsthreebiginterdependentchallenges,asshowninthediagrambelow.
Thefirstisthecurriculumchallengeof:

definingwhatdeeplearningmeans;
scopingandsequencingtherelevant
competencies;
integratingthemintoschoolsubjectsand
disciplinarythinking;and
clearingthecluttersothereistimeand
spaceinthecurriculumtoengageindeep
learning.

Considerable work has been done on the

definitionalproblemandonreviewingideasand
theresearchevidenceregarding21stcenturyskills
and deep learning. 4 The New Pedagogies for
Deep Learning partnership builds upon this and
extensivedevelopmentworkundertakenbytheOECDforPISA20155,theATC21Sinitiative6,Professor
John Hatties work on visible learning 7 and the ITL Research project 8 . New Pedagogies for Deep
Learningfocusesonasetofsixfutureskills,the6Cs,namely:
1. Character
2. Citizenship
3. Collaboration

4. Communication
5. Creativity
6. CriticalThinking

Asetoflearningprogressionshasbeendevelopedbasedonalloftheaboveworkthatoperationalizes
definitionsofdeeplearningcompetencies.Theseprogressionswillallowlearningtasksandstudent
work to be rated according to the degree to which the tasks develop and student evidence
demonstrateshigherlevelsofthesecompetencies.
Thesecondisthepedagogicalchallengeofhowtoteachfordeeplearninginadigitalworld.Within
the New Pedagogies for Deep Learning project, mobilizing and fostering new pedagogies for deep
learning is seen as requiring three elements: 1) new learning partnerships in which students and
teachersbecomecolearnersandstudentsareactivelyengagedinmakingdecisionsaboutwhatand
howtheywilllearn;2)engagementindeeplearningtasksthatprovidetheopportunityforthelearner
4

tobecomethedoerandtodevelopandpracticetheprocessofdeeplearning,and;3)usingdigital
tools and resources in ways that accelerate and deepen learning. Each of these elements is being
addressed within the New Pedagogies for Deep Learning through capacity building frameworks
developedbytheglobalpartnership,byclusterleadersandbyclustercapacityteams.
The third is the assessment challenge. Assessment in a school education context refers to any
appraisal(orjudgementorevaluation)ofastudentsworkorperformance.9Itsprimarypurposeis
toseektodeterminewhatstudentsknowandcando,andhowtheyareprogressing.
That sounds straightforward enough, but in the real world of policy and practice, educational
assessmentisfraughtwithchallengesandcontroversy.Inparticular,policythinkingaboutassessment
hasbecomenarrowlyfocusedonassessmentforcertificationandaccountabilitypurposesandcarried
outinwaysthatmakeassessmentofdeeplearningdifficult,ifnotimpossible.
There is now a consensus among leading experts 10 that the time has come to seriously rethink
assessment(andtheusestowhichitisbeingput)toensurethatitbetterservestheneedsoflearners
andthedemandsofthemodernworldforarangeofdeeplearningcompetenciesanddispositions
that current assessmentprogramsareunable totapinto.Whattheyareadvocatingamountsto a
paradigmshift,thekeycomponentsofwhicharesummarisedinTable1.
Throughtheworkofresearchers,innovativeteachers,schools,systems,andpowerfulcoalitionsand
collaborations,thechangeprocessisalreadyunderwayandtypicalofchangeinthedigitalage,much
of it is occurring spontaneously and from the ground up. The New Pedagogies for Deep Learning
partnershipbuildsonworkalreadycompletedandunderwayandisuniqueininvolvingalargenumber
ofschools(1000)intencountriestopromotedeeplearningandinparticularmeasurementofdeep
learningusingabottomupapproach.

Table1.Theparadigmshiftinassessmentofstudentlearning

AkeyfeatureofNewPedagogiesforDeepLearningwillbethecreationofanintegratedsystemof
measuresnotjustofstudentoutcomes,butalsoofthepedagogicalpracticesandlearningconditions
ineducationsystemsthatmustbeinplacefordeeplearningtoflourish.Itwillmeasurewhetherschool
culture and infrastructure support new pedagogies. It will measure teachers adoption of the
pedagogicalpracticesthatfacilitatedeeplearning.Anditwilldevelopandtestacommonapproach
tomeasuringstudentsdeeplearningcompetencies.

Assessingdeeplearning
Assessment of deep learning is not altogether new. It has long been practised by good classroom
teachers,althoughoftenunsystematicallyandimplicitly.Ithasalso,formanydecades,contributedto
highstakescertificationarrangementsincertainjurisdictions,typicallytakingtheformofextended
research projects, investigations, practical or laboratory work, orals and essays, completed within
defined parameters and subject to external moderation to ensure comparability of standards.11In
addition, a few jurisdictions have incorporated assessment of deep learning into highstakes
accountabilityarrangementsthroughtheadoptionofprofilesofstudentlearning.
However,assessmentofdeeplearninghasalwaysbeenseenasproblematic,becauseitisnotreadily
amenabletoobjectivetesting,butrequiresprofessionaljudgmentandhenceinvolvesanelementof
subjectivity.Whilethekindsofquestionsthatdominatemosttestsandexaminations(namelymultiple
choiceorshortresponseitems)arequiteefficientinobjectivelyassessingmanybasicskillsandlower
ordercognitiveprocesses,theyareunabletomeasuredeeplearningandmosthigherordercognitive,
interandintrapersonalskillsanddispositions.Whenanattemptismadetohavetheseoutcomes
and attributes assessed by teachers in school settings a number of concerns invariably arise,
particularlyregarding:
Feasibility: Are the costs of the assessment and associated moderation processes doable and
affordable,dotheymakeexcessivedemandsonteachertimeandefforttoadministerandscore,and
doestheirimplementationresultinasignificantlossofteachingtime?
Reliability:Howcertaincanonebethatastudentwouldperformatthesamelevelifassessedon
anotheroccasionorwithanotherformoftheassessment,andhowmuchconfidencecanonehavein
theconsistencyofdifferentteachersratingsofstudentperformance(interraterreliability)?
Validity:Towhatextentdoratingsofstudentperformancemeasurewhattheypurporttomeasure
andarenotmeasuringsomethingorsomebodyelse?
Utility: How useful is the information generated by the assessment, does it serve the intended
purposeanddoesitgeneratefeedbackthatissufficientlyhelpfulinimprovingteachingandstudent
learningoutcomesastomakeitworthwhile?
Ofcourse,theseconcernsapplytoallassessmentofstudentlearningandareconcernsthatshould
neverbeignored.WithintheNewPedagogiesforDeepLearningpartnership,thepositiontakenon
assessingdeeplearningisasfollows:
1. Deeplearning,asdefinedabove,istooimportantnottoassess,thereforeweneedtopool
ourexpertiseandresourcestocreatenewtoolsandprotocolstoassessitinwaysthatareas
feasible,reliableandvalidaswecanmakethem,thatyieldinformationthatdirectlyinforms
5

teachingandlearningandthatcanleadtorecognitionforstudentsofdevelopedcapacities
thatareofhighvaluewithinandbeyondtheschool.
2. Assessmentofdeeplearningshouldnotbeviewedasanaddonbutsomethingthathappens
inthecontextofdeepengagementinchallengingandauthenticlearningtasksthatdemand
it. Therefore, assessment of deep learning should be conceived as an integral part of the
learning process, making old concepts of teaching time less relevant, but increasing and
enhancinglearningtime.
3. Wenowhavegreaterconceptual clarityastotheskillsand dispositionsthat contributeto
deep learning and progressions that help us design tasks and then assess student work
(evidence of learning) in a consistent (i.e., reliable) fashion, given appropriate training and
support.
4. Newdigitaltechnologiescanfacilitatemanyaspectsofthedesign,initiation,assessmentand
analysisofstudentwork,aswellasprovidingaplatformforteachersandstudentstoboth
giveandreceiverichandimmediatefeedbacktoinformteachingandlearning.
5. Outcomesareimportantandouraimshouldalwaysbetoseekevidencetobeabletosaywith
confidencethatstudentswhohaveexperiencedthenewpedagogieshaveacquiredthefuture
skillsthatunderpindeeplearning.
6. Yes,assessmentofdeeplearning,aswith complex assessmentsinmostotherprofessions,
inevitablyinvolvesasignificantelementofprofessionaljudgmentandthisshouldberegarded
asapositive,notanegative.EnhancingsuchjudgmentsisafocalpointoftheNewPedagogies
for Deep Learning capacitybuilding framework. The goal is to provide teachers with the
necessaryframework,definitionsandprofessionallearningandpracticetoenablethemto
exerciseprofessionaljudgmentandto obtainevidenceregardingtheextent towhichsuch
judgmentscanbetrusted.

Adeeplearningassessmentstrategy
OnegoaloftheNewPedagogiesforDeepLearningpartnershipistodemonstratethatitisindeed
possibletoassessdeeplearninginwaysthatarefeasible,validandreliableandthatgeneratehigh
valueinformationaboutimportantlearningoutcomes.
The strategy to be followed in pursuing this goal includes several stages. A series of overlapping
activitieswillbecarriedoutasindicatedinthefigurebelow.Theseactivitieswillbefollowedinan
iterativefashionandtheoutputsrefinedinthelightofteachersandschoolsexperiencesasreflected
in measurement system data. Measures will be designed to serve both summative and formative
purposes,andwillbeusedbothtoascertainoutcomesandtogeneratefeedbacktodrivecontinuous
improvement. The starting point will be operationalizing deep learning competencies for learning
tasksandassessingstudentprogress.Anewsetoflearningprogressionsweredevelopedtoexplicitly
defineandoperationalizeeachofthedeeplearningcompetencieslistedbelow.

DeepLearningCompetencies*

Character

Collaboration

Creativity

Citizenship

CriticalThinking

Communication

*DigitalAppliesAcrossAllDeepLearningCompetencies

Teachers and schools involved in New Pedagogies for Deep Learning will be introduced to these
learningprogressionstoguidetheirdesignoftheirownlearningtasksthatdevelopstudentsdeep
learning competencies as part of the capacitybuilding strategy in the New Pedagogies for Deep
Learningproject.
Capacitybuildingwilloccurintandemwiththeidentificationandcaptureofacoresetofsampleor
exemplar learning tasks which will be disseminated as examples and stimuli for teachers and
studentsinclusterschoolstoadaptoruseasmodelsindesigningtheirown,inaccordancewiththe
deeplearningprogressions.Teachersandstudentswillthenbegintoaddtotheexemplardatabase,
submittingandratingtheirowntasksandworkaccordingtotheprogressions.

Developmentofdeep
learningprogressions
Identificationofexemplar
deeplearningtasks
Implementtasksandassessments
andcapturedata
Refineprotocolsfor
progressionratings
Analysedata,generatefeedback
andfosterreflection

Usingthecoresetofdeeplearningprogressionsandprotocols,clusterschoolswillbesupportedin
designingandimplementingtheirowndeeplearningtasks.Thiswillbepartofanoverallinquirycycle
for developing teachers pedagogical capacities. Task and student work progression ratings will be
analysedandusedbyteacherstogeneratefeedbackandinformaniterativecollaborativelearning
cycle of: assessing students needs and aspirations, designing tasks to implement and assess deep
7

learning,implementingthem,monitoringwhathappens,measuringoutcomesandreflectingonwhat
next.
Thelearningprogressionswillserveasaframeworktoguideboththetaskdesignandtheassessment
ofstudentsdeeplearningprogress.Theframeworkprovidedbytheprogressionswillallowteachers
and students to develop a common language for discussing and assessing both learning tasks and
studentprogressintermsofdeeplearningcompetencies.
AkeyelementofNewPedagogiesforDeepLearningisthecreationofatechnologyplatformto:

collectDeepLearningConditionsdataatSystem,ClusterandSchoollevels;
collectratingsofstudentsdeeplearningprogress;
collectandratedeeplearningtasks;and
conductonlinesurveysofallstakeholders.

Thisplatformwillallowbothtasksandevidenceofstudentprogresstobemoreeasilyaccessedand
ratedbyotherstudents,teachers,researchersandsystemadministratorsinaprotectedandprivate
onlineenvironment.Thisdatabasewillalsoserveasameansfordevelopingandtestingprotocolsfor
improvingreliability,akeyaspectoftheassessmentstrategy.Thisprocesswillprovidethemeansto
explicitlyevaluatetheextenttowhichthereisasharedunderstandingofwhatitmeanstodevelop
deep learning competencies, and to develop exemplars and protocols that support that shared
understanding.
Inaddition,theglobalNewPedagogiesforDeepLearningteamworkinginpartnershipwiththeschool
clusterswillidentifyarangeofdeeplearningassessmentsthatareemergingorhavebeendeveloped
byschoolsandotherorganizations.Theseassessmentsmaybeimplementedbyschoolsorclustersto
provideavarietyofdatapointsondeeplearningoutcomesthatteacherscanuseinadditiontothe
learningprogressiondatacollectedthroughtheNewPedagogiesforDeepLearningpartnership.
ThroughoutthelifeoftheNewPedagogiesforDeepLearningpartnership,reviewandcommunication
willbemaintainedwithsimilardeeplearningassessmentprojectsandinitiatives,largeandsmall,in
order to share with and learn from them, and to maximize synergies that develop when there is
networkingwithotherspursuingthesamekindsofbreakthroughsinthinkingandpractice.

Endnotes

Fullan,M.,&Langworthy,M.(2014).Arichseam:Hownewpedagogiesfinddeeplearning.
London:Pearson,p.7.Retrievedfrom:
http://www.michaelfullan.ca/wpcontent/uploads/2014/01/3897.Rich_Seam_web.pdf

Forexample,ofthetopnineOECDnationsinreadingin2000,sixrecordednegativegrowthover
theperiod20002012.

ILO(2013).Globalemploymenttrends2013:Recoveringfromasecondjobsdip.Retrievedfrom
http://www.ilo.org/wcmsp5/groups/public/dgreports/dcomm/
publ/documents/publication/wcms_202215.pdf

Seeespecially:JamesW.PellegrinoandMargaretL.Hilton,(Eds).(2012)CommitteeonDefining
DeeperLearningand21stCenturySkills.EducationforLifeandWork:DevelopingTransferable
KnowledgeandSkillsinthe21stCentury.NationalResearchCouncil.Retrievedfrom:

http://www.leg.state.vt.us/WorkGroups/EdOp/Education%20for%20Life%20and%20Work
%20National%20Academy%20of%20Sciences.pdf

SeePISADraftCollaborativeProblemSolvingFramework.Retrievablefrom:
http://www.oecd.org/pisa/pisaproducts/DraftPISA2015CollaborativeProblemSolvingFramework.pdf

AssessmentandTeachingof21stCenturySkillsproject.http://atc21s.org

Hattie,J.(2009).VisibleLearning:ASynthesisofOver800MetaAnalysesRelatingto
Achievement.NewYork:Routledge.

New21stCenturyLearningDesignRubricsforLearningActivitiesandStudentWork
http://atc21s.org

Sadler,R.(1989).Formativeassessmentandthedesignofinstructionalsystems.Instructional
Science18,p120.

10

Seeespecially:TheGordonCommissionontheFutureofAssessmentinEducation(2013).A
publicpolicystatement.Princeton,NJ:TheGordonCommission,pp.78.Retrievablefrom:

http://www.gordoncommission.org/publicationsreports.html

Masters,G.N.(2013).ReformingEducationalAssessment:Imperatives,principlesandchallenges.
Camberwell,Victoria,Australia:AustralianCouncilforEducationalResearch,p.14.Retrieved
from:http://research.acer.edu.au/aer/12

11

SeeConley,D.T.,&DarlingHammond,L.(2013).Creatingsystemsofassessmentfordeeper
learning.Stanford,CA:StanfordCenterforOpportunityPolicyinEducationforareviewof
systemsandauthoritiesthatactivelyseektoassessdeeplearninginthecontextofhighstakes
assessment.