Escolar Documentos
Profissional Documentos
Cultura Documentos
an Introduction
Tjeerd Plomp
Professor Emeritus
University of Twente
Enschede, The Netherlands
1
Design-Based Research Collective (2003):
educational research is often divorced from the
problems and issues of everyday practice – a split
that resulted in a credibility gap and creates a
need for new research approaches that speak
directly to problems of practice and that lead to
the development of ‘usable knowledge’
(authors from domain of Learning Sciences)
2
Van den Akker (1999; director of Netherlands
Institute for Curriculum Development)
about research in domain of curriculum
development and implementation:
many ‘traditional’ research approaches such as
experiments, surveys, correlational analyses,
with their emphasis on description hardly
provide prescriptions that are useful for design
and development problems in education
3
Reeves (2006) for the domain of educational
technology:
there is “a legacy of ill-conceived and poorly
conducted research that results in no significant
differences or, at best, in modest effect sizes”
4
Quotes illustrate that
many researchers in various ‘corners’
of our field belief that there is a need
for an alternative research approach
that is directly relevant for problems in
educational practice.
5
In this presentation:
• Focus of research, research functions and
approaches
• What is design research
• Research question and other outputs
• Generalizibility in design research
• How to conduct DR – formative evaluation
• Design research dilemmas
• Concluding remarks
6
Focus of research project
11
What is Educational Design Research
(EDR)?
Distinction between:
• Development studies aimed at research-based
solutions for complex problems in educational
practice
• Validation studies aimed at development or
validation of a theory
Research on interventions versus research
through interventions
(McKenney & Reeves, 2012)
12
What is Educational Design Research
(EDR)?
Educational design research – type development
studies - is
the systematic study of designing, developing
and evaluating educational interventions - as
solutions for problems for which no ‘how-to-
do’ guidelines exist - aimed at
(i) optimal interventions AND
(ii) theories (design principles) on ‘how’ and ‘why’
these interventions work
13
What is Educational Design Research
(EDR)?
Educational design research – type validation
studies - is
the systematic study of designing, developing
and evaluating educational interventions –
e.g. learning environments – with the purpose
to develop and validate
theories about learning, learning
environments (and the like) OR to validate
design principles.
14
When is Design Research
Appropriate?
Eamonn Kelly (2009):
• Initial state(s) unknown or unclear
• Goal state(s) unknown or unclear
• Operators to move from initial states to goal
states are unknown or how to apply the
operators is unclear
15
Why embark on Design Research?
Various reasons – e.g.:
1. Need to solve a complex problem in
education practice
2. Need to elaborate and/or validate a
theory
(e.g. whether the theory of Realistic
Mathematics Education is valid in a
certain context)
16
Problems may exist at various levels - for
example:
System level:
Need for system for e-learning to serve specific
group of students
Institutional level:
What are effective methods for collaborative
learning
17
Educational Design Research encompasses
systematic educational design processes.
18
Iterations of systematic design
cycles
evaluation
Revision: yes?
No? STOP
19
Design research methodology can be
expressed as follows:
ID of problems Development of
with practitioners prototype solutions
Revision
21
Phases in Design Research
• Preliminary phase:
needs and content analysis, review of state-of-art of
literature => conceptualisation & design specifications
• Development or Prototyping phase
(iterative design phase + formative evaluation)
<=> micro-cycles of research
• Assessment phase (semi-summative evaluation)
ALL Phases:
systematic reflection and documentation (resulting in
design principles)
22
Generic Design Research Model
Tentative
Consult Formative
Product Practical
Evaluation
Researchers Experts & Products/
Approaches
Practitioners Results
Other Analyze Focused
Practitioners Redesign & Refinement
Sources Practical Literature
Context Review of Products & Theories
Problem in Context
Preliminary Investigation
of Problem, Context,
& Approaches Refinement of
Design Theory
Phases
Identification of Tentative Tentative Products Prototyping & Assessment of Problem Resolution
Problem Identification & Theories Preliminary Products & Theories & Advancing Theory
Products & Design Principles
23
Revision 5, Dated: 05-20-05 based on Tom Reeves second response on 05/15/05 regarding the “initial identification of the problem” and Wademan additional iterative processing enhancements.
On terminology:
(Educational) Design Research
stands for a ‘family’ of related research approaches
• Design studies, design experiments
• Design-based (implementation) research
• Development research
• Participative action research
• Formative research
• Engineering research
(e.g. Van den Akker et al., 2006):
But a number of characteristics in common!
24
Common characteristics
• Interventionist
• Iterative
• Process oriented
• Utility oriented: merit of design
measured
• Theory oriented: based upon &
contributing to
• Involvement of practitioners
25
A closer look:
26
Context Z:
Intervention X Outcomes
Input => Process Y1, Y2, …, Yn
Intervention X Outcomes
Input => Process Y1, Y2, …, Yn
Intervention X Outcomes
Input => Process Y1, Y2, …, Yn
Intervention X Outcomes
Input => Process Y1, Y2, …, Yn
Intervention X Outcomes
Input => Process Y1, Y2, …, Yn
In general:
Research goal may vary, but always:
(i) design/development of intervention
and
(ii) design principles or theory validation
31
Context Z:
Intervention X Outcomes
Input => Process Y1, Y2, …, Yn
Intervention ‘theory’
Local instruction theory
Intervention ‘hypothesis’
Design principle(s)
32
Research question in design research
If research goal is development of intervention:
What are the characteristics of an <intervention X>
to realize outcomes Y in context Z
OR in case of developing/validating a theory
(tentatively):
What is an adequate learning and teaching
strategy for <topic B> in context C in order to cope
with the main difficulties in the learning of this
topic
33
Examples of research questions
34
Examples of research questions
#2 What is an adequate learning and teaching
strategy for genetics in upper secondary
biology education in order to cope with the
main difficulties in learning and teaching
genetics, and
to promote the acquisition of a meaningful
and coherent understanding of hereditary
phenomena? ( Knippels, 2002)
35
Outputs of Design Research
Professional development:
Characteristic is collaboration of researchers
and practitioners
DR as form of professional development
practitioners develop awareness of how
research can contribute to improving
professional context
37
Design principles
(in development studies)
40
Design principles
• Heuristics – no guarantee for success in
other contexts, because developed in
certain context!
• But: will be increasingly powerful when
validated in various contexts.
QUESTION: can design principles or local
(instruction) theories be generalized from
one context to another
• This question of generalization similar as in
experimental research or case studies
41
Generalizability in Design Research:
Yin (2003) about case study research:
striving to generalize findings to some broader theory:
Context Z:
Intervention X Outcomes
Y1, Y2, …, Yn
Input => Process
44
Part 2:
How to conduct Design Research
46
How to conduct Design Reserach
Tenets:
• Rigor of research (see next slide)
• Relevance for practice
– Research team has good working knowledge of
target setting
• Collaboration with professionals from target
setting
• Iterations - emphasis on formative evaluation
to refine intervention and design theory.
47
Formative evaluation
= Evaluation to improve
Aimed at:
(i) Improving prototype towards intended
outcome
(ii) Sharpening underlying tentative design
principles (towards elaborated set)
48
Quality criteria for interventions:
1. Relevance or Content Validity:
• need for the intervention
• based on contemporary scientific insights
2. Consistency or Construct Validity
2. Practicality:
- can be used in settings for which developed
3. Effectiveness:
- yields desired results
- cost-benefit ratio
49
Shift in emphasis on criteria
Stage Criteria
Preliminary Emphasis mainly on relevance
and consistency (i.e. validity),
research
less on practicality
Development initially: consistency and
or practicality;
Prototyping increasingly practicality and
gradually effectiveness
phase
Assessment practicality and effectiveness
phase 50
Development stages of a prototype:
- Design specifications: first and general
description of intervention
- Global intervention: some or all components are
given some detail – horizontal prototype
- Part of intervention in detail- for use by target
group – vertical prototype
- Complete intervention: whole intervention
sufficiently detailed for use in intended target
setting
51
Formulating research questions
for various iterations/phases of project:
The syntax of these research questions is:
'What is the [quality criterium a, b, c and/or d] of
[element of] the intervention that is in
[development stage w, x, y, z].‘
Example:
What is the relevance [quality criterion] of the
content [conceptual element] of a quick reference
manual for using Chinese characters that is at a
global stage of devt [development stage] ?
52
Formative evaluation is part of
process of doing research:
Vali-
dity √ ea √ ea
Practi-
cality √ wt √ wt √ ea √ me √ to √ ft
Effective
ness √ to √ ft
ea: expert appraisal to: tryout
wt: walk through ft: field trial
me: micro- evaluation
58
Formative evaluation e.g.:
Design Global Partly Completely
specs design developed developed
Relevance • Screening • Screening • Screening • Focus group
• Focus group • Focus group • Focus group • Micro-
evaluatie
Consistenc • Screening • Screening • Screening • Focus group
y • Focus group • Focus group • Focus group • Micro-eva
Tryout in
3 classrooms Appraisel by
Appraisel by 3 experts
3 experts
Tryout with
Design guidelines & tchr educ studs
specifications
61
Formative evaluation e.g.:
64
65
66
Design research dilemmas
(see e.g. van den Akker et al., 2006)
67
Design research dilemmas
1. Researcher in multiple roles (contd):
• Quality research design:
– Triangulation: of data source, instruments
– Attention for validity and reliability of
instruments
– Empirical testing
– Systematic documentation and reflection
• Many tactics:
- member checks
- multiple observers
etc 68
Design research dilemmas
70
Concluding remarks:
71
Why ‘design research’ and
not ‘action research’?
Both
• Real-world problems
• Aim at improving practice
• Cyclical in nature
• Participative
BUT:
design research aims to contribute to theory -
has focus on producing generalizable design
principles
72
Important sources:
73
An Introduction to
Educational Design
Research
Tjeerd Plomp & Nienke
Nieveen (Eds)
Published by: SLO –
Netherlands Institute for
Curriculum Development.
Enschede, 2009 (126 pages)
Pdf file freely available via:
http://www.slo.nl/organisatie/int
ernational/publications/
75
End of part 1
Thank you!
(t.plomp@utwente.nl)
Part 2:
How to conduct Design Research
76