Connecting Research,

Policy and Practice

Overview of Education Research &

Development Centers Competition
(84.305C)
Rebecca McGill-Wilkinson, Ph.D.
James Benson, Ph.D.
Erin Higgins, Ph.D. & Jonathan Levy, Ph.D.
National Center for Education Research

ies.ed.gov

Organizational Structure of IES

Office of the
Director

National Board
for Education
Sciences

Standards &
Review Office

National
Center for
Education
Evaluation
ies.ed.gov

National
Center for
Education
Research

National
Center for
Education
Statistics

National
Center for
Special
Education
Research

Missions of the Research Centers

NCER
supports rigorous research that addresses the nations
most pressing education needs, from early childhood to
adult education

NCSER
sponsors a rigorous and comprehensive program of special
education research designed to expand the knowledge and
understanding of infants, toddlers, and students with or at
risk for disabilities from birth through high school

ies.ed.gov

IES Grant Programs:

Research Objectives
Develop or identify education interventions
(practices, programs, policies, and approaches)
that enhance academic achievement
that can be widely deployed

Identify what does not work and thereby encourage

innovation and further research

Understand the processes that underlie the

effectiveness of education interventions and the
variation in their effectiveness
ies.ed.gov

Connecting Research,
Policy and Practice

Education Research & Development

Centers (84.305C)

ies.ed.gov

Research & Development Centers:

Purpose
Contribute to solving U.S. education problems,
improve the education system, and increase student
achievement through

Research
Development
Evaluation
National leadership

ies.ed.gov

Research & Development Centers:

Activities
Focused research program
Contribute to solution of specific education problem
within the Centers topic
Generate new knowledge and theories
Detailed plan of focused program required
75% of activities should be for focused program of
research

ies.ed.gov

Research & Development Centers:

Activities
Supplemental activities
Meetings, smaller-scale studies
Work cooperatively with the Institute to select and design
No detailed plan needed
Show capacity for conducting supplemental activities
Set aside up to 5% of budget annually

ies.ed.gov

Research & Development Centers:

Activities
Leadership and outreach
Engage with researchers, practitioners, and policymakers
Website with reports, papers, etc.
Innovative uses of technology (e.g., webinars, podcasts)
Describe audiences and activities
Show capacity for leadership and outreach

ies.ed.gov

Connecting Research,
Policy and Practice

Research & Development Center

on Knowledge Utilization

ies.ed.gov

10

Purpose
Research
Develop tools for observing and measuring research use in
schools
Illuminate the conditions under which practitioners use
researcher and factors that promote or inhibit research
use in schools
Identify strategies that make research more meaningful to
and impactful on education practice

Leadership and outreach

Demonstrate effective means of knowledge transfer to
improve school performance and student outcomes
ies.ed.gov

11

Background
Findings from high-quality research and evaluation
on impact of education policies and interventions
Concern that this information is not used by
practitioners
To adopt interventions with beneficial effects on students
To discard interventions with adverse or null effects

ies.ed.gov

12

Background: Potential Causes

Misalignment between research questions (e.g.,
does it work?) and practitioners concerns (e.g., how
do I use it with my students?)
Misunderstanding of social context of practitioners
Research use is multilevel phenomenon (e.g. involves
districts, school leaders, and teachers)

Need new ways of working with practitioners

Partnerships
Bidirectional model of relationship
Intermediary organizations
ies.ed.gov

13

The Centers Focused Program of Research

One measurement study
Develop (or modify) and validate tools for observing and
measuring research use in schools and school districts

Two descriptive studies

Understand the conditions under which research is used
and factors that promote or inhibit research use in schools
Identify practices that are associated with greater use of
research in schools and school districts

ies.ed.gov

14

The Centers Focused Program of Research

Option to conduct Development Study or Efficacy
Study
As alternative to one or both descriptive studies

To develop a pilot intervention to improve research

use in schools
Or to conduct an efficacy study to assess the impact
of a particular practice on the use of research in
schools or school districts
Study may range along a continuum of rigor
ies.ed.gov

15

Significance of Center
Significance section at start of Research Narrative
Discuss the significance of your proposed Center
Your vision of the Center
What problems it is intended to address
How your set of research and leadership activities will
work together to promote the use of scientific education
research in schools and districts

Discuss significance of three studies in focused

research program
Describe how you will engage researchers and
practitioners in the work and communication of
findings
ies.ed.gov

16

The Measurement Study:

Research Questions
What are the right indicators to know if school staff
are using education research in their decisionmaking?
How to capture variation in research use across
schools, school district, and over time?
You may expand or add additional research questions
(and should provide justification for doing so)

ies.ed.gov

17

The Measurement Study:

Methodological Requirements
Methods of developing (or modifying) and validating
assessment(s)
Approach for gathering psychometric evidence
Sample and setting
Characteristics, size, and analytic adequacy of the
sample, including justification for inclusion and
exclusion criteria
Timeline

ies.ed.gov

18

Descriptive Study 1:
Research Questions
What role do State and local education agencies play in making
research available to school administrators, faculty, and staff?
What role do school leaders play?
How do education agencies, school leaders, and teachers obtain
and process scientific information?
How do social networks within schools and school districts
appear to support or hinder research use?
What are the major incentives or disincentives for practitioners
to better understand and apply education research?
What contextual factors seem to affect the use of research in
schools and school districts?
You may expand or add additional research questions (and
should provide justification for doing so)
ies.ed.gov

19

Descriptive Study 2:
Research Questions
Do certain communication strategies used by researchers or
intermediaries correlate with greater use of research by
practitioners?
Is the presence or absence of intermediaries associated with
greater use?
Are particular technological innovations associated with
greater use of research by practitioners?
Are partnerships between researchers and practitioners
associated with greater research use? To what extent do
researchers perceive partnerships with practitioners as an
enhancement or burden to their work?
You may expand or add additional research questions (and
should provide justification for doing so)
ies.ed.gov

20

Descriptive Studies:
Methodological Requirements
Population from which youll select your sample and
sampling approach
Measures
Data collection and analysis plan
Timeline

ies.ed.gov

21

Optional Development or Efficacy Study:

Methodological Requirements
Clear description of intervention to be developed or
evaluated, including a theory of change
Research questions
Sample and setting
Study designs may fall along a continuum of rigor but
efficacy studies should meet What Works
Clearinghouse evidence standards
Detailed power analysis
Read Goal 2 and Goal 3 of 84.305A RFA for more
details on methodological requirements
ies.ed.gov

22

Optional Development or Efficacy Study:

Methodological Requirements
Measures, including of fidelity and comparison group
practices
Key moderators and mediators
Plan to analyze implementation fidelity and
comparison group practices
Data analysis plan
Timeline

ies.ed.gov

23

Leadership & Outreach

Innovative approaches to fostering dialogue among
practitioners and researchers
You may coordinate leadership/outreach with annual
conferences to save time/money
Institute encourages website
Institute open to learning how other applied social
sciences encourage research use among practitioners

ies.ed.gov

24

Maximum Awards

5 years
$5,000,000
Maximum of one award
Focused program of research must comprise at least
75% of the total budget

ies.ed.gov

25

Connecting Research,
Policy and Practice

Research & Development Center

on Standards in Schools

ies.ed.gov

26

Motivation
Standards being rapidly deployed
>40 states currently implementing the Common Core State
Standards
Others implementing some form of college- and careerready standards
Little known about the actions that districts are taking to
implement the standards and the influence that the
standards are having in schools and classrooms

ies.ed.gov

27

Motivation
Need for a Network
To coordinate a research agenda on standards that
addresses the needs of practitioners and policymakers for
timely information
To coordinate efforts of researchers from multiple areas of
expertise

ies.ed.gov

28

Purpose
Timely and accurate information
The Center will be a source for unbiased information
regarding implementation of the standards.

Leadership, Outreach and Networking

The Center will bring together researchers, practitioners,
and policymakers who are actively engaged in standardsrelated research, practice, and policy development.

Assess Early Outcomes of College- and CareerReady standards

The Center will be at the forefront of assessing whether
the standards are making a difference, and for whom.
ies.ed.gov

29

Main Focus Areas for the Center

The Center must carry out three focused activities:
1. An Implementation Study
2. A Longitudinal Outcomes Study
3. A Leadership and Outreach Program
The Center may also elect to conduct a Measurement
Study.

ies.ed.gov

30

Implementation Study Goals

How is implementation taking place?
What strategies are SEAs and/or LEAs taking to implement
college- and career-ready standards?
How do these strategies vary across sites?
How are implementation strategies seeking to address the
needs of key student groups including ELLs and students
with disabilities?
What changes, if any, are college- and career-ready
standards bringing about in schools and classrooms?

ies.ed.gov

31

Implementation Study
RFA encourages an expansive view of implementation,
including several key dimensions, such as:

State, district and school leader support for implementation

Professional development for teachers
Curriculum materials
Teacher practice
Student assessment
All of the above as they pertain to ELLs and students with
disabilities
Broader contextual factors that may affect implementation

ies.ed.gov

32

Longitudinal Outcomes Study

OPTIONS: Exploratory or Efficacy Study
Twin goals
1. Assess the relation between standards implementation
and student outcomes
2. Assess the factors that explain variation in student
performance across sites

ies.ed.gov

33

Possible Research Questions

Should reflect the broad objectives for the Center
As CCRS standards are implemented
Do student outcomes improve or worsen as CCRS are
implemented, including for sub-groups?
Do achievement gaps between sub-groups improve or
worsen over the study period?
Which factors (implementation or contextual) moderate
the relation between the standards and student
outcomes?
ies.ed.gov

34

Leadership & Outreach Activities

The Center should be a source of objective information
for a variety of audiences from federal policymakers
to local audiences.

Minimum requirements
Provide specifics about the targeted audiences and
strategies to reach those audiences
Maintain a website
Convene researchers and stakeholders to discuss research
findings and needs
ies.ed.gov

35

Application Sections
Significance of the Focused Program of Research
Research Plan for the Focused Program of Research
Plans for Other Center Activities (Leadership &
Outreach)
Management and Institutional Resources
Personnel

ies.ed.gov

36

Application:
Significance Section
Opportunities and Challenges posed by college- and
career-ready (CCR) standards
Major research issues in need of attention
How will your approach address these issues, and the
goals of the Center?

Conceptual Framework and Literature Review

What is your conceptual framework?
How is it grounded in theory on standards?
How will it guide research design, data collection and
analysis plans?
ies.ed.gov

37

Application:
Significance Section
Sites
Describe geography, school setting, student population,
implementation of standards
How will findings inform policy discussions and analysis?
Describe partnerships with SEAs, LEAs
Letters of agreement (Appendix D)

Research Agenda
Areas of focus for your Implementation Study
Research questions for your Outcomes Study
Justification for measures
IF you propose measure development

ies.ed.gov

38

Application:
Research Plan
Implementation Study
Sample selection, including population and sampling
method
Measures you will use
Data collection strategy
Analysis plan
Timeline

ies.ed.gov

39

Application:
Research Plan
Longitudinal Outcomes Study
Exploration Study

Population and sample selection

Measures you will use; student outcomes
Data collection and analysis plans
Timeline

Efficacy Study

All of the above

Research design to produce impact estimates
Power analysis
Data analysis procedures
Dissemination plan

ies.ed.gov

40

Application: Research Plan

Optional Measurement Study
Provide a rationale for a new measure or substantial
modification to an existing measure

Instructional practice
Classroom activities
Opportunities to learn
Alignment of instruction, classroom activities, or curriculum
materials to college- and career-ready standards

ies.ed.gov

41

Application:
Leadership and Outreach
Target Audiences: Who will you be speaking to and
what will be the key topics for each group?
National audiences (including federal policymakers and
national organizations)
Local audiences where the research will be conducted

Role of practitioners in reviewing findings

Website (required)
Other Dissemination Venues
Products for downloading
Social media
ies.ed.gov

42

Maximum Awards

5 years
$10,000,000
Maximum of one award
Focused program of research must comprise at least
75% of the total budget

ies.ed.gov

43

Connecting Research,
Policy and Practice

Research & Development Center

on Virtual Learning

ies.ed.gov

44

Virtual Learning Lab (VLL):

Background
Many open questions:
Whether education technologies lead to meaningful
improvements in teaching and learning
How big data can be used to improve teaching and
learning

Infrastructure is needed to:

Support the analytic and instructional use of the data
Store and share these data with others, and to mine data
generated from the use of education technologies
ies.ed.gov

VLL: Broad Purpose

Evaluate and improve instructional practices, content,
and/or learning tools offered by one or more widely used
online instructional delivery platforms
Focus: Improving outcomes of students in K-12, particularly lowincome and low-performing students
One platform must be for K-12 students, but additional
platforms can be for either K-12 or postsecondary students

Address how large amounts of data generated within

these platforms can be used to improve teaching and
learning
ies.ed.gov

VLL: Broad Purpose

Inform stakeholders about the use of "big data" for
education research and practice
Create a hub
Offer workshops and other activities

ies.ed.gov

Definition of
Widely Used Online Instructional Delivery Platform
The technological infrastructure used to:
Deliver any of a number of education technologies, and
Collect large amounts of information about students

Platform(s) must be used across multiple states, in

the majority of districts in a single state, or in the
majority of schools in one or more large districts

ies.ed.gov

Definition of
Widely Used Online Instructional Delivery Platform
The online instructional delivery platform(s) you
choose may be used to provide any of the following:
Online courses offered by a state, school district, or
postsecondary institution
Online courses with open enrollment (i.e., MOOCs)
Online schools or certificate/degree programs
Intelligent tutors
Blended learning models (combine online/classroom
instruction)
Flipped models (instruction online and homework/other
activities during class)
ies.ed.gov

Examples of Research Questions

How can online and blended learning be used to address the
instructional needs of learners across the spectrum typical,
advanced, and struggling more effectively?
How can online instruction be designed and implemented to
promote the learning and retention of material, not just in the
short term, but also over longer school-relevant time periods?
How can online courses be designed so that students are more
likely to persist in and complete the instruction?
How can online instruction be targeted and delivered to meet
the needs of low-income students, and to reduce achievement
gaps between higher- and lower-performing students?
ies.ed.gov

50

Examples of Research Questions

What data, metadata, analytical tools, and models are most
useful for understanding and improving student learning?
How can "big data" be used to predict students performance
and help them receive the instruction and services they need?
What practices, protocols, and policies should researchers and
Institutional Review Boards (IRBs) be attentive to as they strive
to protect students privacy?
Which research and academic skill sets are needed to
effectively conduct this type of work, and what training can be
employed so others can develop the necessary skills?

ies.ed.gov

Important Partnerships
The Institute strongly encourages partnerships
between:
Researchers
Platform developers
Practitioners within states, school districts, and/or colleges
and universities who use online instructional delivery
platforms.

Must attach letters of agreement from all the key

partners in Appendix D
ies.ed.gov

Project Narrative Requirements

Significance of the Focused Program of Research
Research Plan for the Focused Program of Research
Leadership & Outreach Activities

ies.ed.gov

Significance of the
Focused Program of Research
Explain vision for the VLL

Identify problems or issues that the VLL will address

Review of research literature
Provide data to support your argument (if possible)

Discuss significance of studies proposed in the

Research Plan
Explain how you will ensure generalizability of findings

ies.ed.gov

Significance of the
Focused Program of Research
Must identify at least ONE widely used online
instructional delivery platform
Platform must be intended for use by students in an
authentic K-12 education setting
Additional platforms may be intended for use either by K12 or with postsecondary students

ies.ed.gov

Overview of
Remainder of Presentation
Components of the focused program of research
Experimental studies (min = 2)
Measurement studies (min = 1)

"Big data"
Leadership and outreach activities

ies.ed.gov

Focused program of research:

Experimental Studies
At least TWO experimental studies
More can be proposed, depending on their scope

Focus
Examination of strategies to improve instructional
practices, content, and/or learning tools within online
delivery platforms

Particular concern
Differentiation for individual learning needs
ies.ed.gov

What can be studied experimentally?

Intervention that are
Fully developed?
still in the development process?

At least ONE study should be designed with the

intention of improving an education technology that
is in wide use

ies.ed.gov

What can be studied experimentally?

Affordances in a digital environment
Easy, quick introduction and changes, based on findings
Support of timely decision making

Comparison study of different iterations or versions

of the intervention

ies.ed.gov

Measurement study (min = 1)

Goals
Explore/create valid and reliable measures
Determine which measures useful in differentiating
student learning needs
Identify mediating variables
Examine strategies to prepare data for user-friendly
consumption

ies.ed.gov

Research Plan:
Experimental Studies
Purpose: To help education technology developers
and education practitioners make improvements in
the instructional practices, content, and/or learning
tools they are using
See enumerated list (RFA, p. 26)

ies.ed.gov

Research Plan:
Data Collection
Describe
Sources of data
Digital interactions
Administrative records
Surveys, Observations, and Interviews

Methodology
Sampling strategy
Data collection plans
Analytic approach

Rationale for each choice

ies.ed.gov

Studying a fully developed technology

What are the critical questions?
See enumerated list (RFA, p. 27)

ies.ed.gov

Research Plan:
Measurement studies
Purpose: What is the studys justification?
See enumerated list (RFA, p. 27)

ies.ed.gov

Developing and validating NEW measures

Describe iterative process
Field testing procedures & processes for item
revision
Validation activities, including types of evidence
to be used for reliability and validation for the
specified purpose, populations, and contexts

See Measurement Goal section in the RFA

84.305A
ies.ed.gov

Concerns with "big data" and education

Issues, concerns and implications on multiple levels
Practical, Ethical, and Legal

Confidentiality, protection of PII

(personally identifiable information) from unauthorized
disclosure

Data access
Who should have access to data? For which purposes?

Sharing, linking
Security rules and protocols

ies.ed.gov

Concerns with "big data" and education

Negative side-effects of labeling
e.g., hes a slow learner

Unintended consequences

ies.ed.gov

Leadership & Outreach Activities

Role of the VLL: Function as a reliable source
For Whom?
Technical: Researchers & developers
Broader audience: Teachers, administrators, policymakers,
general audience

ies.ed.gov

Dissemination
What: Publications, products
How: Describe efforts planned efforts at
dissemination
Website (at a minimum)
Other new media (podcasts, video, webinars) that are
relatively inexpensive yet can reach a large audience

ies.ed.gov

Hub for Training & Research

(some ideas)
Researchers, developers, and practitioners will come
together (virtually and in person)
Discuss research goals and methods related to online
learning
Review emerging findings
Foster new research partnerships and collaborations:
(e.g., hosting non-Lab researchers, developers, etc., to work on Lab
projects)

Provide training and education:

Host summer workshops, conduct courses for graduate students,
non-Lab researchers, practitioners, and others

ies.ed.gov

Hub for Training & Research

(some ideas)
Organizing visits by state or school district leaders
to demonstrate how education technologies are being used,
improved, and having an effect

Your suggestion(s) here: ______

ies.ed.gov

Maximum Awards

5 years
$10,000,000
Maximum of one award
Focused program of research must comprise at least
75% of the total budget

ies.ed.gov

72

Connecting Research,
Policy and Practice

General Requirements

ies.ed.gov

73

Additional Sections
of the Research Narrative for All Centers
In your 35-page Research Narrative, you are to
discuss

Significance
Research Plan for the Focused Program of Research
Other Center Activities (Leadership and Outreach)
Management and Institutional Resources
Personnel

Weve discussed the first three and now well move

to the last two
ies.ed.gov

74

General Requirements:
Management & Institutional Resources
Describe plans and procedures for overall
management of the Center and its activities
Show capacity of supplemental activities and
leadership and outreach
If plan is to work in schools or other education
delivery settings in first year, must include letter of
agreement from education organization(s)
If developer/distributer of program or intervention is
part of Center, describe role and how objectivity will
be maintained
ies.ed.gov

75

General Requirements:
Personnel
Expertise in content area(s)
Methodological expertise
Sufficient experience working with education
delivery settings
Capacity for supplemental activities
Experience relevant to national leadership
Explain why proposed Center staff is qualified to fulfill
leadership role

ies.ed.gov

76

Important Dates and Deadlines

Application
Deadline
August 7,
2014
4:30:00 PM
DC Time

ies.ed.gov

Letter of
Intent Due
Date
June 5, 2014

Application
Package
Posted
June 5, 2014

77

Start Dates

July 1, 2015
to
Sept 1, 2015

Peer Review Process

Applications are reviewed for compliance and
responsiveness to the RFA
Applications that are compliant and responsive are
assigned to a review panel
Two or three panel members conduct a primary
review of each application
Most competitive applications move to full panel

At panel meeting, the most competitive applications

are reviewed by full panel
ies.ed.gov

78

Notification
All applicants will receive e-mail notification that
the following information is available via the
Applicant Notification System (ANS):
Status of award
Reviewer summary statement

If you are not granted an award the first time,

consider resubmitting and talking with your
program officer

ies.ed.gov

Questions?
http://ies.ed.gov/funding
Knowledge Utilization
Becky McGill-Wilkinson (Rebecca.McGill@ed.gov)

Standards in Schools
James Benson (James.Benson@ed.gov)
Virtual Learning
Jonathan Levy (Jonathan.Levy@ed.gov)
Erin Higgins (Erin.Higgins@ed.gov)
ies.ed.gov

80