Instructional Science (2006) 34: 63–87 !

Springer 2006
DOI 10.1007/s11251-005-6922-4

Patterns of instructional-design factors prompting reflective

thinking in middle-school and college level problem-based
learning environments

HAE-DEOK SONG1,*, BARBARA L. GRABOWSKI2, TIFFANY A.

KOSZALKA3 & WILLIAM L. HARKNESS4
1
University at Albany, State University of New York, Education 115B, Albany,
NY12222, USA; 2Pennsylvania State University, 315 Keller Building, University Park,
PA 16802, USA; 3Syracuse University, 336 Huntington Hall, Syracuse, 13244NY,
USA; 4Pennsylvania State University, 326 Joab Thomas Bldg., University Park, PA
16802, USA (*Author for correspondence, e-mail: hsong@uamail.albany.edu)

Received: 8 October 2003; accepted: 4 May 2005

Abstract. Reflective-thinking skills are important in problem-based learning environ-

ments as they help learners become deeply engaged in learning. The literature suggests
several instructional-design factors (e.g., environment, teaching methods, scaffolding
tools) that may prompt reflection in learners. However, it is unclear whether these
factors differ based on age or developmental stage. The results of this study indicate that
middle-school students perceive the learning environment factor as more important to
prompting their thinking, while college students perceive the scaffolding methods factor
as more important. While the elements clustered into two factors, most college students
disagreed with their helpfulness in prompting reflective thinking, a finding opposite to
that obtained for middle-school students. Different patterns were also found between
learners’ perceptions of the most helpful elements within each factor. Based on these
results, suggestions are given for designing developmentally and age-appropriate PBL
learning environments that support reflective thinking.
Keywords: developmental differences, factor analysis, problem-based learning, reflective
thinking

Introduction

A problem-based learning (PBL) environment provides learners with

instructional mechanisms that can increase their higher-order thinking
skills while they are exploring authentic and ill-structured problems,
participating in social interactions, and receiving coaching from peers
and teachers (Albanese & Mitchell, 1993; Hmelo & Lin, 2000). How-
ever, PBL involves many cognitive challenges. Learners are challenged
to understand a problem situation, clarify the causes of the problem,
64

decide on important facts to be investigated, and generate hypotheses

for a solution. To appropriately solve a problem in a PBL environ-
ment, it is important that learners reflect on their understanding of an
issue, acquire new knowledge to help in developing a solution, and
think about how their new knowledge can be used to address the
problem situation. Reflecting on the problem situation therefore, helps
learners generate concepts and abstractions and develop the new
knowledge needed to generate a solution to the given problem (Bar-
row, 1998). If learners lack these reflective-thinking skills, the PBL
environment can be chaotic rather than supportive of the construction
of meaningful knowledge. Thus, successful PBL environments prompt
learners and help them better develop reflective-thinking skills.
Research suggests that various design elements in the learning envi-
ronment can prompt reflective thinking (Andrusyszyn & Daive, 1997;
Lin et al., 1999; Griffith & Frieden, 2000). For instance, ill-structured,
authentic, and complex tasks are known to promote reflective think-
ing. The features of the learning environment help learners think
reflectively by prompting them to investigate complex problems using
multiple forms of information to generate a solution (Stepien & Pyke,
1997). Teaching methods such as inquiry-oriented and explanation-ori-
ented teaching prompt reflective thinking in PBL in different ways
(Virtanen et al., 1999). An inquiry-oriented method facilitates reflective
thinking by asking reflective questions, while an explanation-oriented
method facilitates reflective thinking by guiding learners to reflect di-
rectly on important concepts related to the given problem (Moon,
1999; Virtanen et al., 1999). Creating flexible, active, and student-cen-
tered PBL environments is also important in prompting reflective
thinking. The design elements that make the learning-environment ac-
tive and student-centered include wait-time, which allows learners to
think before answering, learner-controlled instruction, and cooperative
and collaborative learning (Rowe, 1974; Williams, 1996; Aldred &
Aldred, 1998). Finally, scaffolding tools can prompt reflective thinking
during PBL. Andrusyszyn and Daive (1997) report that reflective jour-
nal writing is key in facilitating reflection in graduate students. Others
suggest that question prompts and concept-mapping activities are tools
that can be used to prompt reflective thinking (Barrow, 1998; Griffith
& Frieden, 2000; Kinchin & Hay, 2000). Although it is widely believed
that incorporating these design elements into a PBL environment can
support reflective thinking, little research has been conducted to inves-
tigate learners’ perceptions of the helpfulness of these design elements.
While these design elements have been expected to increase the effec-
 65

tiveness of a PBL environment, learners do not always perceive all

such design elements as effective, efficient, or motivating. In addition,
it is not clear which fundamental types of design element are perceived
by learners as most helpful in prompting their reflective thinking in
PBL environments. Thus, understanding learners’ perceptions of the
underlying factors that help them reflect can inform the practice of
developing more efficient PBL environments.
Initial studies found three factors that prompted reflective thinking
in a science learning environment by investigating middle-school stu-
dents’ perceptions (Koszalka et al., 2002; Song et al., in press). These
factors included teaching methods, learning environments, and scaf-
folding tools. However, these results may be unique to the science
learning environment, and not to the PBL environment. Therefore,
one of the goals of this study was to investigate whether middle-school
students would perceive the same factors as helpful to their reflective
thinking when these factors were experienced in a PBL environment.
An important variable that may influence learner perceptions of the
helpfulness of these design elements is their developmental stage in
terms of reflective thinking (Craig, 1983; Crain, 2000). According to
King and Kitchener (1994), reflective-thinking skills are the outcome
of a developmental progression resulting from interactions between the
individual’s conceptual skills and an environment that promotes or
inhibits these skills. The authors assume seven development stages of
reflective thinking and divide them into three main sets: an early stage
consisting of pre-reflective thinking, a middle-stage consisting of quasi-
reflective thinking, and an advanced stage in which reflective thinking
occurs spontaneously. They suggest that middle-school students are
mostly in the pre-reflective-thinking stage, while college students are at
the middle or advanced stages of reflective thinking. This developmen-
tal progression suggests that learners at the earlier stage of pre-
reflective thinking may need more or different instructional-design
prompts to help them think reflectively and progress into the middle
and advanced stages. For example, Koszalka et al. (2002) report a sig-
nificant difference, based on grade level, in middle-school students’
perceptions of the elements that help to prompt reflective thinking.
Younger students perceive the helpfulness of a scaffolding-tool element
as more important than do older students. This may indicate that
learners at different age levels are at different stages of reflective think-
ing and may perceive the usefulness of reflective-thinking prompts
differently. However, it is unknown whether this is indeed the case, for
middle-school and college students, for example.
66

The primary focus of this study was to extend Koszalka et al.’s

(2002) study by comparing the patterns of middle-school and college
level learners’ perceptions of the helpfulness of factors designed to
prompt reflective thinking in PBL environments. The following re-
search questions guided this study:
1. Which instructional-design elements load together as factors that
middle-school students or college students perceive as helpful in
prompting reflective thinking in PBL environments?
2. Which of the identified factors are perceived as most helpful in
prompting reflective thinking in PBL environments?
3. Which design elements within each identified factor are perceived
as most helpful in prompting reflective thinking in PBL environ-
ments?
4. Are there differences based on developmental stage in students’ per-
ceptions of the identified factors and design elements in PBL envi-
ronments?

Methods

Participants and context

Middle-school and college level students’ perceptions of the design fac-

tors that prompt reflective thinking in PBL environments were exam-
ined. The participants in the study of middle-school learners consisted
of 122 students: 70 boys, 51 girls, and 1 whose gender was not identi-
fied, from three public middle schools in a northeastern state of the
United States. These subjects came from intact groups in six different
classrooms representing the 6th (n=83), 7th (n=28), and 8th (n=11)
grades. The participants in the study of college-level learners consisted
of 749 students attending 10 different sections of an introductory sta-
tistics course at a large northeastern research university in the United
States. The sample included 464 (62%) females and 285 (38%) males.
The participants included freshmen (16%), sophomores (50%), juniors
(30%), and seniors (4%). These students represented more than 30 de-
gree programs from the College of Business Administration, Educa-
tion, Health and Applied Sciences, and Liberal Arts.

Measurement instrument

A survey developed by Koszalka et al. (2002) consisting of 10 questions

for measuring the perceived helpfulness of factors that prompt reflec-
 67

tive thinking, was used (See Table 1 for the question items used in the
instrument). Each question dealt with one of the instructional-design
elements predicted, from the literature, to prompt reflective thinking.

Table 1. Factor loading of instructional-design elements in the first result (middle-

school students)

Item Factor Loading C

Factor 1 Factor 2

8. Drawing pictures to illustrate my understanding 0.81 0.65

of a topic helps me think more about what I am
studying
9. Writing about my understanding of a topic 0.76 0.63
helps me think more about what I am studying
3. When my teacher explains how to solve 0.74 0.69
difficult tasks it helps me think more about
what I am studying
10. Answering questions about a topic 0.62 0.46 0.59
helps me think more about what I am studying
4. When my teacher asks me how to solve 0.58 0.47 0.57
difficult tasks it helps me think more about
what I am studying
2. Working on activities in class related to 0.78 0.63
real problems on earth or in our society
helps me think more about what I am studying
6. Having time to think about a 0.77 0.64
question before answering helps me think
more about what I am studying
5. Working with partners during classroom 0.71 0.51
activities helps me think more about what
I am studying
1. Working on activities in class that have 0.44 0.62 0.57
many different answers helps me think more
about what I am studying
7. Having freedom in class to explore topics 0.42 0.45 0.38
I am interested in helps me think more about
what I am studying

Eigen value 4.74 1.14

The proportion of variance explained 47.34 11.35

Note. C: Communalities.
68

The design elements included the ill-structuredness and authenticity of

tasks; explanations and questions given by the teacher; supportive and
flexible learning-environment elements, including collaborative learn-
ing, wait time, and learner control; and scaffolding tools such as con-
cept-mapping, reflective writing, and reflective question prompts
(Rowe, 1974; Williams, 1996; Andrusyszyn & Daive, 1997; Stepien
& Pyke, 1997; Aldred & Aldred, 1998; Barrow, 1998; Moon, 1999;
Virtanen et al., 1999; Griffith & Frieden, 2000; Kinchin & Hay, 2000).
For each item on the survey, students responded using a five-point
Likert scale from strongly agree (5) to strongly disagree (1).
The Cronbach alpha reliability for the 143 middle-school students
in Koszalka et al. (2002) was reported as 0.890. The resulting
Cronbach alpha reliability for the data collected from the middle-
school students in this study was 0.880, while the Cronbach alpha
reliability for the college students was 0.706.

PBL environments

The middle-school and college level students participated in different

PBL environments. The middle-school students participated in a pro-
ject called Kids as Airborne Mission Scientists (KaAMS), designed
and developed by the researchers to inspire student interest in science,
mathematics, technology, and geography. In this PBL environment,
students were first presented with ill-structured, authentic problem
situations involving basic science concepts related to aeronautics and
remote sensing. The problem topics included definitions of aeronautics
and remote sensing, classification of volcanoes, Bernoulli’s principle of
lift, weather patterns and flight, latitude and longitude, and frequency
and wavelength. To solve the problems, the students needed to be able
to (a) identify the problem situation, (b) propose ideas and search for
information, (c) collect and analyze data, and (d) propose solution
ideas. A variety of resources, mainly NASA web resources about aero-
nautics and remote sensing, were provided. Several reflection strategies
were incorporated into all the KaAMS PBL lesson plans to stimulate
students to use reflective-thinking actively and persistently. From their
classroom observations, the researchers in this study determined that
the students experienced all the key design elements supporting reflec-
tive thinking during their investigations. These included the use of
ill-structured and authentic problem tasks, inquiry- and explanation-
oriented teaching methods, flexible learning environments driven by
wait-time, learner-controlled instruction and collaborative learning,
 69

and scaffolding tools containing question prompts, reflective journals,

and concept maps (Rowe, 1974; Williams, 1996; Andrusyszyn &
Daive, 1997; Stepien & Pyke, 1997; Aldred & Aldred, 1998; Barrow,
1998; Moon, 1999; Virtanen et al., 1999; Griffith & Frieden, 2000;
Kinchin & Hay, 2000).
The college students were enrolled in a statistics course designed to
enhance their problem-solving abilities in terms of using statistics in
their college work and understanding of the role of statistics in their
lives. In this PBL environment, students were presented with ill-struc-
tured problem-solving activities involving introductory statistics. The
statistical topics included descriptive statistics, normal distributions,
null and alternative hypotheses, correlation and regression, and analy-
ses of variance. To further investigate the topics, the students were
asked to collect and analyze more data on the Internet and to pro-
pose a group report in their computer-lab periods. For example, the
students were asked to predict the length of one’s foot knowing the
length of one’s forearm in order to investigate the concept of regres-
sion and correlation. They worked in collaborative groups with data
files containing measurements of these two variables, testing the
hypothesis that two lengths were the same, on average, in the lecture
period. They then worked in computer labs to find examples of an
experiment, explain how it was conducted and what the response
variable was, and describe it in terms of what they had learned in
class. A course website was also used to facilitate student participa-
tion. Essential reflection strategies were incorporated into the activi-
ties to facilitate students’ reflective-thinking skills. These included
authentic and ill-structured problem situations, student-centered and
teacher-centered activities, collaborative work, and guiding journals
and questions. Observations of the class ensured that all the design
elements in the KaAMS PBL environment were also applied in this
course to help students think reflectively.
We acknowledge that direct comparison between middle-school
students’ perceptions and college students’ perceptions is limited
because the two groups participated in different learning environ-
ments in this study. However, both environments were designed based
on the essential design principles of PBL (Barrow, 1998; Hmelo &
Lin, 2000; Land & Greene, 2000): (a) the learning environment sup-
ports student-centered investigation of an ill-structured and complex
problem; (b) students identify knowledge deficits and solve a problem
with their prior knowledge; and (c) technology supports learners in
collecting, analyzing, and integrating information. We believe that
70

these common characteristics of the PBL environment enabled us to

examine patterns in the factors that prompted each group’s reflective
thinking in a PBL environment and to descriptively compare them.

Procedure

The middle-school students’ perceptions were examined after they had

completed the KaAMS PBL lessons. Each class met in a classroom
equipped with laptop computers two or three times a week for 45 min
at a time. The average amount of time these students worked on the
KaAMS PBL lessons was 6 months. The students’ exposure to the
PBL environment over this extended period of time ensured the sta-
bility of their perceptions of the factors in question. A survey was
administered and collected by their classroom teacher at the end of
the 6 months.
The college students’ perceptions were examined after they had com-
pleted a 15-week course. The course met three times a week for 50 min
at a time. There was one classroom lecture and two computer labs each
week. These students’ exposure to the PBL environment over this ex-
tended period of time also ensured the stability of their perceptions of
the factors in question. A survey was administered and collected by the
instructor and teaching assistants at the end of the course.

Data analysis

Since we wanted to develop the factor structure of elements that

prompt reflective thinking, the first step was to conduct an explor-
atory factor analysis (EFA) employing principal component analysis.
An orthogonal (varimax) rotation method was chosen because the
correlation between factors turned out to be relatively low (r<0.30)
from the component correlation matrix, so that the researchers in this
study could assume the independence of the factors. Further analyses
of the data followed to examine the characteristics of the instruc-
tional-design factors that emerged from the EFA. To identify the
most helpful factor, a one-way within-subject ANOVA was run. The
one-way within-subject ANOVA was chosen to examine differences
between mean of factors instead of a paired sample t-test, in order to
control the Type I error. To identify the most helpful element, the
one-way within-subject ANOVA was run again. This tested for differ-
ences between the means of the design elements. Finally, the results of
the one-way within-subject ANOVAs for the two groups were com-
 71

pared to describe the differences according to the student’s age level/

developmental stage.

Results

Factor structures prompting reflective thinking in PBL environments

Before conducting the EFA, we carried out a preliminary analysis to

find out whether the matrix used in this study was appropriate for the
factor analyses. Kaiser–Meyer–Olkin’s (KMO) Measure of Sampling
Adequacy (MSA) and a Bartlett Test of Sphericity were calculated to
determine the appropriateness of the data matrix. The MSAs in this
study were 0.875 for the middle-school students and 0.809 for the col-
lege students. This indicates a sound pattern of correlation. Bartlett’s
measures show significant correlations among the variables of the cor-
relation matrix. For the middle-school students, the measure was v2
(45, N=122)=465.37, p<0.01 while for the college students, the
measure was v2(45, N=749)=848.25, p<0.01. The preliminary analy-
sis supports the evidence that the factor-analysis assumption for the
matrix used in this study is tenable. In addition, tests of homogeneity
of variance (Kolmogrorov–Smirnov Test, Shapiro–Wilks tests, and
the Levene’s Test) also showed that the assumption of equal variance
was met at the 0.05 alpha level.
A principal component analysis with orthogonal (varimax) rotation
was used to examine the factor structure of the 10 items. The number
of factors extracted was based on three criteria: (a) the eigenvalue, (b)
the percentage of variance, and (c) a scree test (Hair et al., 1995). The
decision to retain items was based on the following two criteria: (a)
an item structure coefficient greater than 0.40 and (b) a minimum gap
of 0.10 between salient coefficients on multiple factors (Nunnally,
1978).
In response to the first research question (Which instruc-
tional-design elements can be grouped together as factors that middle-
school students perceive as helpful in prompting reflective thinking?),
two factors emerged with eigenvalues greater than 1, which together
accounted for 61.7% of the variance in the data for the middle-school
students and for 39.0% of the variance in the data for the college
students in the final model.
The initial results of the study of middle-school students’ percep-
tions showed that five items loaded to factor 1 and five items loaded
72

to factor 2 (See Table 1). Interestingly, item 7, learner control, was

prominent with factor 2 (0.45) but also had a strong correlation with
factor 1 (0.42). Because the gap between coefficients was not greater
than 0.10, it was removed, and the model ran again. The result of the
final model also shows that five items loaded to factor 1: concept-
mapping, reflective writing, teacher explanations, reflective question
prompts, and teacher questions (See Table 2). At first, these design
elements appeared to relate to scaffolding tools. However, the items,
teacher questions, and teacher explanations relating to teaching meth-
ods also loaded with other tools that help learners to structure their
thinking. From this clustering, it appears that rather than scaffolding
tools, the category broadens into facilitative scaffolding methods, that
is, methods that help structure reflective thinking, including actions
taken by both teacher and student. Four items loaded to factor 2:
authentic tasks, wait time, collaborative learning, and ill-structured
tasks. The main characteristics of factor 2 appeared to relate to a con-
structivist learning environment because the item with the greatest
factor loading, authentic tasks, is the most critical design element in
constructivist learning environments (Honebein et al., 1993). A con-
structivist learning environment is defined by this data as being flexible,
collaborative, and driven by authentic and ill-structured problem-
solving challenges.
The study of college students’ perceptions also identified two latent
factors. Six items loaded to factor 1: teacher questions, ill-structured
tasks, teacher explanations, reflective writing, reflective question
prompts, and concept-mapping (See Table 3). The items that loaded
to factor 1 appeared to be defined by task-related scaffolding methods.
Four items loaded to factor 2: learner control, wait-time before
answering a question, collaborative work, and authentic tasks. The
main characteristics of factor 2 appeared to relate to a student-
centered learning environment because the item with greatest factor
loading, learner control, has been identified as one of core values
by researchers on student-centered learning environments (Land &
Hannafin, 2000). The student-centered learning environment is defined
by this data as being learner-controlled, collaborative, flexible, and
driven by authentic problem-solving challenges.
To further analyze the data, the internal consistency and bivariate
correlation were examined. Table 4 shows the means, standard devia-
tions, and correlation matrices for the two factors, for middle-school
and college students. For the middle-school students, the internal con-
sistency for factor 1, facilitative scaffolding methods, was 0.84 and for
 73
Table 2. Factor loading of instructional-design elements in the final result (middle-
school students)

Item Factor loading C

Facilitative Constructivist
scaffolding learning
methods environment

8. Drawing pictures to illustrate 0.81 0.65

my understanding of a topic
helps me think more about
what I am studying
9. Writing about my understanding 0.77 0.65
of a topic helps me think more
about what I am studying
3. When my teacher explains 0.74 0.69
how to solve difficult tasks
it helps me think more about
what I am studying
10. Answering questions about 0.63 0.47 0.61
a topic helps me think more about
what I am studying
4. When my teacher asks me how to 0.59 0.48 0.68
solve difficult tasks it helps me think
more about what I am studying
2. Working on activities in class 0.78 0.64
related to real problems on
earth or in our society helps me
think more about what I am studying
6. Having time to think about a question 0.77 0.65
before answering helps me think more
about what I am studying
5. Working with partners during classroom 0.72 0.52
activities helps me think more about
what I am studying
1. Working on activities in class that have 0.44 0.62 0.57
many different answers helps me think
more about what I am studying

Eigen value 4.42 1.14

The proportion of variance explained 49.11 12.61

Note. C: Communalities.
74
Table 3. Factor loading of instructional-design elements (college students)

Item Factor loading C

Task-related Student-centered
scaffolding learning
methods environment

4. When my teacher asks me how to 0.75 0.58

solve difficult tasks it helps me think
more about what I am studying
1. Working on activities in class that 0.58 0.39
have many different answers helps me
think more about what I am studying
3. When my teacher explains how to solve 0.58 0.35
difficult tasks it helps me think more
about what I am studying
9. Writing about my understanding 0.55 0.35
of a topic helps me think more about
what I am studying
10. Answering questions about a topic 0.44 0.24
helps me think more about what
I am studying
8. Drawing pictures to illustrate my 0.41 0.30
understanding of a topic helps me
think more about what I am studying
7. Having freedom in class to explore 0.74 0.56
topics I am interested in helps me think
more about what I am studying
6. Having time to think about a question 0.71 0.51
before answering helps me think more
about what I am studying
5. Working with partners during 0.50 0.28
classroom activities helps me think
more about what I am studying
2. Working on activities in class related 0.50 0.33
to real problems on earth or in our society
helps me think more about what
I am studying

Eigen value 2.78 1.11

The proportion of variance explained 27.83 11.14

Note. C: Communalities.
Table 4. Means, standard deviations, and interfactor correlations for the two factors for middle-school and college students

Middle-school students College students

M SD Factor 1 Factor 2 M SD Factor 1 Factor 2

Factor 1 – Facilitative 3.37 0.87 0.84a Factor 1 – Task-related 2.21 0.63 0.62a
scaffolding methods scaffolding methods
Factor 2 – Constructivist 3.85 0.78 0.65 0.76a Factor 2 – Student-centered 2.00 0.61 0.41 0.55a
learning environment learning environment

Note.a: internal consistency (Cronbach’s alpha) Italics: interfactor correlation.

Mean scores range from 1 to 5.
75
76

Table 5. Means, standard deviations, and correlations for nine items (middle-school
students)

Item 1 2 3 4 5 6 8 9 10

1. Ill-structured tasks
2. Authentic tasks 0.52
3. Teacher explanations 0.56 0.37
4. Teacher questions 0.58 0.47 0.63
5. Collaborative learning 0.34 0.43 0.29 0.25
6. Teacher wait time 0.47 0.49 0.49 0.41 0.49
8. Concept mapping 0.28 0.17 0.48 0.30 0.19 0.21
9. Writing 0.42 0.33 0.56 0.47 0.32 0.32 0.48
10. Question prompts 0.44 0.43 0.52 0.52 0.35 0.56 0.42 0.58
M 3.79 3.66 3.75 3.35 4.05 3.80 3.40 3.00 3.30
SD 0.93 0.99 0.95 1.04 1.19 0.99 1.24 1.20 1.09

factor 2, the constructivist learning environment, 0.76. For the college

students, the internal consistency for factor 1, task-related scaffolding
tools, was 0.62 and for factor 2, the student-centered learning environ-
ment, 0.55. The reliabilities of the two factors were higher for the
middle-school students than for the college students. Table 5 shows
the means, standard deviations, and correlation matrices with nine
items for middle-school students and Table 6 shows the means, stan-
dard deviations, and correlation matrices with10 items for college
students.

Perceived helpfulness of factors prompting reflective thinking

The responses regarding the second research question(Which of the

identified factors is perceived as most helpful in prompting reflective
thinking in PBL environments?) were different for middle-school and
college level students. In the study of middle-school students’ percep-
tions, factor 2, the constructivist learning environment, emerged as the
highest ranked factor, followed by factor 1, facilitative scaffolding
methods. The one-way within-subject ANOVA showed significant dif-
ferences between the constructivist learning-environment factor
(M=3.85, SD=0.78), and the facilitative scaffolding-methods factor
(M=3.37, SD=0.87), F(1, 121)=52.04, p<0.01, g2=0.36. These re-
sults suggest that middle-school students perceive the constructivist
learning environment as more important than the scaffolding method
in prompting them to think reflectively.
 77
Table 6. Means, standard deviations, and correlations for 10 items (college students)

Item 1 2 3 4 5 6 7 8 9 10

1. Ill-structured tasks
2. Authentic tasks 0.25
3. Teacher explanations 0.22 0.22
4. Teacher questions 0.24 0.17 0.32
5. Collaborative learning 0.18 0.17 0.17 0.12
6. Teacher wait time 0.15 0.21 0.20 0.06 0.20
7. Learner control 0.22 0.30 0.17 0.11 0.24 0.33
8. Concept mapping 0.26 0.25 0.15 0.17 0.15 0.15 0.26
9. Writing 0.28 0.17 0.16 0.21 0.16 0.16 0.18 0.30
10. Question prompts 0.23 0.12 0.17 0.18 0.19 0.19 0.17 0.14 0.24
M 2.39 1.98 1.98 2.39 2.16 1.79 2.05 2.10 2.38 2.01
SD 1.01 0.80 1.01 1.08 1.11 0.82 1.00 1.06 1.08 0.92

In the study of college students’ perceptions, factor 1, task-related

scaffolding methods, emerged as a higher-ranked factor than factor 2,
the student-centered learning environment. The one-way within-subject
ANOVA showed significant differences between the task-related scaf-
folding-method factor (M=2.21, SD=0.63), and the student-centered
learning-environment factor (M=2.00, SD=0.61), F(1, 748)=81.00,
p<0.01, g2=0.11. These results suggest that college students perceive
task-related scaffolding methods as more helpful than a student-
centered learning environment in prompting them to think reflectively;
however the mean for both was less than 3.0, indicating some dis-
agreement about the helpfulness of both factors in prompting reflec-
tive thinking. A more accurate statement about these results is that
college students perceived student-centered learning as more unhelpful
than task-related scaffolding methods.

Perceived helpfulness of instructional-design elements prompting

reflective thinking

Middle-school and college students also responded differently to the

items related to the third research question (Which design elements
within each identified factor are perceived as most helpful in prompting
reflective thinking in PBL environments?).
The results from the study of middle-school students’ perceptions
show that the highest ranked element was loaded to the constructivist
learning-environment factor and was of a social nature (See Table 7).
 78

Table 7. Means and standard deviations of question items (middle-school and college students)

Factor Middle-school students Factor College students

Item M SD Item M SD

Factor 1 – 3. When my teacher explains 3.75b 0.95 Factor 1 – 4. When my teacher asks me 2.39c 1.08
Facilitative how to solve difficult tasks it Task-related how to solve difficult tasks it helps
scaffolding helps me think more scaffolding me think more
methods methods
8. Drawing pictures to illustrate 3.40 1.24 1. Working on activities in class that 2.39c 1.01
my under-standing of a topic have many different answers helps
helps me think more me think more
4. When my teacher asks me 3.35 1.04 9. Writing about my understanding 2.38c 1.08
how to solve difficult tasks of a topic helps me think more
it helps me think more
10. Answering questions about 3.30 1.09 8. Drawing pictures to illustrate my 2.10 1.06
a topic helps me think more under-standing of a topic helps me
think more
9. Writing about my understanding 3.00 1.20 10. Answering questions about a 2.01 0.92
of a topic helps me think more topic helps me think more
3. When my teacher explains 1.98 1.01
how to solve difficult tasks it
helps me think more
 a
Factor 2 – 5. Working with partners during 4.05a 1.19 Factor 2 – 5. Working with partners during 2.16 1.11
Constructivist classroom activities helps me think more Student-centered classroom activities helps me
learning learning think more
environment environment
6. Having time to think about a 3.80 0.99 7. Having freedom in class to 2.05 1.00
question before answering helps explore topics I am interested
me think more in helps me think more
1. Working on activities in class that 3.79 0.93 2. Working on activities in 1.98 0.80
have many different answers helps class related to real problems
me think more on earth or in our society
helps me think more
2. Working on activities in class related 3.66 0.99 6. Having time to think 1.79 0.82
to real problems on earth or in our about a question before
society helps me think more answering helps me think more

Note: 1. astatistically different from all other mean scores at the 0.05 level.
b
statistically different from all other mean scores at the 0.01 level.
c
Statistically the same, statistically different from all other mean scores at the 0.01 level.
2. Bold shows the highest ranked item among 10 items.
79
80

For the items in the constructivist learning-environment factor, the

one-way within-subject ANOVA showed that the mean score for item
5, collaborative learning (M=4.05, SD=1.19) was statistically differ-
ent from all the other item mean scores at the 0.05 level. For
instance, it was statistically different from the second-highest ranked
item 6, wait-time, F(1, 121)=5.78, p<0.05, g2=0.05. For the items in
the facilitative scaffolding-methods factor, item 3, teacher explanations
(M=3.75, SD=0.95) was perceived as the most helpful element. The
score was statistically different from all the other mean scores at the
0.01 level. For instance, it was statistically different from the second-
highest ranked item, item 8, concept-mapping (M=3.40, SD=1.24),
F(1, 121)=11.42, p<0.01, g2=0.09.
College students’ perceptions have their most highly ranked ele-
ments loaded to the task-related scaffolding-method factor: item 4 tea-
cher questions (M=2.39, SD=1.08) and item 1, ill-structure tasks,
(M=2.39, SD=1.01), followed by item 9, reflective writing (M=2.38,
SD=1.08). The one-way within-subject ANOVA also showed that the
mean scores for those three items were not statistically different from
each other, but were significantly higher than the rest of the elements
within the factor at the 0.01 level. For instance, they were statistically
different from the fourth-highest ranked item 8, concept-mapping
(M=2.10, SD=1.06), F(1, 748)=35.10, p<0.01, g2=0.05.
For items in the student-centered learning-environment factor, item
5, collaborative learning (M=2.16, SD=1.11), was the highest-ranked
element. It was scored statistically higher than the rest of the elements
within the factor at the 0.05 level. For instance, it was statistically dif-
ferent from the second-highest ranked item 7, learner control
(M=2.05, SD=1.00), F(1, 748)=5.28, p<0.05, g2=0.01.

Discussion

The primary research aim of this study was to identify and descrip-
tively compare the instructional-design factors and elements that mid-
dle-school and college students perceive as helpful in prompting
reflective thinking in PBL environments.

Two factors prompting reflective thinking in a PBL environment

Although the literature reports that several design elements can

prompt reflective thinking, two fundamental categories were discov-
 81

ered during the factor analysis of the data from both middle-school
and college level learners in these two studies. Both middle-school and
college level learners perceived the learning environment and scaffold-
ing methods as helpful factors prompting reflective thinking in a PBL
environment, although the essence of these two and their order of
importance in prompting reflective thinking were different.
These results also differ from those of an early study by Koszalka
et al. (2002), conducted in a science learning environment, in which
three factors emerged. The reflective teaching-method factor reported
in the previous study was not replicated in this one. Two design
elements (teacher questions and teacher explanations) found previously
to load under teaching methods, loaded to the scaffolding-meth-
ods factor for both the middle-school group and the college-level
group. This loading suggests that teaching methods are integral to
scaffolding methods that help learners solve problems in PBL
environments.
The characteristics of two task-related design elements, the
ill-structuredness and authenticity of tasks, both loaded to the con-
structivist learning-environment factor for middle-school students,
whereas ill-structured tasks loaded to task-related scaffolding methods
and authentic tasks loaded to the student-centered learning-environ-
ment factor for the college students. One possible reason why the
task-design elements loaded together for the middle-school students
can be found in the characteristics of constructivist learning environ-
ments, which share attributes with PBL. Both PBL and constructivist
researchers report that authentic and ill-defined tasks are especially
important attributes of the learning environment (Albanese &
Mitchel, 1993; Honebein et al., 1993; Barrow, 1998). Therefore,
students might not perceive the helpfulness of the task-design
elements as distinctive from other factors.
On the other hand, college-level students perceived the helpfulness
of these two task-related design elements as belonging to separate
factors. Ill-structured tasks might have been perceived as asking
about the task of working on activities rather than on the ill-struc-
tured nature of the activity, hence the loading to task-related scaffold-
ing methods. Authentic tasks might have been perceived as related to
students’ experiences in the real world, that is, similar to the environ-
ment for learning, hence the loading to the student-centered learning
environment.
It is interesting that both the internal consistencies of the factors
and the portions of variance explained for the college-level students
82

were lower than those for the middle-school students. One possible
reason for the low reliabilities and low percentages of variance
explained may be related to the developmental stage of the college
students. Since college-level students are more self-regulated than
middle-school students, they might not need the reflective-thinking
strategies provided in their class and thus might not perceive the help-
fulness of the factors and elements that prompted their reflective
thinking as distinctively as the middle-school students did.

Most helpful factor and element

While the middle-school students in this study perceived the con-

structivist learning environment as the most helpful factor, the col-
lege students perceived task-related scaffolding methods as most
helpful. For middle-school learners, it appears that how they engage
in thinking reflectively is more important than what they do. The
most important elements for the middle-school students was collab-
oration, which has generally been classified in the literature as so-
cial interaction in the learning environment. This element is
essential in making PBL constructive because one major learning
approach in PBL is small-group social learning (Evensen & Hmelo,
2000).
College-level learners, however, perceive what they do as more
important to their reflective thinking than how they are engaged, al-
though neither of these factors enjoys the benefit of being a highly
effective method for prompting reflective thinking, according to
them. The most highly ranked elements for the college students were
teacher questions, ill-structured tasks, and reflective writing. These
elements are essential for making students self-directed learners in a
PBL environment (Hmelo & Lin, 2000). These findings imply that
middle-school students prefer a learning climate in which they can
interact with their classmates to prompt their reflective thinking,
whereas the elements ranked highest by the college students were
scaffolds such as teacher questions, reflective writing, or ill-structured
tasks, to help them become self-directed learners.

Differences in perceived helpfulness by developmental stage of reflective

thinking

The pattern of factors prompting reflective thinking by the partici-

pants in this study was found to differ by age group. The result that
 83

middle-school students perceive the constructivist learning environment

factor as more helpful than college students do reveals that younger
students can benefit more from the creation of supportive, experien-
tial environments than older students, as might be expected. On the
other hand, the result that college students perceive task-related scaf-
folding methods as more helpful than the student-centered learning
environment implies that college students perceive the benefits more
from the self-directed approaches of reflection in a PBL environment.
In fact, this implication is supported by the result that item 8, reflec-
tive writing, was ranked as one of the highest design element in all
items by the college students (M=2.38, SD=1.08), whereas it was
ranked lowest by the middle-school students (M=3.00, SD=1.20).
These findings point to the importance of developmentally appropri-
ate instruction. According to the National Middle School Association
(1995), young adolescents between the ages of 10–15 are in a transi-
tion period from concrete thinking to abstract thinking. Since the
middle-school students in this study were sixth to eighth graders, we
can assume that they were in this transition stage and thus needed
interesting and experiential learning more than reflective writing.
The agreement among middle-school learners as to the helpfulness
of the specified factors in prompting reflecting thinking and the dis-
agreement among college-level learners is worth further consideration.
For example, the lowest ranked item for the middle-school students
(item 9) scored substantially higher than the highest ranked items for
the college students (items 4 and 1) (See Table 7). There are two pos-
sible ways of interpreting the negative results found for the col-
lege-level learners. On the one hand, college-level learners could be at
a higher stage of reflective thinking, as noted above by King and
Kitchener (1994), so that tools and environments were not needed to
prompt thinking that they already did. On the other hand, college
level learners might not be interested in reflective-thinking tasks that
require more time and effort than those found in a teacher-directed,
well structured learning environment. Future research should explore
this difference by assessing participants’ stages in terms of reflective
thinking prior to administering a survey. This difference will have
important implications regarding the necessity of providing structured
and targeted reflection tools for middle-school learners but not for
college-level learners. The findings also imply that we must consider
developmental differences in attempting to encourage reflective think-
ing in PBL environments.
84

Conclusion

This study explored the patterns of identified factors that prompt

reflective thinking in students at different age levels in a PBL environ-
ment. After implementing PBL lessons, we examined both middle-
school and college students’ perceptions of the factors prompting
reflective thinking. The findings indicated that both groups of stu-
dents perceived the learning environment and scaffolding methods as
factors that prompt reflective thinking, and yet each group perceived
the characteristics and helpfulness of the identified factors and design
elements differently. This difference in perception implies that we
must consider age level or developmental stage in terms of reflective-
thinking ability when designing a PBL environment.
However, findings of this study need to be placed in the context of
several limitations. First, data were collected from two different PBL
environments using different content areas (science, math, technology
and geography for middle-school students; statistics for college level
students). We acknowledge that direct statistical comparison between
two different groups was limited due to students’ participation in dif-
ferent learning environments. Although the results would be suspect
because of age appropriateness of the teaching materials and content
for the middle school and the college level students, our comparison
is based on the sameness of the instrument and similarity of a PBL
environment to determine what events prompt reflective thinking
rather than the sameness of the content or context. A future empirical
study that compares two groups in the same PBL treatment environ-
ments can provide us with more reliable findings.
Second, a measurement error is likely to be involved in this study
due to the sample size difference. The sample for the middle-school
group was 122, and the sample for the college level group was 749.
Although the difference in sample sizes is a shortcoming of this study,
the important issue is that, in spite of different sample sizes, the find-
ings from the factor analysis were valid. Event if the same samples
were increased to strengthen the robustness of the study for the pur-
pose of comparison, the essential findings, the factor structures that
prompt reflection, would be unchanged. Related to this issue, factor
structures for the college level students may suffer from low reliabil-
ity. For the college level students, the internal consistencies for factors
were lower than those found in the middle-school students, raising a
question for the validity of the factor structures found in the college
group students. The rigorous criteria to extract the number of factors
 85

used in this study, however, enabled us to ensure the factor structures

findings.
Despite these limitations, the study reveals different patterns of de-
sign factors that learners perceived as most helpful in prompting them
to think reflectively according to the different age groups. Thus the
present findings can shed light on understanding of factors that
prompt reflective thinking at different developmental levels. One of
the most meaningful findings of this study was the importance of the
learning environment and of scaffolding methods. Given that both of
these factors include various design elements, it is important to fur-
ther refine and test the attributes of each. Since the characteristics of
a PBL learning environment overlap with those of a constructivist
learning environment, future studies should investigate various design
strategies to identify both the attributes of the student-centered learn-
ing environment and scaffolding methods that might prompt reflective
thinking in a PBL environment. However, since this study did not
investigate any prior experience with PBL environments in either
group, the study results are limited in their generalizability. Future re-
search needs to examine the influence of prior experience on students’
perceptions of the helpfulness of factors that prompt reflective think-
ing. Further research is also needed on the relationship between the
learning-environment factor and the scaffolding-methods factor.
Recently, many scaffolding tools that make use of current technology
are being devised to help students develop their higher-order thinking
skills in a PBL environment. The elements in the learning-environ-
ment factor investigated in this study can be used in major design
strategies for developing such scaffolding tools.

Acknowledgements

This project was made possible through funding from the National
Aeronautics Space Administration, Leading Educators to Applica-
tions, Research, and NASA-Related Educational Resources in Science
(LEARNERS), a Cooperative Agreement Notice from the NASA
Education Division and Learning Technologies Project. Project Num-
ber: NCC5-432: Learning Using ERAST Aircraft for Understanding
Remote Sensing, Atmospheric Sampling and Aircraft Technologies,
(LUAU II).
86

References

Albanese, M. & Mitchell, S. (1993). Problem-based learning: A review of literature on

its outcomes and implementation issues. Academic Medicine 68(1): 68–81.
Aldred, M. & Aldred, S. (1998). Problem-based learning: The good, the bad, and the
ugly. Journal of Dental Education 62(9): 650–655.
Andrusyszyn, M.-A. & Daive, L. (1997). Facilitating reflection through interactive
journal writing in an online graduate course: A qualitative study. Journal of Distance
Education 12(1): 103–126.
Barrow, H. (1998). The essentials of problem-based learning. Journal of Dental
Education 62(9): 630–633.
Craig, G. (1983). Human development. Englewood Cliffs, NJ: Prentice–Hall Inc.
Crain, W. (2000). Theories of development-concepts and applications. NJ: Prentice–Hall
Inc.
Evensen, D. & Hmelo, C. (2000). Problem-based learning – A research perspective on
learning interaction. Mahwah, NJ: Lawrence Erlbaum Associates.
Griffith, B. & Frieden, G. (2000). Facilitating reflective thinking. Counselor Education &
Supervision 40(2): 82–92.
Hair, J.F., Anderson, R.E., Tatham, R.L. & Black, W.C. (1995). Multivariate data
analysis: With readings. Upper Saddle River, NJ: Prentice Hall.
Hmelo, C. & Lin, X. (2000). Becoming self-directed learners: Strategy development in
problem-based learning. In D. Evensen and C. Hmelo, (ed.), Problem-based learning
– A research perspective on learning interaction, pp. 227–250. Lawrence Erlbaum
Associates: Mahwah, NJ.
Honebein, P.C., Duffy, T.M. & Fisherman, B (1993). Constructivism and the design of
learning environments: context and authentic activities for learning. In T.D. Duffy,
J. Lowyck, D.H. Jonassen and T.M. Welsh, (ed.), Designing environments for
constructive learning, pp. 87–106. Springer-Verlag: New York, NY.
Kinchin, I. & Hay, D (2000). How qualitative approach to concept map analysis can be
used to aid learning by illustrating patterns of conceptual development. Educational
Research 42(1): 45–57.
King, P. & Kitchener, K (1994). Developing reflective judgment. San Francisco: Jossey-
Bass.
Koszalka, T., Song, H. & Grabowski, B. (2002). Examining Learning Environmental
Design Issues for Prompting Reflective Thinking in Web-Enhanced PBL. ED
470092.
Land, S. & Hannafin, M.J. (2000). Students-centered learning environments. In
D.H. Jonassen and S. Land, (ed.), Theoretical foundations of learning environ-
ments, pp. 1–24. Lawrence Erlbaum Associates: Mahwah, NY.
Land, S. & Greene, B. (2000). Project-based learning with world wide web: A qualitative
study of resource integration. Educational Technology Research & Development
48(1): 45–68.
Lin, X., Hmelo, C., Kinzer, C. & Secules, T (1999). Designing technology to support
reflection. Educational Technology Research & Development 47(3): 43–62.
Moon, J. (1999). Reflection in learning & professional development, theory and practice.
London: Kogan Page Inc.
National Middle School Association (1995). This we believe: Developmentally responsive
middle level schools. Columbus, OH: National Middle School Association.
Nunally, J.C. (1978). Psychometric theory. 2nd ed. New York: McGraw_Hill.
 87
Rowe, M. (1974). Wait-time and rewards as instructional variables. Journal of Research
in Science Teaching 11: 81–94.
Song, H., Koszalka, T., &. Grabowski, B. (in press). Exploring instructional design
factors prompting reflective thinking in young adolescents. Canadian Journal of
Learning and Technology.
Stepien, W. & Pyke, S. (1997). Designing problem based learning units. Journal for the
Education of the Gifted 20(4): 380–400.
Virtanen, P., Kosunen, E., Holmberg-Marttila, D. & Virjo, I (1999). What happened in
a PBL tutorial session? Analysis of medical learner’s written accounts. Medical
Teacher 21(3): 270–276.
Williams, M. (1996). Learner control and instructional technologies. In D. Jonassen,
(ed.), Handbook of research for educational communications and technology, pp. 957–
983. Macmillan Library Reference: New York, NY.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.