Vol 11 No 1 - April 2015

IJLTER.
ORG
International Journal
of
Learning, Teaching
And
Educational Research
p-ISSN: 1694-2493
e-ISSN: 1694-2116
Vol.11 No.1
PUBLISHER International Journal of Learning, Teaching and
London Consulting Ltd Educational Research
District of Flacq
Republic of Mauritius
www.ijlter.org The International Journal of Learning, Teaching
and Educational Research is an open-access
Chief Editor journal which has been established for the dis-
Dr. Antonio Silva Sprock, Universidad Central de semination of state-of-the-art knowledge in the
Venezuela, Venezuela, Bolivarian Republic of field of education, learning and teaching. IJLTER
welcomes research articles from academics, ed-
Editorial Board
Prof. Cecilia Junio Sabio ucators, teachers, trainers and other practition-
Prof. Judith Serah K. Achoka ers on all aspects of education to publish high
Prof. Mojeed Kolawole Akinsola quality peer-reviewed papers. Papers for publi-
Dr Jonathan Glazzard cation in the International Journal of Learning,
Dr Marius Costel Esi Teaching and Educational Research are selected
Dr Katarzyna Peoples through precise peer-review to ensure quality,
Dr Christopher David Thompson
originality, appropriateness, significance and
Dr Arif Sikander
Dr Jelena Zascerinska readability. Authors are solicited to contribute
Dr Gabor Kiss to this journal by submitting articles that illus-
Dr Trish Julie Rooney trate research results, projects, original surveys
Dr Esteban Vázquez-Cano and case studies that describe significant ad-
Dr Barry Chametzky vances in the fields of education, training, e-
Dr Giorgio Poletti learning, etc. Authors are invited to submit pa-
Dr Chi Man Tsui
Dr Alexander Franco
pers to this journal through the ONLINE submis-
Dr Habil Beata Stachowiak sion system. Submissions must be original and
Dr Afsaneh Sharif should not have been published previously or
Dr Ronel Callaghan be under consideration for publication while
Dr Haim Shaked being evaluated by IJLTER.
Dr Edith Uzoma Umeh
Dr Amel Thafer Alshehry
Dr Gail Dianna Caruth
Dr Menelaos Emmanouel Sarris
Dr Anabelie Villa Valdez
Dr Özcan Özyurt
Assistant Professor Dr Selma Kara
Associate Professor Dr Habila Elisha Zuya
VOLUME 11 NUMBER 1 April 2015
Table of Contents
Using Natural Language Processing Technology to Analyze Teachers’ Written Feedback on Chinese Students’
English Essays ........................................................................................................................................................................ 1
Ming Liu, Weiwei Xu, Qiuxia Ran and Yawen Li
Using Particle Swarm Optimization Approach for Student Engagement Measurement............................................ 12

Ming Liu, Yuqi Wang, Hua Liu, Shujun Wu and Chang Li
A Study of the Development of Courseware and Students’ Learning Effectiveness in Primary Education: Using
Three Teaching Techniques as an Example ....................................................................................................................... 22
Fang-Chun Ou
A Comparative Examination of Teacher Candidates’ Professional Practicum Experiences in Two Program Models
................................................................................................................................................................................................. 36
Nancy Maynes, Anna-Liisa Mottonen, Glynn Sharpe and Tracey Curwen
A Study of Formative Assessment Strategies in Teachers‘ School-Based In-Service Training ................................... 53

Eva Nyberg and Mona Holmqvist Olander
Designing and using interactive e-books in Vietnam ..................................................................................................... 75

Ngoc-Giang Nguyen
Impact Investigation of using a Digital Literacy Technology on a Module: Case Study of Tophat .......................... 99
Xue Zhou and Stella-Maris Orim
Implementation of the 2006 Education Amendment Act on Indigenous Languages in Zimbabwe: A Case of the
Shangaan Medium in Cluster 2 Primary Schools in the Chiredzi District ................................................................. 117
Webster Kododo and Sparky Zanga
The Concept of In Situ Lecturing ...................................................................................................................................... 128

Joachim R. R. Ritter and Ellen Gottschämmer
The Mathematics Problem and Mastery Learning for First-Year, Undergraduate STEM Students ...................... 141
Layna Groen, Mary Coupland, Tim Langtry, Julia Memar, Beverley Moore and Jason Stanley
Teaching Culture through Language: Exploring Metaphor and Metonymy in Chinese Characters ..................... 161
Hu, Ying-Hsueh
Coaches‟ Perceptions of how Coaching Behavior affects Athletes: An Analysis of their Position on Basic
Assumptions in the Coaching Role ................................................................................................................................. 180
F. Moen, R. Giske and R. Høigaard
Regional Educational Development Research and School Improvement: A Systematic Literature Review of
Research ............................................................................................................................................................................... 200
Associate Professor Lena Boström
The Value-Added Assessment of Higher Education learning: The case of Nagoya University of Commerce and
Business in Japan ............................................................................................................................................................... 212
Hiroshi Ito Surname and Nobuo Kawazoe
1
International Journal of Learning, Teaching and Educational Research

Vol. 11, No. 1, pp. 1-11, April 2015.
Using Natural Language Processing Technology

to Analyze Teachers’ Written Feedback on
Chinese Students’ English Essays
Ming Liu, Weiwei Xu, Qiuxia Ran and Yawen Li

Southwest University
Beibei District, Chongqing, China
Abstract. Writing an essay is a very important skill for students to

master, but a difficult task for them to overcome. It is particularly true
for English as Second Language (ESL) students in China. It would be
very useful if students can receive timely and effective feedback about
their writing. In order to build an automatic feedback system, we need
to understand the relationship between textual features and human
teacher feedback, and how well those features were used for predicting
feedback rating. In this study, we analyzed 105 Chinese English majors’
essays with teachers’ feedback and used Coh-Metrix, a computational
linguistic tool, to extract features from their writing. The study results
showed some feedback was moderately correlated to some textual
features (e.g. text easability cohesion and lexical diversity were related
to coherence feedback) and those feedback are more predictable, such as
spelling, grammar, supporting ideas and coherence. This finding has
important implications for building automated writing feedback tool.
Keywords: Writing Feedback, Text Analysis, Natural Language

Processing.
1. Introduction
With the coming of the 21st century and the globalization of English, English
essay writing, as one of the four basic skills of language learning, has become a
more and more important skill. It not only requires some basic writing skill,
such as spelling and grammar, but also asks some high competency of writing,
such as coherence, structure and reasoning. Thus, it is also a difficult task to
overcome. It is particularly so in China. Statistics show that the number of
college students in China has soared to twenty-six million in 2013 (Bureau of
Statistis of China, 2013), accounting for the largest proportion of ESL learners
worldwide. Since 1987, the writing test has become one important aspect of the
College English testing in China. As for college students in China, college
English has been an obligatory course to take. In a typical English course,
© 2015 The authors and IJLTER.ORG. All rights reserved.

2
students have to do 2-3 essay writing assignments and take 1 essay writing test
in order to pass national English tests, such as College English Test (CET) 4 or
Test for English-Major (TEM) 4. Essay writing is the last part of these tests.
Novice writers need feedback to develop their writing skills; however,
providing timely and meaningful feedback is time-consuming and expensive.
Since the early 1980s, researchers have investigated feedback on students’

writing (Brannon & Knoblauch, 1982). These study results showed that written
feedback provided a potential value in motivating students to revise their draft
and improving their writing (Leki, 1991). As a result, written feedback is the
most popular method among various feedback delivery modes (oral feedback,
audiotaped and writing conference) that teachers use to interact and
communicate with students. Straub (Straub, 2000) suggested that the effective
teacher feedback should be written in complete sentences, avoid abstract,
technical language and abbreviations, relate their comments back to specific
words and paragraphs from the students’ text, by viewing students’ writing
seriously, as part of a real exchange. In addition, an increasing number of
studies have also been conducted to see whether certain types of feedback are
more likely than others to help ESL students improve the accuracy of their
writing, such as direct and indirect feedback (Lee, 2004). Direct or explicit
feedback occurs when the teacher identifies an error and provides the correct
form, while indirect strategies refer to situations when the teacher indicates that
an error has been made but does not provide a correction, thereby leaving the
student to diagnose and correct it.
With the advanced development of information technology and natural

language processing techniques, various numbers of automatic essay scoring
(AES) systems have been proposed. Haswell (Haswell, 2006) reviewed systems
for automated feedback tracing back to the 1950s. These systems focused more
on assessment of end products, and less on providing formative feedback
(Shermis & Burstein, 2003; Williams & Dreher, 2004) The Writer Workshop
(Anderson, 2005) and Editor (Thiesmeyer & Theismeyer, 1990) both focus on
grammar and style. Sourcer’s Apprentice Intelligent Feedback system (SAIF)
(Britt, Wiemer-Hastings, Larson, & Perfetti, 2004) is a computer assisted essay
writing tool used to detect plagiarism, uncited quotations, lack of citations, and
limited content integration problems. The Glosser system (Villalon, Kearney,
Calvo, & Reimann, 2008) aims to support reflection in writing through trigger
questions. It uses text mining algorithms to help learners think about issues such
as coherence, topics, and concept visualization. However, Glosser only provides
generic trigger questions. Liu et al. (Liu, Calvo, & Rus, 2014; Liu, Calvo, & Rus,
2010) investigated an automatic trigger question generation system which could
support critical review writing.
The aim of this study is to investigate the frequent type of feedback used by
human teachers and the relationship between the feedback and the textual
features extracted by using the natural language processing techniques.

3
The rest of this paper is constructed as follows: Section 2 presents related work
on feedback classification. Section 3 describes the study and discusses the
results. Finally, Section 4 concludes this paper.
Table 1: Criterion Category
Criterion Category Examples

Grammar Fragments, Run-on Sentences
Subject-verb agreement, Ill-formed verbs
Pronoun Error, Missing Possessive Error
Usage Wrong article, Missing article
Confusing words, Wrong form of word
Preposition Error
Mechanics Spelling, Capitalize Proper Nouns
Missing Question mark, Missing final punctuation
Missing Apostophe, Missing Comma
Style Repetition of words, Inappropriate words or
phrases
Too many short sentences, Too many long
sentences
Organization Background, Thesis, Main-point
Supporting ideas, Conclusion
2. Related Work
Recent development in natural language processing techniques has made it
possible for researchers to develop a wide range of sophisticated techniques that
facilitate text analysis. Some tools, such as Coh-Metrix (Graesser, McNamara,
Louwerse, & Cai, 2004), LIWC (Pennebaker & Francis, 1999) and Gramulator
(Rufenacht, McCarthy, & Lamkin, 2011), are useful in this respect, and have
certainly contributed to ESL knowledge (S．A. Crossley & McNamara, 2012).
Coh-Metrix is a powerful computational tool that provides over 100 indices of
cohesion, syntactical complexity, connectives and other descriptive information
about content (Graesser et al., 2004). Coh-Metrix has extensively been used to
analyze the overall quality of writing (S．A. Crossley & McNamara, 2012) and
one important aspect of writing quality, such as coherence (Scott a. Crossley &
McNamara, 2011a). For example, Crossley and McNamara found that
computational indices related to text structure, semantic coherence, lexical
sophistication, and grammatical complexity best explain human judgments of
text coherence. This study focused on using Coh-Metrix to analyze more aspects
of writing quality including, Supporting Ideas, Conclusion and Sentence
Diversity.
The AES systems, such as Criterion (Burstein, Chodorow, & Leacock, 2004), can
provide feedback on some aspects of writing including grammar, usage,
mechanics, style, organization, development, lexical complexity and prompt-

4
specific vocabulary usage (See Table 1). The Criterion categories are more
relevant to our case since we aim to generate corrective feedback on different
aspects of ESL student writing.
3. Study
We conducted an empirical study in analyzing Chinese ESL college student
essays with teachers’ comments and the relationship between the teacher
feedback and textual features. Section 3.1 describes the annotation process,
where each essay is scored in different aspect, such as Grammar, Spelling,
Coherence, Organization and Supporting Ideas. Section 3.2 shows the textual feature
extraction process. Section 3.3 illustrates the relationship between the textual
features and each feedback category, while section 3.4 examines the predictive
strength of the features in explaining the score variance in the each feedback
score.
3.1 Proposed Feedback Taxonomy
Table 2: Feedback Frequency and Pearson Correlations between Raters

Feedback Category Frequency r
Grammar 48 .824
Spelling 12 .504
Word Count 24 .707
Sentence Diversity 40 .454
Conclusion 44 .747
Supporting Ideas 98 .632
Coherence 40 .716
Chinglish Expression 24 .352
Organization 89 .534
Our dataset containing 105 English majors’ essays with teachers’ feedback was
collected from a large university in China. Two experienced English teachers
volunteered to rate the quality of the essays. They had at least five years of
teaching composition course for English majors. Their first task was to identify
the most frequent feedback type adapted from the standardized rubric used for
grading college English. 9 frequent feedback categories were found, including
Grammar, Spelling, Word Count, Sentence Diversity, Conclusion, Supporting Ideas,
Organization, Coherence and Chinglish (See Appendix I). Table 2 shows that
Supporting Ideas and Organization categories were more frequent than others,
while Spelling and Chinglish Expression and word count were less frequent. We
observed some feedback categories were similar to the Criterion categories, such
as Grammar, Spelling and Supporting Ideas. But, the Chinglish Expression and
Conclusion categories only appeared in our dataset.
The teachers’ second task was to give a score to each feedback category
regarding to the rubric (See Appendix I) on a scale of 3. 1 means negative

5
feedback on the category while 3 means positive feedback on the category. The
Correlations between the raters are located in Table 2. The raters had the highest
correlations for judgments of Grammar, Word Count, Conclusion and
Supporting Ideas and the lowest correlations for Chinglish and Sentence
Diversity.
For further analysis, the dataset was randomly divided into training set (n=70)
and testing set (n=35). A training set was used to identify which of the textual
features most highly correlated with each feedback score. Moreover, the training
set was used to train a multiple regression model to examine the amount of
variance explained by each writing feature. The model was then applied to a test
set to calculate the accuracy of the analysis.
3.2 Textual Feature Extraction

We used Coh-Metrix 3.0, which could retrieve 108 scores of textual features.
More information can be found on the website
(http://cohmetrix.Memphisedu/cohmetrixpr/index.html).
Descriptive indices: It includes the number of paragraphs, number of sentences,

number of words, number of syllables in words, mean length of paragraphs etc.
Cohesion: Cohesion is a key aspect of understanding language discourse

structure and how connections within a text influence cohesion and text
comprehension(Kintsch & van Dijk, 1978). Coh-Metrix employs referential
cohesion including noun overlap, argument overlap, stem overlap, and LSA-
based semantic overlap.
Sentence Complexity: The grammatical structure of a text is also an important

indicator of human evaluations of text quality. Difficult syntactic constructions
(syntactic complexity) include the use of embedded constituents, and are often
dense, ambiguous, or Ungrammatical (Graesser et al., 2004). Syntactic
complexity is also informed by the density of particular syntactic patterns, word
types and phrase types.
Lexical sophistication: Lexical sophistication refers to the writer’s use of advanced

vocabulary and word choice to convey ideas. Lexical sophistication is captured
by assessing the type and amount of information provided by the words in a
text. Words are assessed in terms of rarity (frequency), abstractness
(concreteness), evocation of sensory images (imagability), salience (familiarity),
and number of associations (meaningfulness). Words can also vary in the
number of senses they contain (polysemy) or levels they have in a conceptual
hierarchy (hypernymy).
Moreover, we propose and extract 8 new features that are not available in Coh-
Metrix. These features refer to characteristics of ESL learners’ writing style and
reflect on the importance of the introduction section, conclusion section and
mechanics in errors including spelling errors and grammatical errors. In the
database, each essay is stored as a plain text, where each line is a paragraph. We

6
use Java API to extract the first line and last line text, as introduction and
conclusion section respectively. For checking spelling errors, an open source
spelling error checker, called LanguageTool (http://www.languagetool.org/), is
employed to scan each word. For checking grammatical errors, the Link
Grammar Parser (Lafferty, Sleator, & Temperley, 1992) is used to check the
grammar of a sentence based on natural language processing technology. If the
link grammar could not generate links (relations between pairs of words) after
parsing a sentence, this sentence would be considered as ungrammatical.
Number of words in Introduction: the total number of words in the first paragraph
considered as the introduction section.
Number of words in Conclusion: the total number of words in the last paragraph
considered as the conclusion section.
Introduction Portion: the ratios of number of words in introduction to the total

number of words in the document.
Conclusion Portion: the ratios of number of words in conclusion to the total

number of words in the document.
Spelling errors: the number of spelling errors. We employ an open source spelling
error checker called LanguageTool (http://www.languagetool.org/), which is
part of the OpenOffice suite.
Grammatical errors: the number of sentences with grammatical errors. We use the
Link Grammar Parser (Lafferty et al., 1992) to check the grammar of a sentence,
which is also widely used in ESL context.
Percentage of spelling errors: the ratios of the number of word spelling errors to the
total number of words in the document.
Percentage of grammatical errors: the ratios of the number of sentence with

grammatical errors to the total number of sentences in the document.
Therefore, there are totally 116 features extracted from each essay.

7
3.3 Pearson Correlation
Table 3: Correlations between Textural Features Scores and Raters’

feedback scores
Feedback Category Features R P
value
Gerund incidence .415 <0.05
Chinglish paragraph length .459 <0.05
first person singular pronoun .493 <0.01
incidence
Text Easability Cohesion .433 <0.05
Coherence Lexical diversity .402 <0.05
Conclusion Conclusion Portion .477 <0.05
Lexical diversity .394 <0.05
Intentional verbs incidence .496 <0.05
Supporting Ideas Adjective incidence .503 <0.05
CELEX Log minimum frequency .541 <0.01
for content words
Grammar Grammar errors -.606 <0.01
Hypernymy for verbs .506 <0.01
Sentence Variety Standard deviation of Sentence .413 <0.05
length
Spelling Spelling Errors -.617 <0.05
Organization Number of paragraphs .507 <0.01
Word Count Word count .666 <0.01
Based on the system producing feature scores and the human annotators’ score
on each category, we used IBM SPSS for evaluating the Pearson correlation
between textual features and each category. Over 30 textual features
demonstrated significant correlations with the human ratings of each feedback
category. Table 3 shows the Chinglish was more related to the number of Gerund
used, the paragraph length and the first person singular pronoun incidence. The
Coherence was correlated to Text Easability PC Deep cohesion, consistent with
Crossley and McNamara’s study result (S. Crossley & McNamara, 2010). As
expected, the Conclusion was more related to the features of Conclusion Portion
and Lexical Diversity. We have not defined specific features which can detect the
Supporting Ideas. However, some features, such as Intentional verbs and
Adjective incidence, have shown their moderate correlations with the category
of Supporting Ideas. As we had expected, the Grammar and Spelling were
negatively related to the features of grammar error and spelling error. The Word
Count was correlated to the number of words in an essay. Organization was
correlated to the number of paragraphs since the essays with only 1 or 2
paragraphs were given lower scores by human annotators since they did not
have a clear essay structure, introduction, body and conclusion. Crossley and
MacNamara (Scott a. Crossley & McNamara, 2011b) got the similar study
results, where six features including the total number of paragraphs were
significant predictors in the regression to the raters’ organization evaluations.

8
3.3 Test Set Model

We used the training set to train a regression model for each feedback category
and evaluated the model in testing set. Table 4 shows the performance of each
regression model for predicting essay feedback ratings. It has been found that
Grammar (r2=.881) and Spelling feedback (r2=.886) were easier for prediction,
since some textual features were highly related to those feedbacks. It also
demonstrated that the combination of the textual features accounted for 88.1% of
the variance in the grammar evaluation of the 35 essays comprising the test set.
On the other hand, organization and conclusion were difficult to predict since
r2=.223 and r2=.380 respectively since the textual features were not correlated to
those feedback ratings.
Table 4: Linear Regression Analysis to Predict Essay Feedback Ratings in Testing Set
Feedback R R2 S.E.
Chinglish .764 .584 .349
Expression
Coherence .790 .624 .472
Conclusion .616 .380 .486
Supporting .745 .555 .407
Ideas
Grammar .939 .881 .260
Sentence .735 .540 .423
Variety
Spelling .941 .886 .242
Organization .475 .223 .473
Word Count .756 .572 .535
Notes: S.E. is standard error
4. Conclusion
Human teachers’ written feedback is very useful for students to revise their draft
and improve writing. A great number of researches has been conducted to
investigate the theoretical foundation of feedback in terms of feedback mode,
feedback strategies and feedback classification. With the development of
information technologies, automated essay scoring tools have been proposed,
which can extract textual features and generate corrective feedback on the traits
of writing including grammar, usage, style, mechanics and organization.
However, these AES systems are mainly designed for international ESL
students, who take TOFEL test. Those students can only represent a small
portion of ESL students, because they obviously possess a higher English
competency. Thus, we conducted an empirical study to investigate the frequent
feedback types and examine the feasibility of using existing natural language
processing tools to automatically measure the feedback.
In the study, we collected 105 essays written by English majors and some
teachers’ comments at a large university in China. Two English teachers first
found 9 frequent feedback categories based on the teachers’ comments. Some
feedback categories are consistent with the Criterion category. Then, they gave a

9
score on a scale of 1 to 3 to each feedback category of each student essay. The

study results showed that the feedback had moderate correlations with some
features extracted by using Coh-Metrix, a computational writing analysis tool,
and some proposed new features. For example, coherence feedback was highly
related to Text Easability Cohesion and Lexical diversity, while Supporting Ideas
was related to Intentional verbs incidence and Adjective incidence. Moreover, it
has been found that some feedback, such as supporting ideas, coherence,
grammar and spelling, were more predictable. It indicated the feasibility of
using existing NLP tools to measure the quality of feedback.
Our future work will examine teachers’ comments in detail and collect non-
English major student essays for analysis. In addition, we will focus on building
an automatic essay feedback generation system. Specifically, we will investigate
the feedback generation mechanism by using association rule mining
algorithms. In addition, we will look at how to incorporate effective feedback
strategies, such as formative feedback theory, into feedback generation
templates.
Acknowledgment
The authors would like to thank those teachers and student participants. This
work is partially supported by Chongqing Social Science Planning Fund
Program under grant No. 2014BS123, Fundamental Research Funds for the
Central Universities under grant No. SWU114005, No. XDJK2014A002 and No.
XDJK2014C141 in China.
Appendix A
Table 5: Nine Traits Rubric for Essay Writing
Category Scoring
Organization 1 Rudiment of organization apparent, but may be illogical,
ineffective or different to understand the sequencing of ideas
2 Satisfactory organization of sections, but the sequencing of
paragraphs within sections may be problematic.
3 Effective method of organization for both section and for
paragraphs within sections.
Supporting Ideas 1 Minimal use of examples and facts to support the writer’s
idea.
2 using some examples and facts to discuss
strengths/weakness of some opinions, but may have difficulties
(1) choosing appropriate facts; (2) sufficiently explaining those
facts; (3) connecting them to present thing.
3 Effective supports the strengths and weakness of one’s
opinion; Generally effective use of choice of examples and facts,
although some material may be extraneous or not adequately
explained
Grammar 1 Uses simple sentence constructions, but there are still
numerous errors (greater than 7).

10
2 Uses simple sentence with minor errors (between 5-7).

3 Uses complex sentence with minor errors (less than 5).
Sentence Variety 1 Little complex sentences or longer sentences (less than 2) are
used
2 Moderate number of complex sentences or longer sentences
(between 2 and 4) are used
3 A Effective use of complex sentence construction or longer
sentence (greater than 4)
Coherence 1 Some apparent sequencing of sentences within paragraphs,
relying primarily on a limited set of cohesive devices (e.g. first,
second, third) and basic connection words (e.g. however, also,
because). However, there may be frequent points in which the
reader has difficulties understanding sequencing of ideas.
2 Writer sequences ideas, relying primarily on a limited set of
cohesive devices; some errors or unclear transitions, but they do
not significantly impair understanding of the text.
3 Coherent and logical sequencing of ideas, using a wider
range of cohesive devices (e.g. pronominalization, passive, etc;)
only minor and occasional errors.
Word Count 1 Less than 50 words
2 Between 50 and 150 words
3 Around 200 words
Conclusion 1 No conclusion key words found; Conclusion is
inappropriate; No conclusion
2 briefly summarized some points
3 It stresses the importance of the thesis statement, gives the
essay a sense of completeness.
Spelling 1 greater than 3
2 within 1 and 3
3 no spelling error
Chinglish 1 greater than 5
Expression 2 within 3 and 5
3 less than 2
References
Anderson, J. (2005). Mechanically Inclined:Building Grammar, Usage, and Style into

Writer's Workshop.
Brannon, L., & Knoblauch, C. H. (1982). On students' rights to their own texts: A model
of teacher response. College Composition and Communication, 33, 157-166.
Britt, M. A., Wiemer-Hastings, P., Larson, A. A., & Perfetti, C. A. (2004). Using Intelligent
Feedback to Improve Sourcing and Integration in Students' Essays. Int. J. Artif.
Intell. Ed., 14, 359-374.
Bureau of Statistis of China, N. (2013). China Statistical YearBook.
Burstein, J., Chodorow, M., & Leacock, C. (2004). Automated essay evaluation: The
Criterion online writing service. AI Magazine, 25, 27. doi: 10.1002/rcm.5057
Crossley, S., & McNamara, D. (2010). Cohesion, coherence, and expert evaluations of
writing proficiency. The 32nd Annual Conference of the Cognitive Science Society.
Austin: TX.

11
Crossley, S. a., & McNamara, D. S. (2011a). Text Coherence and Judgments of Essay
Quality: Models of Quality and Coherence. The 33rd Annual Conference of the
Cognitive Science Society.
Crossley, S. a., & McNamara, D. S. (2011b). Understanding expert ratings of essay quality:
Coh-Metrix analyses of first and second language writing. International Journal of
Continuing Engineering Education and Life-Long Learning, 21, 170. doi:
10.1504/IJCEELL.2011.040197
Crossley, S. A., & McNamara, D. S. (2012). Predicting second language writing
proficiency： The role of cohesion， readability， and lexical difficulty. Journal
of Research in Reading, 35, 115-135.
Graesser, A. C., McNamara, D. S., Louwerse, M. M., & Cai, Z. (2004). Coh-metrix:
analysis of text on cohesion and language. Behavior research methods, instruments,
& computers, 36, 193-202.
Haswell, R. (2006). The complexities of responding to student writing; or, looking for
shortcuts via the road of excess. Across the Disciplines, 3.
Kintsch, W., & van Dijk, T. (1978). Towards a model of text comprehension and
production. Psychological Review, 85, 363-394.
Lafferty, J., Sleator, D., & Temperley, D. (1992). Grammatical Trigrams: A Probabilistic
Model of Link Grammar. Paper presented at the Proceedings of the AAAI
Conference on Probabilistic Approaches to Natural Language.
Lee, I. (2004). Error correction in L2 secondary writing classrooms: The case of Hong
Kong. Journal of Second Language Writing, 13, 285-312. doi:
10.1016/j.jslw.2004.08.001
Leki, I. (1991). The preferences of ESL students for error correction in college-level
writing classes. Foreign Language Annals, 24, 203-218.
Liu, M., Calvo, R., & Rus, V. (2014). Automatic Generation and Ranking of Questions for
Critical Review. Educational Technology & Society, 17, 333-346.
Liu, M., Calvo, R. A., & Rus, V. (2010). Automatic Question Generation for Literature
Review Writing Support. Carnegie Mellon University, USA: Springer's Lecture
Notes in Computer Science
Pennebaker, J. W., & Francis, M. E. (1999). Linguistic inquiry and word count (LIWC).
Rufenacht, R. M., McCarthy, P. M., & Lamkin, T. A. (2011). Fairy Tales and ESL Texts:
An Analysis of Linguistic Features Using the Gramulator. Proceedings of the
Twenty-Fourth International Florida Artificial Intelligence Research Society Conference.
Shermis, M. D., & Burstein, J. (2003). Automated essay scoring: A cross-disciplinary
perspective. 16.
The student, the text, and the classroom context: A case study of teacher response, 7 23-
55 (2000).
Thiesmeyer, E. C., & Theismeyer, J. E. (1990). Editor:A System for Checking Usage,
Mechanics, Vocabulary, and Structure.
Villalon, J., Kearney, P., Calvo, R. A., & Reimann, P. (2008). Glosser: Enhanced Feedback
for Student Writing Tasks.
Williams, R., & Dreher, H. (2004). Automatically Grading Essays with Markit©. Issues in
Informing Science and Information Technology, 1, 693-700.

12

Vol. 11, No. 1, pp. 12-21, April 2015
Using Particle Swarm Optimization Approach

for Student Engagement Measurement
Ming Liu, Yuqi Wang, Hua Liu, Shujun Wu and Chang Li

Southwest University
Beibei, Chongqing, China
Abstract. Measuring Student Engagement is a difficult task. Previous

research has used a cloud-based writing platform, Google Docs, which
can store a number of document revisions with timestamps.
Engagement measurement algorithm has taken the advantages of each
timestamp in a revision and calculated how much time the student
spent on a writing task. However, the parameters passed to the
algorithm were fixed and hard to determine, for example, how much
time means fully engaged or partially engaged. In this paper, we
proposed a new student engagement measurement algorithm based on
a computational intelligence approach, Particle Swarm Optimization
technique, to find the optimized parameters for the engagement
measurement algorithm. In the study, the proposed algorithm measures
the engagement of two groups of students in two different writing
activities (long-term and short term writing activities) carried out in our
cloud-based writing platform. The study results show that the
correlations between the engagement measurement and student self-
report are high. In addition, it indicates that this approach is robust to
measure student engagement in both long-term and short term
activities.
Keywords: Student Engagement Measurement, Advanced Educational

Technologies, Particle Swarm Optimization.
Introduction
Student engagement plays an important role in a learning activity. Studies
(Fredricks, Blumenfeld, & Paris, 2004) show that a student who is engaged and
intrinsically motivated in a task is more likely to learn from an activity and
models of school engagement identify three core dimensions: behavioral,
cognitive and emotional engagement. ‘Behavioral engagement’, which is the
focus of the present study, refers to student participation in school related
activities and involvement in any learning tasks such as those being done online
(Fredricks et al., 2004). ‘Cognitive engagement’ refers to motivation,
thoughtfulness and willingness to make an effort to comprehend ideas and

13
master new skills. ‘Emotional engagement’ includes emotions and interest, such
as affective reactions in the classroom towards teachers. These three aspects are
interrelated and helpful to understand engagement as a whole.
The measurement of behavioral engagement is more obvious because behavioral

patterns can be defined, observed and interpreted. Traditionally, student
engagement is measured by teachers’ observation (Bulger, Mayer, Almeroth, &
Blau, 2008; Martin, 2007). But, this approach is time consuming and subjective. In
the era of ‘big’ data, a large amount of student data about their behavior being
harnessed to improve learning interactions and to personalize the learning
experience can be collected by the system (Tanes, Arnold, Selzer King, & Remnet,
2011). For instance, when a student participates in an activity that is technology
mediated, a detailed collection of behavioral events can be recorded. Computer
keystroke-logging (Leijten & Van Waes, 2013) or screen capturing (Latif, 2008)
allow a detailed account of the behavior of a writer including actions such as
starting a new paragraph or deleting a text portion and these are all considered
indicators of behavioral engagement. Thus, new computer technology permits the
observation and identification of learning events, which can then be examined in
relation to other indices of engagement. However, these technologies require
specialized setups and often hardware.
In the recent year, with the development of the cloud-based online writing
platform, such as Google Doc or Wiki, it is possible to capture student’s writing
behavior easily by utilizing document revision history (Cole, 2009; Liu et al.,
2013). However, the engagement measurement algorithm requires so many
predefined parameters, such as the time threshold for full engagement or for
partial engagement. Previously, the thresholds are determined by educational
experts, which is too subjective. If the thresholds are set too high or too low, it
would affect the accuracy of engagement measurement and effect of engagement
visualization.
Particle swarm optimization (PSO) is a population-based metaheuristics used for
stimulating social behaviour such as fish school to a promising position (S. W.
Lin, Ying, Chen, & Lee, 2008). PSO is a subset of swarm intelligence which was
occurred in the late 1980s to relate to cellular robotic systems, where a number of
agents in an environment interact based on local rules. Over the past years,
particle swarm optimization technique has lately been illustrated to have the
ability to solve complex problems, such as automatic group composition(Y.-T.
Lin, Huang, & Cheng, 2010), e-learning problems(Huang, Huang, & Cheng,
2008), automatic test sheets generation (Yin, Chang, Hwang, Hwang, & Chan,
2006). These studies suggested that swarm intelligence is useful for providing
high scalability and robust computation. In our study, we use PSO to optimize
the engagement measurement algorithm.
Behavioural Engagement
Studies of behavioural engagement in learning environments typically use
evidence collected by human observers, such as teachers or students (Lane, 2009;
Martin, 2007). For example, using scales such as the Student Engagement

14
Walkthrough Checklist, observers such as administrators, instructional

supervisors or teachers, have examined the degree to which students exhibit
engagement in the classroom, by measuring behaviors such as positive body
language, consistency of focus, spoken participation (Jones, 2009). The observer
ratings are then compared to simultaneous and anonymous ratings by students
of their level of engagement according to the extent to which the work is
interesting and challenging, and the degree to which they understand why and
what they are learning.
Jones (2009) have defined the models of general engagement including
behavioral, emotional and cognitive engagement as consisting of three
dimensions; intensity, consistency and breadth. Intensity relates to the level of
engagement of each student. Consistency refers to how long students remain
engaged at high levels throughout the class period and breadth refers to how
broadly the class as a whole is engaged. Measuring dimensions of engagement
allows teachers to provide differentiated feedback. For example, if the
engagement intensity is low, teachers can focus on adding rigor and relevance to
expectations and lessons.
To date, most of the research on student engagement has occurred in classrooms
(Sheldon & Biddle, 1998), yet researchers are increasingly exploring learning
theories in web-based activities (Chena, Lambertb, & Guidryb, 2010), social
software (2009), smart interactive devices (Blasco-Arcas, Buil, Hernández-
Ortega, & Sese, 2013) and virtual environments (Bouta, Retalis, & Paraskeva,
2012). ‘Clickers’ (Blasco-Arcas et al., 2013) allowed students to quickly answer
questions presented in class. Responses can be anonymized or identified and
software programs are usually used to summarize responses and present
visualizations in the form of charts. Technology-based tools such as Wiki
technology (2009) have been used to support learning engagement. Cole (2009)
tested Wikis in a third year undergraduate course to examine the degree to
which they supported student knowledge construction, peer interaction and
group work. However given the optional nature of this form of technology in
the course, students did not contribute to the Wiki as was intended. Thus focus
groups were used to examine barriers to uptake rather than the effects of Wikis
on student engagement per se. However, a limitation of previous studies is that
they have not addressed how to automatically track and analyze student
behaviour patterns and present them in a way that is understandable. Given the
difficulties identified by previous studies (2009) related to student use of web-
based techniques the present study was conducted within a laboratory
environment rather than as part of a course.

15
Engagement Visualization and Measurement
Figure 1: Line-based Visualization: green lines with different thickness show that
a user has done several intensive writing in the drafting process.
Graphs are copied from (Liu, Calvo, & Pardo, 2013).
Engagement is critical to the success of learning activities such as writing, and

can be promoted with appropriate feedback. Tracer is a learning analytic system
(Liu et al., 2013) which derives behavioral engagement measures and creates
visualizations of behavioral patterns of students writing on a cloud-based
application. Figure 1 shows that the Line-based Visualization uses a line to
connect the points and the thickness of a line indicates the intensity of the user’s
behavior during a period of time. This information is derived from Intensity-
based engagement measurement algorithm (IbA), where a series represents a
line and its weight represents a line thickness. Therefore, the whole graph is
made of lines. The weighting process is defined as follows:
1. A hashmap is predefined, where each entry contains a time threshold and a
corresponding weight value. For example, (0.5h, 0.8) indicates that the time
threshold is 0.5h and its corresponding weight is 0.8.
2. If the duration between neighboring events is less than the shortest time
threshold, we assign that corresponding weight to the series. For example, in
one month project proposal writing assignment, the following
combinations/hashmap: (0.5h, 1), (1h, 0.8), (3h, 0.4) and (12h, 0.2) is considered
based empirical experience. For example, if the duration of an activity is 2 hours,
we assigned 0.4 as a weight to the series because 3h is the shortest time defined
in the hashmap that is longer than 2h.
Thus the total engagement score is calculated as the following weighted sum:
n
Engagement= i si ∗ wi (1)
where i is the index of a series, Si is the duration of the series i and Wi is the
weight assigned to i.

16
Particle Swarm Optimization

PSO looks through a collection of individual solutions called particles that update
iteratively. Each particle at iteration t can be represented by a D-dimensional state
vector as 𝑥𝑖𝑡 = 𝑥𝑖1𝑡 𝑡
, 𝑥𝑖2 𝑡
, … , 𝑥𝑖𝐷 . Then, to obtain the optimal solution, we define D-
dimensional velocity vectors 𝑉𝑖𝑡 = 𝑉𝑖1𝑡 , 𝑉𝑖2𝑡 , … , 𝑉𝑖𝐷 𝑡
for each particle and
determined by its own best previous experience, denoted as pbest, and the best
experience of all the particles, denoted as gbest. Particles change velocity based
on the pbest and gbest as follows:
Vidt  Vidt 1  c1r1  pbestidt  X idt   c2r2  gbestidt  X idt 
,d=1,2,3…D (2)
Where 𝑐1 𝑎𝑛𝑑 𝑐2 are the learning factors which are commonly set to 2 and 𝑟1 , 𝑟2
are random numbers distributed uniformly in the range [0, 1]. Then, each particle
updates to a new potential answer based on the velocity as:
𝑡+1 𝑡
𝑋𝑖𝑑 = 𝑋𝑖𝑑 + 𝑉𝑖𝑑𝑡 (3)
When the iteration number reaches a pre-determined maximum iteration

number, the update process is terminated and the best individual of the last
generation is the final solution to the target problem.
PSO enhanced Engagement Measurement Algorithm

In this section, we describe the proposed PSO-EM algorithm for predicting the
total time a student spent on the writing task. The aim of this study is to optimize
the accuracy of the engagement prediction by estimating the best values of an
engagement measurement function parameters described above. We used the
Matlab to implement this algorithm. The evaluation matrix for SVR is MSE (mean
square error).
1 𝑛
MSE = 𝑛 𝑖=1(𝑓 𝑥𝑖 − 𝑦𝑖 )2 (4)
MSE is a common evaluation measurement for numeric value prediction, which

has been adapted in education (Tang & Yin, 2012).
In our study, PSO starts with 20-randomly chosen particles and looks for the
best particle iteratively. Each particle is a 6-dimensional vector including three
time thresholds and three weights represents a candidate solution. The
engagement measurement algorithm is constructed for each candidate solution
to estimate its performance. The procedure describing proposed PSO-SVR
approach is as follows.

17
Function PSO-EM () {
Initializing PSO with 20 particles and each engagement measurement algorithm with
each particle.
Evaluating the fitness (MSE) of each particle.
For each iteration in 200
For each particle in 20
Calculating the particle velocity and updating the particle
Calculating the fitness of the particle by passing the parameters to
engagementMeasurement()
Comparing the fitness values and updating the local best and global best
particle.
End
End .
}
Study
In order to evaluate the feasibility of the proposed engagement measurement
algorithm, we have conducted a study, where 120 students were writing an
individual document in a web-based writing system. This system is developed
based on etherpad (http://etherpad.org/), which is an online real-time text
editor, letting authors to write a text document, and look all the revision history
of the document. Each document revision history has been recorded in a textual
database. We need to extract the timestamp of each revision as an input to the
engagement algorithm.
Participants and Procedure
A total of 120 university students participated in this study. The participants’
age ranged from 20 to 30 years (M: 25, SD: 5) and there were 61 males and 59
females. Those student participants came from different disciplines, including
computer engineering and education. They had no prior knowledge of the
system and did not participated in any previous related study. We arranged a
separate one hour writing activity for 60 education majors (writing a personal
best travel experience) while one month writing activity (writing a project
proposal) for 60 engineering students. We conducted this study in a controlled
environment so that each participant could only write in our system (see Figure
2), thus avoiding the ‘copy-and-paste’ issues. Once the writing activity was
finished, each participant was asked to estimate their engagement time in the
writing session. The dataset was divided into the training set (n=30) and testing
set (n=30) for each activity. We used the training set to train the parameters of the
engagement algorithm and testing set to evaluate the performance of the
algorithm.

18
Figure 2: the user interface in the online writing system
Results
The correlation among participants and engagement measurement functions is
presented in Table 1. This study results show that correlations between the
proposed engagement algorithm (PSO-EM) and human are highly correlated
(r=.73 and r=.81) in both writing activities. This algorithm outperformed IbA
which has moderate correlation (r=.49 and r=.59) with student self-report
(Human). We also observed that the student engagement time in the one-hour
writing activity is more predictable than in the one month writing activity,
because the one-hour writing activity produced less document revisions.

19
Table 1: Correlation of engagement time
One month writing One hour writing
PSO-EM IbA Human PSO-EM IbA Human
PSO-EM 1 1
IbA 0.67 1 0.69 1
Human 0.73 0.49 1 0.81 0.59 1

Self-Report
After 200 iterations, PSO-EM converges. Table 2 shows that PSO-EM algorithm
(MSE:15.88 in one hour;MSE:31.89 in one month) gets lower MSE scores than
traditional IbA (MSE:16.13 in one hour;MSE:64.95 in one month) in both writing
tasks (one hour and one month writing tasks).
Table 2: Performance of PSO-EM Algorithm and Its best parameters. T1 means 1

Time Threshold1 Parameter while W1 means weight1 Parameter
Evaluation
Writing Parameters
Measure
Task
T1 T2 T3 W1 W2 W3 MSE
0.5 m 1.0 m 2.0 m 0.33 0.66 1 16.13
IbA
One
Hour PSO 3.30m 4.20 m 5.12m 1.09 2.34 2.89 15.88
-EM
0.5h 1.0h 2.0h 0.33 0.66 1 64.95
ibA
One
Month PSO 3.3h 4.20h 5.12h 1.09 2.34 2.89 31.89
-EM
In the one hour writing task, PSO-EM finds the best parameters for this dataset
include Threshold1 as 3.30, Threshold2 as 4.20 and Threshold3 as 5.12 minute,
and Weight1 as 1.09, Weight2 as 2.34 and Weight 3 as 2.89.
In addition, in the one month writing task, the best parameters for threshold are
different from those parameters in one hour writing task and the unit is hour.
This result indicates that the PSO-EM algorithm is robust to automatically adjust
its parameter values based on the dataset or the nature of the task. It also
suggests that PSO-EM outperformed the traditional method.

20
Conclusion and Future Work

In this paper, we introduce a novel algorithm, called PSO-EM for engagement
measurement, particularly student engagement in a writing activity. This
algorithm is based on a computational intelligence approach, called Particle
Swarm Intelligence, to find the best parameters for engagement measurement
algorithm. Our study result indicates that this algorithm outperformed the
traditional engagement measurement method and can automatically adjust the
function parameters based on the writing task. We also found that the short-time
writing activity (one-week) was more predictable than the long-time writing
activity (one-month), since the short-time writing activity produced less revision
data for analysis. However, PSO-EM can still perform well in complex revision
data due to its robust capability. Our future work will focus on generating real
time visualizations based on the engagement algorithm to support individual
and collaborative writing.
Acknowledgements
This work is partially supported by Chongqing Social Science Planning Fund

Program under grant No. 2014BS123, Fundamental Research Funds for the
Central Universities under grant No. XDJK2014A002 and No. XDJK2014C141
and No. SWU114005 in China.
References
Blasco-Arcas, L., Buil, I., Hernández-Ortega, B., & Sese, F. J. (2013). Using
clickers in class. The role of interactivity, active collaborative learning
and engagement in learning performance. Computer & Education, 62, 102-
110.
Bouta, H., Retalis, S., & Paraskeva, F. (2012). Utilising a collaborative macro-
script to enhance student engagement: A mixed method study in a 3D
virtual environment. Computers & Education, 58(1), 501-517.
Bulger, M. E., Mayer, R. E., Almeroth, K. C., & Blau, S. D. (2008). Measuring
Learner Engagement in Computer-Equipped College Classrooms. Journal
of Educational Multimedia and Hypermedia, 17(2), 129-143.
Chena, P.-S. D., Lambertb, A. D., & Guidryb, K. R. (2010). Engaging online
learners: The impact of Web-based learning technology on college
student engagement. Computers & Education, 54(4), 1222-1232.
Cole, M. (2009). Using Wiki technology to support student engagement: Lessons
from the trenches. Computer & Education, 52(1), 141-146.
Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School Engagement:
Potential of the Concept, State of the Evidence. Review of Educational
Research, 74(1), 59-109.

21
Huang, T. C., Huang, Y. M., & Cheng, S. C. (2008). Automatic and interactive e-
learning auxiliary material generation utilizing particle swarm
optimization. Expert Systems with Applications, 35, 2113-2122.
Jones, R. D. (2009). Student Engagement: Teacher Handbook. Rexford:NY:
International Center for Leadership in Education.
Lane, E. (2009). Clickers: can a simple technology increase student engagement in the
classroom? Paper presented at the International Conference on
Information Communication Technologies in Education, Corfu, Greece.
Latif, M. M. A. (2008). A state-of-the-art review of the real-time computer-aided
study of the writing process. International Journal of English Studies, 8, 29-
50.
Leijten, M., & Van Waes, L. (2013). Keystroke Logging in Writing Research:
Using Inputlog to Analyze and Visualize Writing Processes. Written
Communication, 30, 358-392. doi: 10.1177/0741088313491692
Lin, S. W., Ying, K. C., Chen, S. C., & Lee, Z. J. (2008). Particle swarm
optimization for parameter determination and feature selection of
support vector machines. Expert Systems with Applications, 35, 1817- 1824.
Lin, Y.-T., Huang, Y.-M., & Cheng, S.-C. (2010). An automatic group composition
system for composing collaborative learning groups using enhanced
particle swarm optimization. Computers & Education, 55, 1483-1493. doi:
10.1016/j.compedu.2010.06.014
Liu, M., Calvo, R. A., & Pardo, A. (2013). Tracer: A tool to measure student
engagement in writing activities. Paper presented at the the 13th IEEE
International Conference on Advanced Learning Technologies, Beijing,
China.
Martin, A. J. (2007). Examining a multidimensional model of student motivation
and engagement using a construct validation approach. British Journal of
Educational Psychology, 77, 413-440.
Sheldon, K. M., & Biddle, B. J. (1998). Standards, accountability, and school
reform: Perils and pitfalls. Teachers College Record(100), 164-180.
Tanes, Z., Arnold, K. E., Selzer King, A., & Remnet, M. A. (2011). Using Signals
for appropriate feedback: Perceptions and practices. Computers &
Education, 57(4), 2414-2422.
Tang, H.-W. V., & YIN, M.-S. (2012). Forecasting performance of grey prediction
for education expenditure and school enrollment. Economics of Education
Review, 31(4), 452-462.
Yin, P. Y., Chang, K. C., Hwang, G. J., Hwang, G. H., & Chan, Y. (2006). A
particle swarm optimization approach to composing serial test sheets for
multiple assessment criteria. Journal of Educational Technology & Society, 9,
3-15.

22

Vol. 11, No. 1, pp. 22-35, April 2015
A Study of the Development of Courseware and

Students’ Learning Effectiveness in Primary Education:
Using Three Teaching Techniques as an Example
Fang-Chun Ou
Overseas Chinese University
Taichung, Taiwan
Abstract. As Taiwan has carried on its educational reform, many

problems have emerged over the past ten years. These issues have to be
solved as soon as feasible. Specifically, primary education is facing
severity in competition and stern challenges in a fast globalizing world.
This study aims to explore Taiwan’s English education so as to find out
new approaches to revision and innovation. In Taiwan, most students
have to learn English since elementary school. English teachers usually
adopt different methods to teach students so as to achieve teaching
excellence. Three groups of primary school students participated in a
study with three teaching methods involved for learning English as a
foreign language (EFL). TPR (Total Physical Response) was employed
with the first group, giving instruction and then students responding
with body movement. CLT (Communicative Language Teaching) was
adopted with the second group, which emphasized interaction and
communication genuinely. Conventional teaching method was used
with the third group, in which students learned from what teachers
taught in class. The pre- and post-test were carried out to investigate
which teaching method was the most significant. The present study
indicates under controlled conditions that TPR & CLT, proven beneficial
in TPR & CLT context, can yield a positive outcome. In contrast, the
traditional teaching method has the least progress among the three
teaching methods. In addition, the findings of the study support that the
participants enhanced in the vocabulary and picture matching of the
posttest. The result of this study could be a good demonstration for
teachers to provide more options in English learning. Through the
© 2015 The author and IJLTER.ORG. All rights reserved.
23
curriculum, teachers could promote the ability to devise a flexible

variety of activities in order to stimulate pupils’ learning as well as make
them better interested in English. This study has set up a great value for
other similar researches and should be replicated with students at
various English proficiency levels.
Keywords: TPR (Total physical response), CLT (Communicative

language teaching), Conventional teaching method, EFL (English as a
foreign language).
Introduction
English is one of the indispensable languages in modern society, which can be
employed to do a great deal of trades between countries as well as spoken to
interact with foreigners. It is also an essential bridge that connects people from
variety of events. In response to the requirement of international society,
strengthening English ability has become an important issue of education.
Moreover, with English learning, learners can blend into social and cultural
activities in English-speaking countries in good time. Language learners should
understand and respect multiculturalism in order to be cosmopolite.
Nowadays, being capable of speaking English fluently has become one of the
basic requirements in the global village. The purpose of English teaching and
learning is to build up learners’ ability of communication, increase the
motivation and interest of English learning, and develop a global perspective.
Additionally, language learners are expected to enhance the ability of handling
international matters and conflicts.
It has been a quite normal phenomenon cultivating English capability since a

very young age, particularly in Taiwan. In line with the government policy to
improve international competitiveness, MOE (Minister of Education) stipulates
English teaching and learning should be implemented in Grade1-9 Curriculum.
According to MOE, the teaching methods should be active and interactive. The
content of teaching material should be related to daily life, practical and
interesting. By means of diverse teaching materials and activities participation,
the four skills including reading, listening, speaking, and writing can be built up
gradually, and then be put into practice.

24
In order to break the myth of grades, the aims of 12-year compulsory education
are leading students toward creative learning on the initiative, having
knowledge from the learning process, experiencing pleasure, cultivating their
own characteristics, and communicating with English. However, the major
problem of English education is teaching too much and too difficult. Teachers
usually make students remember vocabularies and grammars compulsorily.
Therefore, the mechanical drills kill students’ learning motivation toward
English. As a consequence, the policies of 12-year compulsory education are set
up to teach efficiently, blend information technology into teaching, and
encourage students to think and express creatively. In light of this, English
ability and practicality are more important than they used to. Additionally, in
2010, Attar and Chopra pinpoint the teaching methodology and approach
should keep changing in order to meet the needs of language learning. Namely,
how to design effective teaching modes and cultivate students' communicative
competence have become the major concerns in English teaching and research.
Tracking back to the early period, English teaching mostly put emphasis on
Grammar Translation Method and Audio-Lingual Method. The traditional
teaching method is drill-oriented, which introduces and practices language
knowledge and skills in details. Worse still, students tend to be bored and
punctilious gradually. Until 1994, Ministry of Education started highlighting
Communication Language Teaching, which aims at meaningful interactions,
language skills, genuine material, language ability development, and English
communication under different social situations properly. As a consequence,
designing diverse teaching techniques as well as appealing activities and
courseware should be taken into consideration so as to benefit students by
increasing achievement and learning outcomes.
Research Questions
1. Does the intervention in the use of teaching methods help improve
elementary school students’ English proficiency?
2. Which type of question (vocabulary, picture matching, and reading
comprehension) was influenced most after exposed to these three teaching
methods?

25
Literature Review
The advantages of Communicative Language Teaching have been proved and
employed successfully in ESL/EFL (English as a Second Language/ English as a
Foreign Language) classrooms around the world (Kelch, 2011). Chang (2011)
explores to compare the feasibility of Grammar Translation Method and
Communicative Language Teaching in English grammar teaching as well as tries
to find out which on is more appropriate in Taiwan. In his study, the result
shows students benefit more from grammar instruction with Grammar
Translation Method (GTM) adoption. With contrast to GTM, Communicative
Approach focuses on fluency rather than accuracy. The teacher corrects errors
immediately if the scope of the classroom activity is accuracy, but if the scope of
the activity is fluency the errors will be corrected later on. As a result, combining
both methods might be the best way to improve circumstances in English
grammar teaching. Wei (2010) reviews the advantages of Communicative
Language Teaching method and analyzes the obstacles of implementation in
EFL classroom context. In his study, it provides guidelines for compromising
CLT with the conventional teaching approach. Additionally, it recommends
some techniques and principles for English teaching implementation in EFL
environment.
Teacher Training
The main purpose of language education is to enhance the quality of teachers as
well as the quality of education. The English teachers should possess
professional knowledge related to ELT (English Language Teaching) and be
capable of employing varieties of teaching methods. Regarding mid- and
long-term teacher training (MOE, 1999), MOE encourages normal universities to
establish departments of English education. Besides, school should provide
English subgroups, English minors or second specialty students a twenty-credit
course of ELT.
Teaching Methods
TPR (Total Physical Response) was originally developed by James Asher. In the
1960s, TPR makes good use of physical movements and associates with the
theoretical framework of mother tongue. Most importantly, teachers can check
young learners’ comprehension through their reactions linked to body
movements, which reinforce their comprehension ability.

26
CLT (Communicative Language Teaching) emphasizes interaction and

communication in classrooms. In fact, it was a response to Chomsky’s theory
(Chomsky, 1965). Chomsky showed linguistic competence is not the mastery of
structures, but communication competence in real situation. Teachers should
create a wide variety of authentic situations for students to interact with their
classmates. Then students have the opportunity to share their individual
experience in target language. Most importantly, students gain more
self-confidence through practices and keep enthusiastic toward language
learning.
Methodology
Subjects
The target subjects were an unselected convenience sample. Thirty 5th and 6th
elementary school students voluntarily participated in this study. They were
asked to take the identical pre- and post-test to evaluate the appropriateness of
three different teaching techniques (TPR, CLT, conventional teaching) in
different classroom settings.
Course Material
The researchers created an innovative story that students have never read before.
In addition, ten sentences and vocabulary cards were made to emphasize
grammar instructions and practices.
Instruction and Testing Procedure

Three groups of subjects were administered the pretest to obtain initial scores of
the students’ English proficiency. There are three parts in the test. Part one is
multiple-choice questions of vocabularies, part two is matching correct pictures
according to the story, and the last part is reading comprehension. The actual
instruction lasted three hours with three different teaching methods adopted in
three different classroom settings, respectively. After the instruction, a posttest
was implemented to investigate the differences among the three different
teaching methods. The students completed both the pre- and posttest as the
requirement. All subjects were given the same test used in pre-test as a post-test.
Results
Analyses
The test contains twenty questions. Among these twenty questions, ten are
27
vocabulary, five are picture matching, and the remaining five are reading
comprehension. The result of the test focuses on which teaching technique was
the most suitable for primary school students.
Table 1 (Test Question Distribution)
Question Categories Numbers Percentage
Vocabulary 10 50 %
Picture matching 5 25 %
Reading comprehension 5 25 %
Results
The means and standard deviations of the pre-test and post-test scores for the
conventional teaching method were presented in Table 2.
Table 2 Descriptive Statistics of Pretest and Posttest (Conventional)
N=20
Conventional M SD
Pretest 25 11.055
Posttest 57 6.770
A paired-samples T test was conducted to evaluate whether the conventional

teaching method increases students’ scores. The results indicated the mean
scores for posttest (M= 57, SD= 6.770) was not significantly greater than the
mean scores for pretest (M= 25, SD= 11.055), t(9) = -8.677, p= .12 (Table 3). The
results revealed there is no effect of the conventional teaching method
adoption.
Table 3 Results of Paired Samples T Test
Pair 1 Mean Std. t df Sig.
Conventional Deviation
Pretest-posttest -32.50 11.844 -8.677 9 .12
The means and standard deviations of the pre-test and post-test scores for Total
Physical Response method were presented in Table 4.
Table 4 Descriptive Statistics of Pretest and Posttest (TPR)
N=20
TPR M SD
Pretest 25 7.45356
Posttest 76 10.28753

28
A paired-samples T test was conducted to evaluate whether TPR increases

students’ scores. The results indicated the mean scores of posttest (M= 76, SD=
10.28753) was significantly greater than the mean scores of pretest (M= 25, SD=
7.45356), t(9) = -11.057, p= .000 (Table 5). The results confirmed the effectiveness
and appropriateness of the total physical response adoption.
TPR Deviation
The means and standard deviations of the pre- and post-test scores of the
communicative language teaching method were presented in Table 6.
Table 6 Descriptive Statistics of Pretest and Posttest (CLT)
N=20
CLT M SD
Pretest 32 15.12907
Posttest 90 5.77350
A paired-samples T test was conducted to evaluate whether the communicative

language teaching increases students’ scores. The results indicated the mean
scores for posttest (M= 90, SD= 5.77350) was significantly greater than the mean
scores for pretest (M= 32, SD= 15.12907), t(9) = -16.900, p= .000 (Table 7). The
results confirmed the effect and appropriateness of the communicative language
teaching adoption.
CLT Deviation
The second question of the present study was the following “Which type of
question (vocabulary, matching, and reading comprehension) was influenced
most after exposed to these three teaching techniques?” A multivariate analysis
of variance (ANOVA) was performed on the data with the three scores (scores of
vocabulary questions, matching questions, and comprehension questions) used
as dependent variables and Group as the independent variable. The three
dependent variable scores were calculated by subtracting test scores of each
question type obtained at the beginning of the instruction (pre-test scores) from
29
those obtained at the completion of the instruction (post-test scores).
The ANOVA for the Group main effect was found to be significant, F (6,50)=
25.515 (Wilks’ Λ = .061), p < .001. As a result, the univariate ANOVAs on each
dependent variable were conducted as follow-up tests to the MANOVA. Using
Bonferroni method, each ANOVA was tested at the .0167 level (.05/3). There
was a significance in the vocabulary question scores, F (2, 27) = 63.224, p < .001,
eta squared = .824. The difference in the picture matching questions scores was
significant as well, F (2, 27) = 8.113, p = < .001, eta squared = .375. The difference
in the reading comprehension questions scores was nonsignificant, F (2, 27) =
25.317, p= .159, eta squared = .652. (Table 8)
Table 8 Results of Comprehension Difference Scores by Question Types
Note: adjusted Alpha = 0.0167

Findings
The first important finding of this study suggests that the teaching methods,
TPR and CLT do enhance elementary students’ English proficiency. The present
study demonstrates under controlled conditions that TPR& CLT, proven
beneficial in TPR & CLT context, can yield a positive outcome. In contrast,
traditional teaching method has the least progress among the three teaching
methods.
Moreover, the research evidence indicates that explicit, overt physical

movements can greatly increase the positive outcome of instruction. To students
who just listen to teachers and repeat after them do not possess much
comprehension because they do not really understand the context of the course,

30
nor do they know how to apply them to the real life. The teaching methods, TPR
and CLT can help students become more confident and have more involvement
in class.
Overall, the findings of the study support that the participants enhanced in the
vocabulary part (requiring respondents to select the best word according to the
picture) and picture matching (requiring respondents to choose the best sentence
describes the pictures) of the posttest. The second important finding of this
study deals with the question that which type of test questions was influenced
most after the instruction of these three teaching methods. It was found that the
participants in this study in fact did tend to use the physical movements to link
the meaning of the vocabularies. Besides, the pictures cards do assist them to
have better understanding of plots of the story.
Discussion
The first results show students achieve better improvement in TPR and CLT
classrooms. The reasons are provided as follow. Firstly, during the instruction of
TPR, instructors gave a lesson in target language, and students responded with
whole body actions. Students were not forced to speak, and instructors waited
until students acquire enough language input through listening comprehension,
then they would speak out without any fear. Namely, language learning should
not involve any stress and the lively interaction could impress the physical
response upon students’ mind.
Secondly, during the instruction with CLT teaching method, students were
taught the story along with picture cards, and they were asked to communicate
with instructors. By means of these, more interactions were expected. As a result,
students could keep the story in mind easier and more efficient.
Lastly, during the instruction with the conventional teaching method, instructors
taught by simply reading aloud the story lines and made explicit translation.
Compared to TPR and CLT, the conventional teaching method was not lively
that the students only sat tediously and sometimes did not catch what were
taught thoroughly.
The second results indicate that students achieve better toward vocabularies and
picture matching than reading comprehension. The reasons are explained in
31
detail.
Vocabulary
Vocabulary picture cards were created to employ during the instruction.
Students saw the picture at the first glance and were encouraged to guess the
meaning of the vocabulary. Then, vocabulary card was revealed and students
were requested to repeat after the instructors and sounded it out. Moreover, an
exciting game was designed for students to play in class. As a consequence,
students learned through the action and memorized novel words more easily
and efficiently.
Picture Matching
The story picture cards were used to associate and connect the pictures with the
content. Through viewing picture cards, students found the key words from
story lines, which enhanced their visual-mental correspondence. While having
an exam, students were easier to reason the story and match the right pictures.
Reading Comprehension
Instructors invented the story taught in class, and it has never been heard before.
Although students learned with the visual aid of picture cards and some exciting
games were set up especially for them, most of the students still had difficulties
reading as well as comprehending long paragraphs. As a consequence, while
having a test, students expressed they guessed instead of answering
conscientiously.
When it comes to TPR method, some recommendations are provided as follow.

First of all, realia is a good choice. Teachers can make good use of objects from
the real life to make the instruction more clearly and attract more attentions. In
addition to real objects, picture cards and posters are helpful as well. In fact,
students are able to associate the images of picture cards with new vocabularies
easily, which makes them have less pressure when memorizing new words.
Secondly, physical movement is strongly recommended. In class, the actions
demonstrated by instructors make the commands or instructions more
meaningful and clear. Moreover, students, especially young children, have more
interests in learning when they leave their seats and do some actions around.
Thirdly, instead of using a long sentence to direct students’ behaviors, teachers
can use combinations of commands. For instance, teachers give one command
32
first, and then do the action spontaneously. Gradually, when students are
familiar with the commands, teachers can add more commands at one time.
However, do
not add more than three commands at one time because students might get
confused while receiving the signals. Most importantly, teachers can observe
students’ comprehension easily and directly. If students can correctly do the
action after the command, then they do really comprehend what teachers teach
in class, which makes them feel confident and self-achieved.
With regard to CLT, there are some suggestions provided as follow. Situational
Language Teaching (SLT) can motivate students’ interests in learning. When
teachers introduce a new target language in words or phrases, instead of
translating them into students’ native language, teachers can demonstrate the
lessons through the use of realia, pictures or pantomime. Teachers may also use
intonation, rhythm, and concert pseudo-passiveness to get students’ attention
and motivate their interests in the lesson. Initially, students are really dependent
on their teachers. After teachers’ questions, students tend to make themselves
understand first, and then they are encouraged to answer in front of the whole
class. Gradually, with more practices, they may be more independent and have
greater security. Meanwhile, students can also listen to other’s opinions, and
learn from each other little by little. In fact, the interaction goes both ways, from
teachers to students and from students to teachers. Although students might
make mistakes, teachers usually employ various techniques to get students to
self-correct. Namely, the feeling of security is enhanced by many opportunities
of the cooperative interactions with their fellows and teachers. By means of this,
teachers evaluate not only students’ accuracy, but also their fluency. Teachers act
as advisors or co-communicator. Rardin (1988) mentioned language learning is
neither student-centered, nor teacher-centered, but rather teacher-student
centered. The CLT method makes students feel proud to use the knowledge to
express in different languages.
Two reasons are provided to explain why these three teaching methods were
chosen in the first place. First, TPR and CLT are the most popular teaching
methods adopted in educational institutions. Most instructors consider students’
interest in learning foreign languages is the priority. When students feel
interested in English, they will feel more comfortable and easy to communicate
with others by using a foreign language. Next, the traditional teaching method is
33
still employed now and then. Under the circumstance, most language learners
deem memorizing vocabularies a rather tough task; needless to say, speaking
English causes pressure and anxiety. Worse still, it surely lessens learners’
motivation toward English learning.
Teacher Training
During the past decade, the communicative language teaching approach has
been recommended especially for language teachers because of the essential and
emphasis of language use in foreign/second language classrooms (Mangubhai
et al., 2005). In addition, Li and Yu (2001) have identified the communicative
language teaching method has improved communicative ability of language
learners in which the conventional teaching approach has been demonstrated
unsuccessfully. However, due to the lack of sufficient teacher training in CLT,
teachers usually do not know how to implement CLT as well as do not possess
confidence in English speaking capabilities to carry out the communicative
approach (Butler, 2011). Specifically, most language teachers lack of this kind of
training and they are often afraid of “losing face” or feel embarrassed when
making errors or when they are not capable of answering students’ questions
promptly (Park, 2012). In light of the significance, Carrier (2003) points out the
different teaching approaches should be demonstrated and highlighted through
direct explanation, explicit teacher modeling, and extensive feedback in teacher
training programs in terms of the implementation in language classrooms.
Specifically, in the environment of English as a foreign language in Taiwan, the
supply of language input and practice opportunities are insufficient for the
learners to become immersed. Therefore, teachers should value process-oriented
instruction more highly than content-oriented or grammar-oriented instruction
because it is beneficial for students to become independent learners.
The language teacher should also bear in mind that elementary school children
are not mature enough to take full responsibilities for their own language
learning. Therefore, children’s proficiency levels and their cognitive maturity
would determine the types of activities (strictly-controlled ones, semi-guided
ones, or free communicative ones) the teacher puts into practice in a
communicative classroom.

34
Limitation
The sample size is small, which causes the effect of the experiments was not
statistically significant, so the results cannot be completely generalized to young
EFL learners from other areas. In addition, time duration of each class is two
hours. Within this short period of instruction, it was at time difficult for the
instructors to circle around the classroom while the activities were conducted
since the instruction involved observing the class and providing assistance.
Consequently, language learners would benefit from the instruction with
sufficient guiding period.
Pedagogical Implication
The result of this study could be a good demonstration for teachers to provide
more options in English learning. Through the curriculum, teachers could
promote the ability to devise a flexible variety of activities in order to stimulate
pupils’ learning as well as make them better interested in English. This study
has set up a great value for other similar researches and should be replicated
with students at various English proficiency levels. For instance, in addition to
TPR and CLT, The Direct Method, Community Language Learning, and
Reciprocal teaching are strongly recommended as the integrated teaching
method to promote the teaching process.
This study explores Taiwan’s education to find out new approaches to revision
and innovation. According to Jarvis and Atsilarat (2004), new teaching
approaches have been addressed so as to diversify the approaches in existence to
accomplish global innovation. As for the future investigation, more
breakthroughs in curriculum and instruction need to be put into consideration,
in order to gain an overall picture of the optimal outcomes of education.
References
教育部 (1999). 國小英語師資培育檢核相關報導。 Retrieved from

http://content.edu.tw/junior/english/scedu/rimage/r04.htm.
教育部 (1999). 培訓國小英語師資完整計畫方案。 Retrieved from http://
npl.ly.gov.tw/npl/report/880517/14.pdf.
Attar, M. & S. S. Chopra (2010). “Task-Based Language Teaching in India”. MJAL 2:4.

35
Butler, Y. G. (2011). The implementation of communicative and task-based language

teaching in the Asia-Pacific region. Annual Review of Applied Linguistics, 31, 36-57.
Carrier, K.A. (2003). NNS teacher in Western-based TESOL programs. ELT Journal, (3),
57- 242.
Chang , Shih-Chuan. (2011) “A Contrastive Study of Grammar Translation Method and
Communicative Approach in Teaching English Grammar” English Language
Teaching, Vol. 4, No. 2. Published by Canadian Center of Science and Education.
Chomsky, N. (1965). Aspects of the Theory of Syntax. Cambridge, MA: MIT Press.
Jarvis, H & Atsilarat, S. (2004). Shifting paradigms: from a communicative to a
context-based approach. Asian EFL Journal, (6), 4-8.
Kelch, K. (2011). Curriculum development in English language teaching: Innovations
and challenges for the Asian context. International Journal of
Organizational Innovation (Online), 3(3), 22-42.
Li, D. (1998). It's always more difficult than you plan and imagine: Teachers' perceived
difficulties in introducing the communicative approach in South Korea. TESOL
Quarterly, 32 (2), 677-703.
Mangubhai, F., Marland, P., Dashwood, A., & Son, J. B. (2005). Similarities and
differences in teachers' and researchers' conceptions of communicative language
teaching: Does the use of an educational model cast a better light? Language
Teaching Research, 9(1), 51-86.
Park, S. M. (2012). Communicative English Language Teaching in Korea. Humanising
language teaching, 14(6), 1-6.
Wei, H. (2010). Communicative Language Teaching in the Chinese Environment. US-
China Education Review, 7(6), 78-82. Retrieved March 15, 2011, from
http://eric.ed.gov/PDFS/ED511286.pdf.
Yu, L. (2001). Communicative Language Teaching in China: Progress and Resistance.
TESOL Quarterly, 35(1), 194-198

36

Vol. 11, No. 1, pp. 36-52, April 2015
A Comparative Examination of Teacher

Candidates’ Professional Practicum Experiences
in Two Program Models
Nancy Maynes, Anna-Liisa Mottonen, Glynn Sharpe and Tracey Curwen

Nipissing University, North Bay, Ontario
Abstract. This paper reports on one aspect of a larger study, examining

the relationship between teacher candidates’ self-reports of knowledge
and confidence related to many key areas of professional practice.
Survey information was provided by concurrent and consecutive
bachelor of education students. Perceptions of professional gains
through the practicum were examined. Students who are studying
education through a concurrent program feel that they have acquired
significantly more professional background about teaching through
practicum experiences than students acquiring a comparable degree
though a consecutive route. As the practical applied knowledge that
students acquire through practicum experiences is essential for teacher
development, this finding is relevant, especially as each of these
programs is undergoing structural changes as a reflection of new
provincial directions about teacher education. The results of this study
demonstrate that the amount and placement over time of practicum
provided in a teacher’s pre-service program matters to the level of
professional expertise they feel that they have acquired overall.
Keywords: practicum, consecutive education programs, concurrent

education programs.
Introduction
This paper reports on a study regarding whether or not pre-service teacher
candidates feel knowledgeable and confident in the acquisition of skills they
need to teach in their own classrooms at the completion of their respective
teacher preparation programs. The study contrasted responses from teacher
candidates who completed their teacher preparation programs in different
models. One group graduated through an eight month program, involving 13
weeks of classroom practicum time; the second group graduated with a 5 year
concurrent education degree, including 19 weeks of classroom practicum. The
focus of this study is on teacher candidates’ perceptions of what is gained
through practicum experiences in the classroom. We investigated how effective

37
in some tasks new teachers perceive themselves to be as a direct result of what

they have learned through practicum experiences.
Background
Theories may provide the knowledge that teacher candidates require to work
effectively with students in the classroom. However, without opportunities to
apply these theories to practice during practicum time, candidates may lack the
necessary confidence to address new contexts with equal effectiveness, and they
may lack the pedagogical content knowledge to determine strategy efficacy as
they encounter new situations early in their career. Practicum time in a teacher
education program is typically designed as a professional internship of short
duration, strategically placed in the teacher candidates’ professional program.
The practicum allows the teacher candidate to try out ideas that they have
learned in courses in the context of a classroom where a certified teacher can act
as a mentor for them.
However, not all teacher preparation programs provide the same amount of
classroom practicum experience for teacher candidates. In the jurisdiction where
this study took place, teacher candidates are required by their accreditation body
to acquire a minimum of 12 weeks of successful practicum experience. Success in
the practicum is assessed by the professional judgment of the mentor teacher,
who is referred to as an associate teacher (AT) in this jurisdiction. In this study,
however, two paths to acquiring the professional teacher accreditation are
examined in relation to the perceived impact of the practicum on knowledge and
confidence of the new teacher. Students acquiring their accreditation through a
consecutive program route in this jurisdiction engage in 13 weeks of practicum
(i.e., one week more than required by the local accreditation body), while those
who acquire their accreditation through the concurrent program route acquire
19 weeks of practicum (i.e., 7 weeks more than required by the local
accreditation body). Additionally, the 19 practicum weeks in the concurrent
program are distributed across the 5 years of the program, while the 13 weeks of
the consecutive degree route are spread across 8 months.
While we acknowledge that the quality of the practicum experience each teacher
candidate may experience can be vastly different due to many circumstances,
our study focuses solely on examining perceptions related to how the length and
placement of the experience may have an instructional impact. As teacher
candidates, prospective teachers enter the professional arena through practicum
experiences; however, they are often unequally exposed to many learning
opportunities (Beck, Kosnik & Rowsell, 2007). It is logical to assume that more
time in a practicum context would allow more exposure to a greater variety of
learning opportunities. Many of the learning opportunities that a pre-service
teacher candidate may have during any practicum may be wholly dependent on
the skills and resources of the teachers to whom they are assigned for their
practicum. Additional practicum time may allow new teachers to have
otherwise unavailable exposure to strategies utilized by experienced teachers,
and they may lack contextualized opportunities to apply their course-based
knowledge in contexts that would allow the teacher candidate to develop

38
confidence in their ability to use these strategies if they have little or no time to
see them in operation and to adapt theoretical ideas to pragmatic contexts.
Therefore, the current study provides us with a benchmark of current reports of
knowledge and confidence acquired through practicum experiences on which to
base program design decisions for this aspect of teacher preparation.
Additionally, in the jurisdiction where this study is taking place, the government
has recently made significant changes to accreditation criteria, which will come
into effect in fall of 2015. In response to the demands for new program designs in
the accreditation program for teacher certification in this jurisdiction, many
accrediting institutions are considering the elimination of the concurrent
program route and retaining the single option of a 2-year consecutive program.
This study may shed some light on the efficacy of this decision as it relates to
decreased opportunities for longer program embedded practica.
Teacher preparation programs include a combination of course work in a

university setting, and internship style practicum placements in classroom
settings. In the jurisdiction where this study was completed, practicum
placements are arranged in any of 52 school boards in the province. Teacher
candidates are able to identify any three of these school boards as areas where
they might ultimately apply for a teaching position. Then, program placement
officers approach school boards to arrange the number of placments required in
their area. Usually, school boards have employees who are then responsible for
placing the teacher candidate in a specific classroom for a specific placement
block.
As this university offers two routes to the completion of the same bachelor of
education (B.Ed.) degree, with two approaches to the placement and differences
in the total amount of time provided for the practicum, we identfied the need to
compare teacher candidates’ perceptions of the relative value of these
differences in providing them with the skills and strategies needed to support
their developing professional skills to prepare to be successful with the role of
teacher. The skills that were identified for this aspect of the larger study were
selected because, while some theory for each skill can be provided in the context
of their courses, each skill could reasonably be expected to develop more fully if
teacher candidates had contextualized opportunities in schools to use these skills
and to consider the impact of their practices in relation to the outcomes they
achieved.
Six skills were identified by researchers in this category of professional practice.

They include: the ability to manage a classroom; the knowledge and confidence
to interact with parents; the knowledge and confidence to interact with school
and board administrators; the ability to manage difficult student behaviours; the
ability to deal with difficult situations; and the knowledge and confidence to
address the learning needs of all children.

39
Literature Review
During the past 15 years there has been a considerable amount of intensive
investigation into the value and learning afforded to teacher candidates whose
professional preparation program provides opportunities for them to hone their
theoretical course knowledge by participating in classroom placements, usually
referred to as practicum experiences, or collectively as practica. While we were
able to find many studies related to the perceived value of teacher practicum
experiences, there seems to be an absence in the professional literature regarding
investigations of the relative perceived value of different approaches to
providing the practicum experience and the perceived value of different
amounts of practicum experience. It seems reasonable to assume that more time
in a classroom practicum placement is likely to provide more opportunities for
the teacher candidate to gain a wider variety of professional skills, but there is a
dirth of literature about existing programs to support this contention.
Much of the existing research literature about teacher practicum placements

addresses perceptions of how effective this experience is as a contributor to the
overall professional preparation of a new teacher. A study by Brouwer &
Korthagen (2005) confirmed the role of the practicum in the overall development
of competent teachers. While both classroom theory and practicum experiences
were found to be contributors to a new teacher’s development, the practicum in
a school context was more influential than the course components of the teacher
education program on the development of teaching competence. However, the
nature of the practicum has also been found to matter when teacher competency
are the desired outcome. In a study by Beck, Kosnik, and Rowsell (2007),
researchers identified the need for more focus in the practicum on practical
issues related to the daily tasks of functioning in a classroom. In this study,
teacher candidates identified six characteristics or skills needed to be provided
and developed in their preparation programs to prepare them to teach,
including: theoretical understanding, practical knowledge and skills,
comprehensive program planning ability, knowledge of what must be done in
the first few weeks of school, understanding and skill in assessment and
evaluation, and knowledge of how to implement effective group work. It is
interesting to note that five of these six characteristics relate to implementation
practices that might be expected to develop in teacher candidates during their
practicum placements, even though the participants in the study also identified
the need to have theoretical understanding.
It seems clear from this study that prospective teachers recognize and value the
theoretical aspects of the preparation program to help them understand what
they should do, but they value the practical experiences of the practicum to
show them how and when to do these things. The Brouwer and Korthagen
(2005) study also demonstrated that by gradually increasing student teaching
activity complexity, by increasing cooperation among students (triads of student
teachers), cooperating teachers, and university supervisors, and by alternating
between student teaching and college (in-class) sessions, teacher education
programs allowed student teachers to relate theory and practice. This need for
balance between the course theory and the practicum experiences is supported

40
by the research of Ng, Nicholas, and Williams (2010). This research revealed that
pre-service teacher beliefs may also be influenced by placement experiences,
suggesting that placements may be an important factor in shaping beliefs about
teaching and teaching efficacy. Also, this study argued that teacher education
programs should strive to improve pre-service teachers’ teaching efficacy, since
efficacy leads to improvements in teaching ability.
Schultz (2005) provides support for the concept of day-to-day problem solving
capacity development through practicum learning. The study highlighted the
need for teacher preparation to support new teacher inquiry to help teacher
candidates use problem solving approaches when they face the day-to-day
challenges in a classroom.
However, other research shows that the type of school where a practicum takes
place influences the learning that a teacher candidate acquires from the
practicum. Results of a study by Ronfeldt (2012) demonstrated that teachers
who had field placements at easier-to-staff schools were more capable of
improving students’ test scores and also more likely to remain teaching in
challenging city schools during their first five teaching years. Thus, according to
the results of this study, teacher education programs should consider assigning
pre-service teachers to field placements at easier-to-staff schools. This study also
emphasized the importance of identifying what and how pre-service teachers
learn at easier-to-staff schools. The authors argued that it may be that these
schools have many characteristics of overall effective schools where good
teaching and learning flourish, such as high quality administration and support,
professional staff relations and collegiality, and more experienced teachers
(Ronfeldt, 2012). It seems logical that exposure to such contexts would influence
a teacher candidate’s learning about how to teach well.
The research literature about practicum experiences is also very clear about two
other key findings. First, the structure of a practicum matters to what can be
learned from it. Second, but far from less important, is the nature of the
relationship between the teacher candidate and the classroom teacher who hosts
their practicum placement is critically important to how successful that
placement will ultimately be, as measured by the teacher candidate’s
perceptions of their learning in a classroom context.
In the context of this study, practicum placements have some provincially

mandated expectations about the amount of placement required for certification.
However, there is no requirement outside of that minimum time that identifies
how or where a placement is structured. Therefore, teacher candidates may be in
a position to take advantage of opportunities that have been shown to have
considerable professional learning value. For example, in a study by Bryan and
Sprague (1997), teacher candidates reported their perceptions of the value of
having practicum opportunities to teach in a foreign country. First, the
placement was helpful with respect to initial hiring by helping teacher
candidates to obtain a teaching job and to improve the experience of initial
teaching interviews. Second, retention was positively affected by the foreign

41
country placement, as demonstrated by the 80% retention rate compared to the

50% retention rate characteristic of typical populations of new teachers. Third,
attitudes toward pupils were improved by the placements in foreign countries,
in that students reported developing more respect for individual differences
among students. Fourth, practicum experiences in foreign countries improved
attitudes regarding second languages. Teacher candidates’ appreciation for and
patience with speakers of second languages increased. Fifth, overseas
experiences increased curriculum choices, as teacher candidates were able to use
curriculum units they had previously used overseas. Sixth, overseas experiences
increased teaching flexibility, since teacher candidates often had to create
interesting lessons with limited materials. Seventh, overseas experiences
increased the variety of teaching strategies used by teacher candidates, since
they had to learn to understand different learning styles and adapt their
teaching to them (Bryan & Sprague, 1997).
The perceptions that these foreign experiences helped teacher candidates

manage learning for a diverse group of learners is related to the present study’s
investigation of perceptions of the ability to manage classrooms, manage
difficult behaviours, deal with difficult situations, and address the learning
needs of all children. Similar perceived advantages of foreign practicum
placements are also evident in several other studies (Grierson & Denton, 2013;
Maynes et al., 2012; 2013). The Grierson study documents how participating in
an international practicum in rural Kenya supported the development of global
perspectives in Canadian teacher candidates. Through analysis of data gathered
through a post-practicum focus group, individual interviews, and written
reflections, the participants demonstrated how they came to recognize the
significant assets of those who live in this developing country, in lieu of
perceiving their differences as deficits. Relating their insights to the literature
reveals how their experiences have prepared these teacher candidates to provide
global education and to differentiate instruction in response to the needs,
interests, and background experiences of diverse student populations. The
findings of this study provide support for international field experiences that
include cultural immersion and collaborative reflection, and document how the
provision of this practicum in association with the Canadian NGO Free the
Children provided these opportunities and had the potential to increase
participants’ teaching confidence and competence. Since both sets of
participants in the present study had opportunities to participate in international
practica, their skill sets and personal perceptions of knowledge and confidence
may have been influenced by these professional experiences, as well as by the
timing and focus of other practica.
Another structural variation in a teacher candidate’s professional practicum

experience that has been found to have a strong impact on professional
preparation is participation in practicum experiences with a peer. In a 2003
study that compared teacher candidates’ perceptions of practicum experiences
when they were alone in the classroom practicum, with a second group who
were partnered with a peer in the same classroom, they found that teacher
candidates preferred the pairing situation for its perceived advantages to their

42
learning (Bullough at al., 2003). In the paired context, teacher candidates

reported more opportunities and experience with creating lessons that were
more innovative; improved classroom management; and the paired participants
felt that the students in their classrooms learned more material and learned it
more quickly. When the pairing is done between a more advanced teacher
candidate (i.e., in a later year of a multi-year preparation program) and a teacher
candidate newer to the professional preparation context (Grierson at al., 2011)
the newer teacher candidates report positive learning through the pairing and
mentoring which they received from more experienced teacher candidates when
they participated in a paired practicum peer mentorship program. The majority
of these participants perceived the mentorship program enhanced their
confidence and professional growth, although some challenges were also
acknowledged. Clearly pairing teacher candidates in their practicum, for
different purposes, has perceived learning advantages for the participants.
The nature of the relationship between the teacher candidate and the classroom
teacher who hosts their practicum placement has been found to be critically
important to the perception of the teacher candidate about how valuable their
placement has been to their professional preparation to teach. A study by
Korthagen, Loughran, & Russell (2006) attempted to identify central principles
that can be used to create teacher education programs and practices that address
teacher candidates’ and teacher educators’ expectations, needs, and practices
accruing from a teacher preparation program. By analyzing three pre-service
teacher education programs (one from each of The Netherlands, Canada, and
Australia), the researchers identified seven principles of practice for teacher
candidate learning and for guiding change and improvement in teacher
education programs. The seven principles identify requirements for enhancing
the process of learning about teaching. These seven principles include the
perceptions that: learning about teaching involves continuous conflict and
competition among various demands; knowledge about how to teach should be
perceived as a subject that is yet to be created, rather than as an already created
subject; learning about teaching means that the teacher’s focus must be on the
learner, not the curriculum; teacher candidate research enhances learning about
teaching and teacher candidates can guide their own professional development
by conducting research on their own teaching practice; learning about teaching
requires that those learning to teach work closely with peers, in horizontal, not
vertical relationships; meaningful relationships must exist among the schools,
universities, and teacher candidates to promote learning about teaching; and, to
enhance learning about teaching, teacher educators should model the teaching
and learning approaches used in the teacher education program in their own
practice. All of the principles are based on learning from experience.
It is interesting to note that at least five of these seven principles of an effective

teacher education program can only be developed fully within the context of a
successful practicum component of the program. Furthermore, the researchers
note that the principles are interconnected, and improvements to all of them
consecutively will be the most effective approach to create programs that
address the expectations, needs and practices most suitable for both the new

43
teachers and their students. The principles reflect three major program (change)
components: perceptions of knowledge and learning that guide teacher educator
practices, program structure and specific practices, and staff and organization
quality (Korthagen, Loughran, & Russell, 2006). Each of these principles also
connects to the nature of the practicum and to its role in extending the
theoretical learning of a course into the situated learning of practice in the
classroom.
Other studies that highlight the critical nature of the relationship between the
hosting teacher in the classroom and the teacher candidate help to identify the
specific behaviours that support successful practicum experiences. Beck and
Kosnik (2002) identify seven components of the relationship between the
classroom teacher (often referred to as the associate teacher or AT) and the
teacher candidate including: the provision of emotional support by the AT;
having a peer relationship characterized by mutual respect (as opposed to a
supervisory one) with the AT; opportunities for ongoing collaboration with the
AT in planning but independence in teaching a lesson; room to be flexible with
the content and the methods they use to teach; feedback from the AT;
opportunities to observe good teaching by the AT; and a demanding but not
excessive workload while the teacher candidate is on a practicum placement.
This study highlights the complexities and the necessity of providing
opportunities for teacher candidates to develop their skills to interact
productively with other professionals and to deal with difficult situations in a
school context. It does not, however, address how such skills are developed in
teacher candidates.
The need for the development of such positive relationships is supported by the
work of Evelein, Korthagen, and Brekelmans (2008) in another study which
found that new teachers have much lower measures of need fulfillment in the
early stages of their careers than more experienced teachers. This difference
might be attributed to more skill, and therefore more success, in dealing with
day-to-day interactions that form the basis of classroom implementation. This
study is supported by the work of Ferrier-Kerr (2009) who found that the
professional relationships between the AT and the teacher candidates are based
on several factors, including: personal connections; interpretation or
understanding of respective roles (of the AT and the teacher candidate), the AT’s
style of supervision; and, engagement in reflective practice.
Grundoff (2011) studied first year teachers’ perceptions of how their practicum
experiences helped them prepare for early career teaching. Findings supported
the importance of the practicum in developing contextualized skills of the
profession but found that some practicum features supported skill development
while other features hindered development. Participants in this qualitative study
found that the practicum had many differences from the reality of actually
teaching in their own classrooms. Differences that facilitated the transition of a
teacher candidate into the role of teacher included feeling like they were part of
the school community once they were teachers; they felt respected by other

44
teachers and by students. They also valued having time to develop relationships
with the children. They enjoyed the increased sense of autonomy.
However, they also found some differences between the practicum and their
professional teaching roles that they felt had hindered their transition into their
professional teaching roles. Differences that hindered the transition included a
feeling of shock and anxiety upon beginning to teach due to a discrepancy
between expectations and reality, which mostly had an impact on teachers for
their first few weeks of teaching. Participants’ transition was disrupted in two
areas due to the mismatch. First, new teachers did not have a clear
understanding of what they had to do at the beginning of the year. Second, new
teachers did not recognize the size and scope of teaching. Practicum placements
were limited to practicing and developing skills within the context of the
classroom. As new teachers, they were overwhelmed by their responsibilities
outside of the classroom. As a result, student teachers underestimated the range
and amount of work, and were frequently tired as new teachers.
Findings suggested that teacher preparation programs should include a

practicum at the beginning of the year so that student teachers can experience
what needs to be done firsthand. Also, the authors argued that practicum roles,
relationships, and sites should be re-evaluated. Specifically three key
recommendations resulted from this study, including: practicum should be
viewed as an opportunity for collaboration so classroom teachers should have a
larger role in teacher education programs, and schools rather than classrooms
should be recognized as placement sites. By providing teacher candidates with a
school-situated practicum, the candidates can gain exposure to a variety of
teaching styles and also have embedded opportunities to experience the out-of-
classroom aspects of this professional role during practicum times.
Furthermore, the authors of the study reported that, with this structure
governing practice teaching, teacher candidates could experience more
interaction with administrators by being required to attend meetings and
perhaps participate in more interaction with parents.
Each of these prior research studies informed the selection of questions we used
to structure the current study. While the current body of literature about the
importance, nature, location, relationships, and structure of the practicum as a
component of a teacher preparation program has been examined, the
comparative perceptions of these characteristics across programs routes toward
a B.Ed. degree in the local jurisdiction does not appear to have been studied.
Method
Participants. Participants in this study were from both the consecutive and the
concurrent programs at three campuses from one Northern Ontario University,
during the 2011-2012 acadenic year. A total of 212 respondents (25 males,186
females, 1 gender not reported) completed the survey and were included in the
study. Respondents’ ages ranged between 18 and 58 years old (M = 23.18, SD =
4.91). Respondents were completing or had completed a consecutive teacher

45
preparation program (n = 81) or were completing or had completed a

concurrent teacher preparation program (n = 131).
Demographic data were collected to identify the details of each respondent’s

program route and the stage of completion of their teacher preparation. Of the
131 concurrent education respondents, 31 were in the final year or recently
graduated from their teacher education program. Twenty-one respondents had
previously graduated. Only data from those students completing the final year
of their B.Ed. degree or recently having completed this degree were included in
this aspect of the study.
Research Questions. Therefore, the 2 key questions for this aspect of the larger
study are: 1. Do the additional 6 weeks of practicum experience make a
perceived difference in the level of knowledge and confidence of the teacher
candidate?; and 2) Do student teachers perceive that the distribution across time,
of the practicum experiences, influences their knowledge and confidence as
teachers?
Measures. Demographics. Several types of demographic data were collected in

this survey to support comparisons across groups. Data about age, gender (0 =
male, 1 = female), the participants’ current status in the education program (i.e.,
concurrent or consecutive program; their year of graduation from consecutive or
concurrent), were collected for descriptive information and to allow
investigation of relationships between demographics and dependent variables.
Knowledge and Confidence. A total of 6 questions developed by the researchers

was used to assess teaching knowledge and confidence which participants
attributed to having related practicum experiences. Each of the 6 questions
focused on pre-service teachers’ perceptions of how the practicum may have
helped them develop abilities to address common classroom tasks.
Questions were responded to on a 5 point scale from 0 = definitely not to 4 =

definitely. The 6 questions were summed to obtain an overall total score that
could range between 0 and 30; higher scores indicated the perception that more
knowledge and confidence was gained from the practicum experience. Internal
consistency was calculated for the following six items related to the question
“How well do you think your practicum placements have prepared you to”:
manage a classroom, interact with parents, interact with administrators, manage
difficult behaviours, deal with difficult situations, and address the learning
needs of all children? Cronbach’s alpha demonstrated the measure (6 questions)
to be highly consistent (α = .89).
Procedure. An invitation to participate in a comprehensive study of pre-service

teachers’ confidence and knowledge was posted on an existing Facebook group
designed to promote professional support amongst teacher candidates. A brief
description of the purpose of the study was provided. This site included a link to
the participant information letter. Those who were interested in the study as

46
potential participants followed the link to the information sheet which provided
all informaton necessary for informed consent.
Potential participants could agree to continue or could exit the program, after
reading the introductory letter and examining the informed consent form.
Completion of the questionnaire indicated each respondent’s agreement to
participate in the study. One reminder of the opportunity to participate in the
survey research was posted on the Facebook site approximately one month after
the study was first advertised. Data collection was completed over a two month
period. Completion of the entire questionnaire required approximately 15
minutes. Only those questions related to perceptions of the value of learning as
a direct result of the practicum experiences were analysed for this
subcomponent of the larger knowledge and confidence study.
Results
Independent samples t-tests were conducted to compare participants from the
consecutive and concurrent education programs on their average responses to
each of the six survey questions.
For the question, “How well do you think your practicum placements have
prepared you to manage a classroom?” results demonstrated significant
differences between consecutive and concurrent program participants, t(207) = -
2.018, p = 0.045, with concurrent program participants (M = 3.30, SD = 0.89)
scoring, on average, higher than consecutive program participants (M = 3.05, SD
= 0.82).
prepared you to interact with parents?” results did not demonstrate significant
scoring comparably to the consecutive program participants (M = 1.35, SD =
1.24).
prepared you to interact with administrators?” results did not demonstrate
significant differences between consecutive and concurrent program
participants, t(207) = -1.429, p = 0.154, with concurrent program participants (M
= 2.20, SD = 1.24) scoring comparably to the consecutive program participants
(M = 1.95, SD = 1.25).
prepared you to manage difficult behaviours?” results demonstrated significant
= 0.98).

47
prepared you to deal with difficult situations?” results demonstrated significant
= 1.08).
prepared you to address the learning needs of all children?” results
demonstrated significant differences between consecutive and concurrent
program participants, t(207) = -2.489, p = 0.014, with concurrent program
participants (M = 3.27, SD = 0.81) scoring, on average, higher than consecutive
program participants (M = 2.96, SD = 0.93).
These results are displayed in Table 1.
Table 1
Comparison of Concurrent and Consecutive Education Program Participants’ Responses
to the Individual Items Related to the Question: How well do you think your practicum
placements have prepared you to:
Education Program
Concurrent Consecutive
M SD M SD t
How well do you think your practicum

placements have prepared you to:
manage a classroom? 3.30 0.89 3.05 0.82 -

2.018*
interact with parents? 1.45 1.30 1.35 1.24 -
0.568
interact with administrators? 2.20 1.24 1.95 1.25 -
1.429
manage difficult behaviours? 2.98 1.06 2.65 0.98 -
2.205*
deal with difficult situations? 2.80 1.08 2.46 1.08 -
2.265*
address the learning needs of all children? 3.27 0.81 2.96 0.93 -
2.489*
*p<0.05
When the six questions were combined to create a total score, results
demonstrated a significant difference between the participants from the
concurrent and consecutive education programs on average responses to the
overall question: How well do you think your practicum placements have
prepared you to…?, (t(207)=-2.186, p=0.030). Specifically, participants from the
concurrent education program (M=2.67, SD=0.86) scored higher, on average,

48
compared to participants from the consecutive education program (M=2.40,

SD=0.84).
The significant group differences related to 4 of the 6 indicators studied in this

survey; while there were differences in the average overall scores on the
remaining 2 indicators, those differences were not found to be significant. In
other words, responses from concurrent program participants indicated that,
generally, they thought they were better prepared by their practicum
placements, at least compared to consecutive program participants, whose
responses generally indicated that they thought they were less well prepared by
their practicum placements.
Next, we considered if the number of different classroom placements that

teacher candidates experienced on average could account for any differences in
their perceptions of the learning value of their practicum placements. The
combined participant group (consecutive and concurrent students together)
completed practicum placements in a minimum of 2 and a maximum of 4
different classrooms. The average number of classrooms experienced by this
group of survey participants was 3.39.
An ANOVA was used to compare the average responses on the 6 survey

questions in relation to the number of different classroom practicum placements
each person had experienced. There were no significant differences on any of the
6 areas of this set of questions that are attributable to the different numbers of
classrooms teacher candidates had experienced during their practicum
placements. In other words, the number of practicum placement classrooms had
no effect on how well the teacher candidates thought the practicum placements
prepared them to: manage a classroom, interact with parents, interact with
administrators, manage difficult behaviours, deal with difficult situations, or
address the learning needs of all children.
Discussion
This cross-sectional study was an attempt to understand the knowledge and
confidence of current and recently graduated faculty of education students
regarding their perceptions of how well prepared they felt to handle complex
interactions that are required in a classroom. Results indicate that teacher
candidates in this program are feeling well prepared to handle some of the
interaction tasks that will be required of them as teachers but feel severely
underprepared in other areas. Both groups of teacher candidates feel fairly
knowledgeable and confident in their ability to manage the day-to-day
operation of a classroom and to address the learning needs of all children. Both
groups in this study reported feeling they had achieved considerable knowledge
and confidence in both of these areas through their practicum experiences,
reporting 3.30 (concurrent) and 3.05 (consecutive) on measures related to
managing a classroom and 3.27 (concurrent) and 2.96 (consecutive) on their
ability to address the learning needs of all children. Since each of these abilities
is a crucial aspect of a teacher’s role in the classroom, these results are
encouraging, although significantly more so among concurrent students.

49
However, the remaining four measures of this aspect of the larger study are a
cause for concern about the efficacy of practicum experiences as they are
currently structured. While survey participants felt that they had developed
some knowledge and confidence to manage difficult behaviours in the
classrooms (2.98 concurrent; 2.65 consecutive), participants felt less
knowledgeable and confident in their ability to deal with difficult situations,
interact with administrators, and interact with parents (Table 1). In fact, our
results suggest that as teacher candidates move further and further away from
interaction with students and face the need to address situations with greater
complexity or which involve other adults in the school environment, they feel
less prepared to do so confidently.
Preparation to interact with parents is an area that is very weak for both
concurrent and consecutive teacher candidates, indicating that the practicum
experiences as they are currently structured, provide them with insufficient
exposure to these situations to allow them to develop knowledge and confidence
in this area.
In this jurisdiction, the practicum placement for both concurrent and consecutive
teacher candidates is evaluated by associate teachers and by faculty advisors.
The evaluations are structured by criteria for assessment, which include foci that
reflect the standards of the profession in the province. Among these standards is
a category of professional behaviours titled “Leadership in Learning
Communities”. The Ontario College of Teachers, the accreditation body for the
teacher preparation institutions in the province, explains this aspect of the
standard as follows (http://www.oct.ca/public/professional-
standards/standards-of-practice; Accessed January 28, 2014):
Leadership in Learning Communities  Members promote and participate in the

creation of collaborative, safe and supportive learning communities. They recognize their
shared responsibilities and their leadership roles in order to facilitate student success.
Members maintain and uphold the principles of the ethical standards in these learning
communities.
In most jurisdictions, this standard forms the basis of assessment of the teacher
candidates’ professional interactions in the school context. The standards are
broken down into observable behaviours that are assessed by both the associate
teacher and the faculty advisor at set times during the teacher preparation
programs, which differ across the concurrent and consecutive program routes.
Each Faculty of Education has internal control over how they define details
within each of the standards of practice. In the jurisdiction where this study took
place, these standards are associated with two professional behaviours: 1)
collaborating with others to create a learning community; and 2) assuming
professional responsibility (the planning binder, duties, meetings, punctuality,
and initiative). It may be that these behavior descriptions are too vague to draw
the attention of the teacher candidate, the associate teacher, or the faculty
advisor to the specific types of knowledge and skills addressed in this survey. It
is often said that what is assessed gets attention and that adage may apply in

50
this instance. If we were specific about the types of interactions that need
development through a practicum experience, we might expect that teacher
candidates could more fully develop knowledge and skills with classroom
management, interaction with parents, interaction with administrators,
managing difficult behaviours, dealing with difficult situations, and addressing
the learning needs of all children.
There might be value to consider a more targeted approach to teaching practica

than may exist in some jurisdictions at present. If newly revised teacher
education programs in this jurisdiction will allow for 18 weeks of practicum
experience before a teacher is accredited, those weeks could be focused on the
acquisition of specific skills that can only be learned in context, rather than on
the general application of theoretical skills and knowledge that have been taught
in course work. For example, later in the program, practicum opportunities
might include several chances for the teacher candidate to interact with parents
(e.g., through school committees, parent-teacher interviews, special events
which they may arrange to involve parents, etc.). Following these experiences,
associate teachers and faculty advisors would have a wealth of assessment
information about the teacher candidate’s skills in working with parents, which
can be used to provide guidance for further professional growth.
The findings in this study are timely as Ontario’s provincial jurisdiction is

currently undergoing significant program review to extend the program of
teacher preparation to align with requirements in most other Canadian
provinces. While the jurisdiction has a reputation of providing strong teacher
education programs, an additional year of preparation will allow for additional
course and practicum time. This combination should provide additional
opportunities for teacher candidates to acquire both theoretical knowledge
through further course work and pedagogical content knowledge through
practicum work.
However, to ensure that the combination of theoretical and practical knowledge

is acquired and that, therefore teacher candidates have an opportunity to
develop stronger pedagogical content knowledge, we believe that targeted
practicum experiences hold potential. During a targeted practicum experience,
teacher candidates would focus on developing strength and confidence in one
set of professional competencies during each teaching block, rather than being
diversified across the usual focus on five different sets of competencies.
Potentially, this could allow teacher candidates the freedom to explore, and use,
a variety of interaction strategies, thereby having greater opportunities to
develop knowledge and confidence in this area. Since schools are increasingly
focusing on a professional team approach and making efforts to partner with
parents in ensuring the success of the child, this approach to a targeted
practicum would provide a focus on learning supported by parents and other
professionals. Such an endeavor would seen to be invaluable as program review
efforts progress.

51
Conclusions
This study was part of a larger study, which investigated the knowledge and
confidence, which is perceived by teacher candidates as a result of many aspects
of their teacher preparation programs, including the practicum components.
Due to the length of the larger study, investigation into the practicum aspects of
the study focused on only 6 measures. It may be of value to examine the
practicum experiences in either or both of these participant populations more
thoroughly through more detailed questions and to triangulate survey responses
through other forms of data.
While we cannot over extend the interpretation of our data since it reflected
responses from only 212 teacher candidates. While this is a solid basis for some
conclusions, this number of participants represents only about 20 percent of the
total teacher candidate population from this university in the study year. It may
be that a larger participant group would reveal different trends.
As this jurisdiction undertakes significant structural changes in its teacher

preparation program and the practicum time involved in the consecutive
program increases from 13 to 18 weeks, over two years, it is expected that we
may see a close in the gap in perceptions of knowledge and confidence resulting
from practicum experiences in the two groups of teacher candidates (i.e.,
consecutive and concurrent). Additionally, work is presently underway to
develop more targeted skill development during practica than has previously
been used at this university. It will be important to monitor these experiences
closely as this new program delivery model is implemented.
References
Beck, C., & Kosnik, C. (2002). Components of a good practicum placement: Student
teacher perceptions. Teacher Education Quarterly, Spring, 81-98.
Beck, C., Kosnik, C., & Rowsell, J. (2007). Preparation for the first year of teaching:
Beginning teachers’ views about their needs. The New Educator, 3, 51-73.
doi:10.1080/15476880601141581
Brouwer, N., & Korthagen, F. (2005). Can teacher education make a difference? American
Educational Research Journal, 42(1), 153-224.
Bryan, S.L., & Sprague, M.M. (1997). The effect of overseas internships on early teaching
experiences. Clearing House, 70(4), 199-201.
Bullough, R.V., Young, J., Birrell, J.R., Clark, D.C., Egan, M.W., Erickson, L., Frankovich,
M.,Brunetti, J., & Welling, M. (2003). Teaching with a peer: A comparison of two
models of student teaching. Teaching and Teacher Education, 19, 57-73.
Evelein, F., Korthagen, F., & Brekelmans, M. (2008). Fulfilment of the basic psychological
needs of student teachers during their first teaching experiences. Teaching and
Teacher Education, 24, 1137-1148. doi:10.1016/j.tate.2007.09.001
Ferrier-Kerr, J.L. (2009). Establishing professional relationships in practicum settings.
Teaching and Teacher Education, 25, 790-797. doi:10.1016/j.tate.2009.01.001
Grierson, A., & Denton, R. (2013). Preparing Canadian Teachers for Diversity: The
Impact of an International Practicum in Rural Kenya. In L. Thomas (Ed.), What is
Canadian about Teacher Education in Canada? Multiple Perspectives on Canadian
Teacher Education in the Twenty-First Century (pp 187-210). Sherbrooke, PQ:

52
Canadian Association for Teacher Education/Association Canadienne pour la

formation à l’enseignement.
Grierson, A., Cantalini-Williams, M, Wideman-Johnston, T., & Tedesco, S. (2011).
Building scaffolds in the field: The benefits and challenges of teacher candidate
peer mentorship. Brock Education, 20(2), 85-103.
Grudnoff, L. (2011). Rethinking the practicum: Limitations and possibilities. Asia-Pacific
Journal of Teacher Education, 39(3), 223-234. doi:10.1080/1359866X.2011.588308
Korthagen, F., Loughran, J., & Russell, T. (2006). Developing fundamental principles for
teacher education programs and practices. Teaching and Teacher Education, 22,
1020-1041.doi:10.1016/j.tate.2006.04.022
Maynes, N., Allison, J., & Julien-Schultz, L. (2013). An Examination of Longevity of
Impact of an International Practicum Experience on Teachers’ Beliefs and
Practices Four Years Later, International Education Studies.6 (4).
Maynes, N., Allison, J., & Julien-Schultz, L. (2012). International practica experiences as
events that of influence in a teacher candidates’ development. McGill Journal of
Education, 47 (1), 69-90.
Ng, W., Nicholas, H., & Williams, A. (2010). School experience influences on pre- service
teachers’ evolving beliefs about effective teaching. Teaching and Teacher
Education, 26, 278-289. doi:10.1016/j.tate.2009.03.010
Ronfeldt, M. (2012). Where should students teachers learn to teach? Effects of field
placement school characteristics on teacher retention and effectiveness.
Educational Evaluation and Policy Analysis, 34(1), 3-26.
doi:10.3102/0162373711420865
Schulz, R. (2005). The practicum: More than practice. Canadian Journal of Education,
28(1&2),147-167.

53

Vol. 11, No. 1, pp. 53-74, April 2015
A Study of Formative Assessment Strategies in

Teachers‘ School-Based In-Service Training
Eva Nyberg and Mona Holmqvist Olander

Department of Pedagogical, Curricular and Professional Studies
University of Gothenburg
Gothenburg, Sweden
Abstract. The aim of this study was to explore a team of teachers‘ (n=4)
use of theoretically based formative assessment strategies within the
course of a learning study. The thematic analysis is based on video
observations of teachers‘ discussions during planning meetings,
teaching in the classroom and evaluation meetings. The subject-specific
content focused on learning about fractions, specifically the concepts of
double and half, in three groups of six- to seven-year-old students (n=51
in total). An iterative process was used in which the teachers in the
study used video recordings as a tool for analyzing their work in the
classroom. The thematic analysis shows that the use of a general
learning theory – variation theory – strengthens the effect of the
teachers‘ formative assessment. Without explicit use of the assumptions
from the theoretical framework, the formative assessment strategies had
only a minor impact on students‘ learning outcomes.
Keywords: Formative assessment, Classroom study, Elementary school,

Variation theory; Learning study.
1. Introduction
There have been several attempts to develop teachers‘ formative assessment by
means of in-service training. A range of studies concerning programs for in-
service training aiming at developing teachers‘ competence in performing
formative assessment have been carried out, with various outcomes (e.g., Bennet
2011, Phelan, Choi, Vendlinski, Baker & Herman, 2011). It thus seems rather
difficult to transform formal training about formative assessment into classroom
practice. Wiliam (2006) claims that ―tools for formative assessment will only
improve formative assessment practices if teachers can integrate them into their
regular classroom activities‖ (p. 287). School-based in-service training could
therefore be one way to develop teachers‘ abilities to use formative assessment
to increase the students‘ learning outcomes. This school-based research project
involved a team of teachers who had previously participated in an in-service
training course on formative assessment, and the focus was on their use of

54
formative strategies in their classrooms during an iterative process in which the

researchers were partly involved.
Formative assessment can be approached in several different ways. In this study,

the in-service training was aimed at encouraging the teachers to base their
formative assessment actions on theoretical conjectures. Our hypothesis is that
formative assessment strategies such as feedback become more powerful if they
are based on a theory of learning, and we therefore explore the effect of applying
the principles of variation theory (e.g., Marton, 2014) during planning,
implementation and evaluation of teaching and learning, in order to increase
student learning.
Research has shown that formative assessment often lacks a theory of action,
which makes it difficult to evaluate and understand the mechanisms causing the
intended effects:
Unless we understand the mechanisms responsible for change, we won‘t

know if the effects are due to those mechanisms or to irrelevant factors.
We also won‘t be able to predict the conditions, or population groups, for
which the formative assessment is likely to work (Bennet, 2011, p. 14).
Bennet (2011) also argues that a teacher‘s hypothesis about a students‘

understanding is dependent on the teacher‘s cognitive model, which can help
the teacher to evaluate a student‘s understanding. Without a theory of learning,
there is a risk that the teachers carrying out the formative assessment will fail to
identify the underlying mechanism for how the learning takes place. This is also
stressed by Black and Wiliam (2009, 2012) and Wiliam (2009), who argue that to
know what feedback to give to students, the teacher needs both a theoretical
model of how students learn and the ability to apply this theoretical
understanding in a specific context. In light of this, this research project aims to
further study the way in which the theoretical assumptions of variation theory
can be used as guiding tools for teachers to assess their students formatively.
In formative assessment, the data gathered during assessment is used to inform

decisions regarding modifications and adaptations of the teaching to meet the
learning needs of the students (e.g., Bennet, 2011; Black & Wiliam, 2009). This
includes using evidence about difficulties collected in one group of students ―to
modify instruction for another group of students at some point in the future‖
(Wiliam, 2009, p. 26). This means that it is only if the result from the assessment
is used to inform teaching and learning that it can be said to have a formative
function. Phelan et al. (2011) produced a study pointing out the difficulties in
identifying what constitutes effective formative assessment. They studied
teachers and students in a formative assessment intervention, aimed at
improving student performance in mathematics. Surprisingly, the authors did
not find any significant differences between the treatment group and the control
group; the hypothesis that the improvement would be greater in the treatment
group—in which formative assessment was used—remained unverified.
Although the extensive research overview by Black and Wiliam (1998) delivered

55
the clear message that more formative assessments in school can lead to great
improvements in student learning, the above-mentioned study by Phelan et al.
(2011) emphasizes that it is not simply the case that any formative assessment
tool leads to an improvement in student performance. The teachers‘ competence
in performing the formative assessment is most likely crucial for the outcome,
and therefore teacher variables need to be more closely analyzed. What matters
seems to be how the formative assessment is carried out, i.e., what is noted by the
teacher and the way in which this is addressed during the instruction. This means
that effective formative assessment is constituted by both knowledge of the
content in question and knowledge of what it takes to learn this specific content,
in line with Bennet (2011), as well as Black and Wiliam (2009). This is the reason
why we have added a theoretical framework to strengthen the teachers‘
knowledge about learning during the in-service training.
In this article, we have taken into consideration the fact that to be formative, the
instruction needs to be specific at a micro-level in relation to what is elicited
through the assessment. This involves determining in what way the content is
offered in relation to how it is experienced by the students and doing this
according to theoretically based assumptions about what it takes to learn. The
teachers in our study were guided by the variation theory of learning (Lo &
Marton, 2012; Marton & Booth, 1997; Marton, 2014; Lo, 2013; Holmqvist,
Gustavsson & Wernberg, 2008; Holmqvist, 2011), to which they were introduced
by the researchers participating in the interventions during the introduction to
the in-service project. The concept of variation in variation theory does not refer
to varying methods, but rather to varying the features of the content that have
not been previously discerned by the students. The assumptions are that to
discern new aspects of the object of learning, these have to vary against an
invariant background consisting of the aspects already known. Variation theory
will be more thoroughly described below.
2. Aim and research questions

The aim of this study was to explore a team of teachers‘ (n=4) use of theoretically
based formative assessment/feedback strategies during a school-based in-
service training in a primary school. The subject-specific content focused on
learning about fractions, specifically the concepts of double and half, in a group
of six- to seven-year-old students. The research questions were:
1. In what way do the teachers take advantage of the variation theory of

learning in their formative feedback to the students?
2. What impact does the theoretically based formative assessment have

on the teachers‘ way of constructing lessons in new groups of students?
3. The iterative in-service training process

In this section we describe the tools used by the teachers in their in-service
training: variation theory (theoretical framework) and the learning study process
(an iterative process including video recordings).

56
3.1 Variation theory as a guiding principle

Educational learning theories often describe learning in terms of conditions for
learning. For example, a theory might state that learning takes place in an
interaction between people, or as an internal construction by the individual, or
through a combination of both. However, these theories do not usually offer
explicit guidance on how to develop domain-specific learning or how to use the
theories in practice. Variation theory is built on the research field of
phenomenography, which originated in the 1970s and is concerned with
studying qualitatively different ways of experiencing the same phenomena
(Marton, 1981; Marton & Booth, 1997; Marton, 2014). Variation theory is thus
useful in providing a way to analyze what it takes to learn. It is also used as a
guiding principle in lesson planning (Holmqvist, Gustavsson & Wernberg, 2008;
Kullberg, 2010; Lo & Marton, 2012). Its point of departure is non-dualistic,
meaning that the content to be learned cannot be separated from the learner‘s
understanding of it (Lo & Marton, 2012).
Discernment, simultaneity and variation are cornerstones in variation theory. To

learn, one has to discern what one has previously not been aware of and relate it
to what one already knows. By varying aspects of the content that have not been
previously discerned by the learner against an invariant background of aspects
that are already known, new aspects become discernible. These patterns of
variation are taken into consideration during planning (the intended object of
learning) and implementation of the lesson (the enacted object of learning), and
when analyzing the learning outcomes shown by the students‘ test results (the
lived object of learning). One important point is that the pattern of variation will
differ depending on the students‘ previous knowledge; it is based not only on
what it takes to learn in relation to specific content but also on what it takes to
learn specific content in relation to what is already known by the learner (Pang
& Marton, 2013). The differences in improvement are thus not due solely to the
design, method or other strategies used – for example formative assessment
techniques as such (Phelan et al., 2011) – as the students‘ perspective has to be
addressed in an adequate way if the full power of formative assessment is to be
exploited. The design of the instruction aims to make visible those aspects that
are necessary for further learning, called critical aspects. In variation theory, this
refers to aspects of the content that have not yet been discerned by the learner
but have to be discerned in order to develop further knowledge. As soon as the
aspect is discerned, it stops being critical. If a lesson offers only aspects of the
content that are not critical for learning, for example aspects that the learner has
already discerned, then no learning will take place.
3.2 Learning study as a school developmental model

The learning study is a model developed by researchers from Hong Kong and
Sweden (e.g., Lo, Pong & Chik, 2005; Holmqvist, 2010; Marton, 2003; Marton &
Tsui, 2004), inspired by the Japanese lesson study (Lewis, Perry & Murata, 2006).
It entails a systematic and cyclical process consisting of a number of stages,
starting with the choice and study of the object of learning to be addressed and
taught. In this iterative model, the design of new lessons is based on the analysis
of earlier lessons, with the aim of further developing students‘ previous learning
outcomes. The object of learning is initially chosen by the teachers and defined

57
in detail after a screening, an interview or test to assess students‘ pre-

knowledge, which identifies the aspects that may be critical for learning. A
pretest is thereafter constructed based on the results of the initial screening and
guided by the chosen theory of learning, in this case, variation theory. The
teachers and researchers collaborate in planning the first lesson, which is taught
by one of the teachers in the group. The lesson is video recorded. During the
subsequent meeting the lesson is analyzed by the teachers and the researchers
using both the video recording and the results of the pretest and posttest. The
focus in these meetings is the learning outcomes, which provide the basis for the
development of a new lesson to be taught to another group of students. If the
learning study also has a research aim, an analysis of the whole process is made
to report the results of the study. Learning studies, which involve teachers and
researchers in collaboration (Marton & Tsui, 2004), have been implemented in
120 schools in Hong Kong in a project called ‗Variation for the improvement of
teaching and learning project‘ (VITAL). The project has been evaluated (Elliott &
Yu, 2008) showing the benefits for both student and teacher learning (Elliott &
Yu, 2013). On the other hand, research on the collaboration between teachers and
researchers shows tensions of several different kinds: ―outsiders‘ versus
insiders‘ perspectives; academic versus grounded knowledge; unclear
hierarchical statuses; and diverse and conflicting agenda‖ (Adamson & Walker,
2011). Empirical studies on the implementation of learning studies in other parts
of the world, in this case Sweden, have also found them to be effective both for
students and for professional development among teachers (Holmqvist, 2006,
2011). The first research project where learning studies were implemented in
Sweden was funded by the Swedish Research Council (Holmqvist, 2002), and 18
learning studies in three different school subjects were studied (Gustavsson,
2008; Wernberg, 2009; Kullberg, 2010). The results show an increased learning
outcome both in the short-term and the long-term (Holmqvist, 2011). In a similar
model for school development, the lesson study (Stigler & Hiebert, 2009),
teaching is developed through a similar cyclical process. The main difference is
that learning studies are guided both by a theory of learning and a focus on
content-specific research into students‘ understandings of the specific object of
learning (Holmqvist, Gustavsson & Wernberg, 2007, p. 189). The focus in
learning studies is on the learners‘ understanding of the content, while the focus
in lesson studies is mainly on the improvement of the lesson itself or on other
issues needing to be improved in the classroom. The results of an evaluation of
the Swedish learning studies show that the teachers see learning studies as time-
consuming and difficult to work with continuously, despite the high student
learning outcomes (Olteanu & Lennerstad, 2011). Lewis (2015) points out the
lesson study as a so-called improvement science, which theorizes the need for
two different types of knowledge sciences: a system of basic knowledge from the
discipline of education and a system of profound knowledge. The learning study
process, in which a combination of theoretically based conjectures and the
teachers‘ deep knowledge about teaching is needed, also seems to fulfill the
requirements for being labeled as improvement science.

58
3.3 Formative assessment in the learning study model

In a learning study, the aim of the assessment is to make in-depth microanalyses
of the students‘ learning in order to identify what the critical aspects of the
content in question are in relation to the learners, and how these aspects can be
made discernible in a powerful way. Having identifying the critical aspects, the
teachers use and try out this information in their formative assessment—that is,
they use it to guide their teaching in the classroom. Our hypothesis, therefore, is
that formative assessment informed by variation theory enables teachers to
pinpoint the aspects that should be focused on in the learning situation and to
change this focus depending on the learners‘ needs. James and Pedder (2006),
when discussing results from a project including a survey of 1000 teachers,
assert that programs of professional development ―should be focused on
classrooms and classroom practice‖ (p. 39). One of their conclusions is that both
individual and social processes are ―important conditions for the promotion of
assessment for learning in classrooms‖ (James & Pedder 2006, p. 39). They put
forward the concept of ―research lessons‖ as described by Stigler and Hiebert
(1999) as one possible approach to developing assessment practices in the
classroom. Pedder and James (2012) further stress the importance of
collaborative, classroom-based professional learning ―for fostering effective
assessment for learning‖ (p. 41). We agree with these claims, and consider the
learning study as a school-development process in line with the findings about
powerful professional development. By the use of a theoretical perspective on
learning, such as variation theory, the formative assessment in the learning
study helps the teachers to understand what needs to be changed in instruction
or which feedback is relevant in order to increase students‘ learning, using the
theory as a guiding tool.
4. Procedure of the Study
4.1 Method
The analysis of the teachers‘ strategic use of variation theory is qualitative and
based on video recorded meetings and interventions as well as observations. A
thematic analysis was made (Boyatzis, 1998) based on several readings of the
material (which was transcribed verbatim), as well as watching and re-watching
of the video recorded meetings and lessons. The students‘ summative
assessments were used as a triangulation to strengthen the observations and find
out whether the teachers‘ use of theoretical assumptions was reflected in
students‘ learning outcomes.
4.2 Context
The study took place during an in-service training project, which was conducted
in a school district in a rural area close to a small town in the south of Sweden. In
the Swedish school system, all classes are mixed with regard to both gender and
abilities. Students spend 9 years in compulsory school, from age 7 to 16. The first
time the children receive grades is at the age of 12, i.e., in grade 6. All classes
include children of the same age and during the first 6 years, the classes
normally consist of groups of 15-25 children.

59
The content of the lessons in the Swedish school system is governed by a

national curriculum (Swedish National Agency for Education, 2011), but the
design of instruction and methods used is left to the teachers. The education to
become a primary school teacher consists of four years of academic teaching
studies and vocational training at university level.
4.3 Participants and implementation

Two researchers and four teachers from two different primary schools
participated in the study, which took place over one semester. 51 seven-year-old
students (24 girls and 27 boys) were involved in the study. The students‘
learning process and learning outcomes are not, however, in focus in this paper;
the findings about the students form another part of the project (Holmqvist &
Nyberg, 2014). Three of the teachers were each in charge of one lesson, including
pretest, research lesson and posttest. One teacher, who was responsible for
developing mathematics instruction in the school district, participated only in
the planning meetings. Before the first meeting, as a part of the overall in-service
project in the school district, the teachers were introduced to variation theory
and the learning study model in a lecture by one of the researchers. The teachers
were also given a book about the theoretical background and concept of the
learning study (Holmqvist et al., 2006). The researchers‘ deeper knowledge of
variation theory, however, was important throughout the discussions. The object
of learning chosen by this group of teachers was the ability to double and halve
numbers. This topic was chosen based on the teachers‘ previous observations
that children had difficulty learning how to perform these operations with
numbers, but not with concrete objects. The teachers had previously been
introduced to formative assessment.
The lessons and the planning meetings were video recorded. Before each new
lesson, the recording of the previous lesson was analyzed, and experiences from
that lesson were evaluated and discussed, including the results of the pretest
and posttest. Three lessons were conducted, meaning that there were four
meetings between the teachers and researchers (see Fig. 1). During the first
meeting, the pretest was constructed and the first lesson was jointly planned.
During the second and third meetings, the previous lessons were evaluated with
respect to learning outcomes, and the coming lessons were planned on the basis
of this evaluation. During the fourth meeting, the last lesson and the results from
the tests were discussed, as was the outcome of the learning study as a whole.
RESEARCH RESEARCH RESEARCH

PM 1 LESSON 1 LESSON 2 LESSON 3
PM 2 PM 3 EM
Week Week 1, 1st Week 2, 2nd Week 3, 3rd
Week Week Week
group of group of group of
1 students, 2 3 students, 4
students,
1st teacher 2nd teacher 3rd teacher
Figure 1: The four-week time-line of the learning study. PM=Planning meeting,

EM=Evaluation meeting.

60
The length of intervention for each group of students was one lesson comprising
approximately 15 minutes of pretest, 30 minutes of instruction, and 15 minutes
of posttest. The lessons were planned and evaluated on the basis of variation
theory.
4.4 Data
The data collected during the school-based study consists of video recorded
meetings (3), video recorded lessons (3) and one final observed meeting.
Participating observations: Four planning meetings with the teachers were
conducted during the four weeks of the learning study. Each meeting took place
at the schools where the teachers worked. The meetings lasted 2 hours.
Videotaped lessons: Three lessons were videotaped. Each lesson lasted 1 hour.
Group 1 (Cycle 1) consisted of 24 students, group 2 (Cycle 2) consisted of 13
students and group 3 (Cycle 3) consisted of 14 students.
4.5 Analyses
The video recordings from the lessons (n=3) and planning meetings (n=4) were
transcribed verbatim. The data was analyzed as an exploratory single case study
(Stake, 2006; Yin, 2009) at a fine-grained level (Phelan et al., 2011). For the
qualitative analysis of the teachers‘ use of theoretically based formative
assessment strategies and reasoning, a thematic analysis was used as a first step
(Boyatzis, 1998). The analysis was based upon how the core concepts of variation
theory – contrast, simultaneity and variation (Holmqvist, 2011; Kullberg, 2010;
Marton, 2014; Lo, 2012) - were used by the teachers, and in what respects they
were used in the planning, implementation and evaluation of each lesson.
Thereafter a more detailed and specific analysis was made across the three
lessons including the planning and evaluation meetings, during which the
different data sources were compared and analyzed in parallel.
4.6 Ethical considerations

The study followed the ethical considerations described by the Swedish
Research Council (Hermerén, 2011). All participants, including parents, teachers,
school leaders and children, were continuously informed about the aims of the
study, the use of the data and their rights to confidentiality and to withdraw
from participation. The parents also signed an agreement to let their children
participate in the study and to enable the researchers to use the data.
5. Results
The analysis describes the different ways the teachers‘ formative assessment
strategies are expressed during the study. The analysis ends up with three
themes regarding the teachers‘ development of formative assessment strategies
guided by the theoretical assumptions. The first theme describes how the
teachers, through their formative assessment, gradually developed insight about
what critical aspects are and how they can be used to increase the students‘
understanding of halving and doubling (Theme A). The second theme highlights
another example of how the teachers‘ formative assessment led to increased
evidence-based insight into the students‘ understanding during the course of the

61
learning study (Theme B), by taking into consideration the connection between
the object of learning and the learners pre-knowledge (non-dualism). The third
theme addresses the teachers‘ feedback during their respective lessons, focusing
mainly on how the teachers used variation theory (Theme C) in their feedback to
the students.
5.1 Theme A: Critical aspects of the content

The knowledge about what critical aspects are and how they can be dealt within
instruction is the main focus in this theme. The teachers found that to
understand the concept of doubling, the students had to discern variation in
amounts at the same time as discerning the invariant aspect of doubling—that is,
taking the same amount ―one more time‖ and adding it to the original amount.
This section presents the analysis of how this was addressed and handled
during the course of the learning study.
5.1.1. The first meeting

During the first planning meeting, the test questions and the design of the first
lesson were discussed and constructed. The design was based on the results
from the screening carried out by the teachers themselves, the teachers‘ previous
experiences from teaching this content, the researchers‘ knowledge of common
obstacles for students, and the researchers‘ extensive experience of using
variation theory for test construction and lesson planning. The various aspects
the object of learning were analyzed, to determine which aspects the students
had already discerned and which ones they needed to discern in order to
develop their knowledge.
During the screening, when one of the students was asked by the teacher to
double the number four, the child answered five (i.e., four plus one). This
student evidently interpreted ―add one more time‖ as ―add one (the number 1)‖.
During the planning meeting before the first lesson, one of the teachers
confirmed that this was a critical aspect of the object of learning.
Excerpt 1, first planning meeting (T1 = Teacher 1):
T1: Yes, we have many children who cling to that; they don‘t get
doubling, for them it is … plus one. We tried previously when we taught
this particular topic not to start with one, because we thought that was
what made them stick to this, but there are those who still hold on to
―plus one‖ even if we start by asking them to double three.
The teachers and the researchers consequently set up the hypothesis that one
important part of the instruction might be to avoid the expression ―one more
time‖, which the students could interpret to mean ―plus one‖. Instead, the
expression ―the same amount one more time‖ was to be used during the first
research lesson to make it possible for the students to discern that ―one more
time‖ is not the same as ―one more‖. For the students who had not already
noticed the difference, this might be a critical aspect.

62
One of the concepts in the theoretical framework used here is about how to
make aspects of the content discernible for the students. In this case the concept
of simultaneity was used. The first lesson was therefore planned to allow the
students to apply doubling and halving at the same time, but without having the
original number of objects on their desks. The decision not to let the students see
the objects representing the original number was based on previous experience
of students making a mechanical visual doubling, without really understanding
the concept of doubling, e.g., by adding the same amount once again, which
results in an understanding of double as ―the same as‖; doubling four then gives
four instead of eight, as the student added four to the original amount.
5.1.2 Second meeting

The first lesson was evaluated in the second meeting. The analysis during the
meeting of the pretest and posttest focused on the theoretical concept of
simultaneity. However, in their analysis of the test results, the teachers found no
significant improvement regarding mean points. The students (n=24) scored 3.9
(out of 10 maximum points) in the pretest and 4.7 in the posttest (p=0.073). The
analysis of the test revealed that after instruction, the students still had problems
discerning what doubling means. In one of the test questions the students were
asked to draw double or half of a particular number of objects, having chosen
the original number themselves. This was as difficult for them after instruction
as before instruction. It seemed that the students had not yet learned to separate
the original number from the concept of doubling or understood that doubling
and halving was not related to a specific number but involved the same general
processes regardless of what number used. To allow the students to separate the
rules for doubling and halving from the object being processed, the group of
teachers and researchers decided that simultaneity was to be used. By
contrasting halving and doubling, the aim was to enable the student to separate
the original amount from half the amount as well as double the amount. The aim
of looking at halving and doubling with different numbers was to help the
students to generalize and discern the general idea of halving and doubling
instead of e.g., adding one to the original amount in all cases. The review of the
video recording of the lesson during the meeting revealed another way of
understanding doubling that the group had not previously encountered, in
which doubling an amount was taken to mean adding two, instead of taking the
original amount once and once again. The information ―take the original amount
twice‖ was understood as +2. The contrast between the original amount and the
doubled amount had thus not been explicit enough and therefore the difficulty
remained.
Excerpt 2, second planning meeting (R1 = Researcher 1):
R1: We said it had to do with ―one more time‖, adding or subtracting

one, but here they seem to come to the conclusion to add two because
they understand ―take the same amount twice‖ to mean ―add two‖. You
responded to that well, but we will take this to the next lesson because
we didn‘t know it before.

63
The review during the meeting of the video recording of the first lesson clearly
showed that the teacher had followed the agreed plan based on the micro-
analysis, that is, using the expression ―the same thing one more time‖ rather
than ―the same thing twice‖:
Excerpt 3, first lesson (T1 = Teacher 1):
T1: Now, you are to add twice as many as six. I put six pieces here. I have
now added six one time. If we want to have double the amount, we must
have this six and then I have to add another six pieces.
One conclusion after this discussion was that it was very important to be even
more explicit and use the phrase ―the same amount as the first and then the
same amount one more time‖ instead of ―the same amount twice‖.
Another discussion concerned how to design the task to help the children
distinguish between the original amount and the new amount. While watching
the video recording, the teacher who had taught the first lesson reflected on
whether one reason for the students‘ confusion might be that the students did
not have the initial number of objects in front of them while working with their
tasks.
Excerpt 4, second planning meeting (T1 = Teacher 1, R1 = Researcher 1):
T1: So, the question is whether it is best to put the objects away or
whether it would have been better to let them have the original number
of objects in front of them and let them do it again, so to speak … do it
next to … so that both were there to make it possible for them to compare
…
R1: Maybe it would be even clearer to have a borderline in between …

so: ―This is what we had from the beginning, and now we are going to
put down double that amount, so this is what should be there now‖ …
We would then not get the problem of their just adding the same amount
… to be clearer that they should not just list … but to understand
doubling, you must understand that you have a new amount that is
twice as large as the other.
Here, the teacher who had given the first lesson wondered whether it might
have been better for the students‘ learning if they had had the original amount
that they were asked to double or halve in front of them so that they would be
able to contrast this with the new amount. This discussion concerned how the
children could be helped to separate the original amount from the solution,
avoiding just adding the same amount as the initial and adding this to the
‗doubled‘ amount (double of four experienced as plus four). In the end, the
group agreed upon increasing the contrast between the amounts, and the plan
for the second lesson was therefore to work with the original number of objects
and its double or half simultaneously but separated from each other. To make

64
this relation clear, it was decided that the original amount should be placed on
one side of a border and the doubled or halved amount should be placed on the
other. In this way, the original number of objects and the halved or doubled
amounts could be contrasted and discernible simultaneously. This should be
done by both the teacher and the students while they solved tasks during the
lesson, and the initial number of objects should be explicitly contrasted with the
new amount, in accordance with the assumptions of variation theory.
5.1.3 Third planning meeting

After the second lesson, during the third planning meeting, the analysis of the
pretest and posttest indicated that the second group of students had understood
the content better than the first group. The analysis of the pretest and posttest for
this group of students (n=13) showed significant differences between pretest and
posttest. Mean scores increased from 6.5 to 8.2 (p=0.015), thus indicating
increased learning, in contrast to the first group of students, where, as
mentioned above, no significant differences were found.
The increased mean scores indicated that the design chosen for this lesson had
been successful. However, the tasks in the test that allowed the original number
of objects to be chosen by the students themselves were still problematic in this
second group of students. This third meeting therefore included discussions
about various ways to improve the students‘ abilities to solve these tasks.
The teacher who taught the second lesson (T2), the teacher who would teach the
third lesson (T3), and one of the researchers (R1) discussed the issue of
simultaneously contrasting different amounts when working with the concepts
of doubling and halving in class.
Excerpt 5, third planning meeting (T2 = Teacher 2, T3 = Teacher 3, R1 =

Researcher 1):
R1: We talked last time about having the original amount, and then half
and twice that amount. This change, I think, would be interesting
because you would have the example of both half and twice the same
original amount. … You put the original amount there at the same time
as you tell them to put down half and double that amount.
T3: However, why have you put the ruler along there? (Points at R1‘s
paper, on which a ruler has been laid down to divide the paper into two
areas.)
T2: To make them see the point with … the original amount.
R1: It would have been good … if you had … It may be a paper or

something like that which you just copy … and then they put down the
same original amount as you as the teacher have, in the middle. … So, if
you make copies of a handout you can mark HALF and DOUBLE …
there … if you want to emphasize that.

65
T2: Then, you go on with the simultaneity the whole time … in

everything you do, so you don‘t … it is quicker too.
During this third meeting, the group decided that simultaneity, not only
regarding the initial amount and half or double but all three, should be used in
the subsequent lesson for both the original numbers and their doubles and
halves. The students would work with double and half of the original amount at
the same time, meaning that even more simultaneity would be involved in the
teacher‘s instruction. The original amount, half the amount, and double the
amount would be contrasted with each other simultaneously. Another
important conclusion was that the original amount should remain unchanged in
the middle of the children‘s worksheet, and the teacher should say ―the same
thing again and once more‖ to explain the doubled amount. This would be
emphasized by the use of one more borderline than in the second lesson—that
is, the paper used for the exercise would be divided into three fields, instead of
two, with the initial amount placed in the middle, and the doubled and halved
amounts on either side, separated by lines.
The pretest and posttest associated with the third and last research lesson did
not show improvements over the second lesson. The mean score in this group of
students (n=14) was 6.1 in the pretest and 7.4 in the posttest, a difference that
was not significant (p=0.162). This can be compared to the previous research
lesson (lesson 2), after which the mean posttest result had increased significantly
(from 6.5 to 8.2 points). The tasks in which the students themselves chose the
original number to double or halve were still problematic, as were other tasks
dealing with halving and doubling a predetermined amount.
This result illustrates how small changes in instruction make differences in what
the students can learn. It also shows how, even if the teachers in their joint
planning of the lesson had found a way to adjust the instruction to the students‘
knowledge, these plans were not always actually understood and implemented
by the teacher giving the lesson. To understand the result, the teachers and the
researchers reviewed the videotape of the lesson. This showed that the exercise
where the intention had been to use simultaneity to clarify the difference
between the original amount and the doubled and halved amounts, was
performed mostly by the teacher, but not by the students during their work in
the classroom. The change planned for this lesson, to use simultaneous contrast
between all the amounts to make the difference even more explicit, was thus
only partly carried out by the teacher in charge of the third lesson. The formative
assessment used during the lesson was therefore not informed by the theoretical
assumptions discussed during the planning meeting. One example was the
decreased number of examples of amounts used during the teacher-led
instruction, which only included one number (6). Another issue was when the
teacher, demonstrating the examples, twice placed the objects in the wrong area
on the overhead projector (placing the doubled amount in the area for half). She
corrected the error when it was pointed out by a student, but did not explain to
the class what had been wrong or why she changed the placement of the objects.

66
This oversight might give students the impression that this placement is
randomly chosen and not a deliberate content-based choice.
In conclusion, the third lesson could not verify the group‘s hypothesis that
increased contrast between the different amounts would help students to discern
the concepts of double and half, since simultaneity was not used as was agreed
upon. The theory was thus not used as planned and the students‘ learning
outcomes did not improve. However, the teachers became even more aware of
how small changes might have a great impact on students‘ performance and in
what way theoretical assumptions can be handled in the classroom.
5.2 Theme B: The undivided learning space

One of the assumptions of variation theory is that the learner and what is going
to be learnt cannot be separated from each other. This means that the way the
learner experiences the object of learning is unique and this has to be taken into
consideration in a teaching situation. Finding powerful ways to develop student
learning requires both deep knowledge about the content being taught and deep
insight in the students‘ pre-knowledge. Otherwise it becomes difficult to design
powerful learning situations. It is clear from the analysis of the planning
meetings that the teachers‘ insights into the students‘ understanding of double
and half increased during the study, both as a result of the repeated analyses of
the pretests and posttests, and as a result of the discussions based on the video
recordings of the lessons. One example concerns the teachers‘ opinions about the
children‘s abilities to understand the concept of ―half‖. The teacher who
performed the first research lesson (T1) declared, during the first planning
meeting, that the problem children had with understanding doubling and
halving was not the concept of half, but the use of the expression ―half as many‖.
Excerpt 6, first planning meeting (T1 = Teacher 1):
T1: The difficulty with ―half‖ is not really the concept of ―half‖. As long
as you stick to saying ―half‖, then you can do it. Because these things,
they can divide them. What makes it complicated, as we have understood
it, is when you say half as many. For that, of course, makes it difficult
because there should be fewer and yet you say many.
Analysis of the pretest and posttest results associated with the first lesson
showed that before the lesson, more students gave a correct answer to the
questions asking for ―twice as much‖ than to the questions asking for ―half as
many‖. After the lesson, more students were better able to answer questions
about halving, but still not as well as they could answer questions about
doubling. Five items in the pretest and posttest concerned doubling, and five
concerned halving. The students‘ mean scores on the pretest were 0.45 for the
items on doubling and 0.33 for those on halving, while the corresponding scores
on the posttest were 0.51 and 0.44. However, despite this data, at the second
planning meeting, the teacher quoted above still believed that the concept of
―half‖ was easier for students.

67
Excerpt 7 (T1 = Teacher 1, T2 = Teacher 2, T4=Teacher 4, R1 = Researcher

1):
T1: However, it feels like half is so much easier.
R1: Yes, but we thought last time that half was harder … half as many.
T2: However, I think you learn to take ―half‖ before you learn how to
―double‖, that is … to share with a sibling or with a friend … they have
known this since before …
R1: However, they‘re not better at ―half‖ than ―double‖.
T2: Aren‘t they?
R1: They do worse at half in all the items.
T4: It‘s the b-alternative, which …
R1: Yes, exactly, and the b-alternative is the most difficult straight
through, except for the question with the squares.
T2: Halving is more difficult than doubling.
T1: The question is whether it would have been easier if they had had
this stuff [in front of them], then I think certainly they can take half of it,
but only if they can actually see it …
R1: Yes, but they cannot either. They have the stuff there, but still …
T1: However, don‘t they get it more right when they have to split it in
half?
This example shows that eliciting the students‘ understanding through fine-
grained analysis of the scores on diagnostic test questions gradually changed the
teachers‘ view of what the students found hard to understand. Before the first
test was conducted (during the first research lesson), the teachers assumed that
halving was easier for the students to understand than doubling, but when the
tests revealed that this was not the case, their opinion was slowly altered. The
excerpts indicate, however, that their initial opinion was quite persistent and
difficult to change; they were not easily convinced of the contrary even if the test
results showed this. The design of instruction thus risks focusing on aspects that
are not problematic for the students and neglecting the aspects that are critical.
The test was designed on the basis of variation theory, and by contrasting
halving with doubling, it was possible to compare the students‘ knowledge of
these concepts. The results of the tests informed the teacher‘s formative
assessment, as long as the teachers accepted what the tests really said about the

68
students‘ knowledge and let go of their own beliefs. As exemplified in the

exchange quoted above, it was however difficult for the teachers to take the
students‘ perspectives when these perspectives contradicted the teachers‘ own
beliefs. Previous research results have indicated that formative assessment
might not always have an impact on students‘ learning outcomes, and if
teachers‘ instruction depends on their own beliefs instead of students‘
knowledge, it is obvious that the impact of formative assessment is weaker than
if they take the students‘ knowledge into consideration.
5.3 Theme C: Theoretically guided formative feedback

In this theme, the teachers‘ formative feedback in the classroom is in focus. The
analysis of the teachers‘ feedback about their respective lessons focuses on how
they used variation theory to understand and respond to the students‘
understanding of the content.
In spite of the teacher‘s obvious engagement with teaching in the first lesson and
in the evaluation of this lesson, the subsequent analysis of the video recording
showed that the lesson rarely included direct feedback to the students. The
teacher listened to the students‘ answers, but seldom commented on whether the
answers were correct.
Excerpt 8, first lesson (T1 = Teacher 1, S1 = Student):
S1: We had six, then we thought that half of them was all of them.
T1: You thought half was all and so you took them all away. So, we have
some different answers. Let us see what the others thought. What did
you come up with, [student name]? (Turns to another group.)
After listening to the first student, the teacher turned directly to the next group
without revealing the correct answer, and this answer was not revealed until
summing up the instruction at the end of the lesson. This lack of feedback was
not highlighted or discussed when the group evaluated the first lesson during
the second planning meeting, so we do not know whether it was common for
this teacher or not; it is possible that the teacher conducted the lesson differently
from how she usually would due to her awareness of being filmed.
Nevertheless, it is evident that the students were left in doubt several times.
According to the videotapes, the teacher who conducted the second lesson gave
the students more detailed feedback regarding critical aspects than did the first
teacher. It is probable that the discussion during the evaluation of the first lesson
had an impact on this teacher‘s awareness of which aspects of the concepts of
doubling and halving the children might not have discerned, which may have
led her to challenge the students‘ answers with a more developed formative
assessment than the first teacher was able to. In the example below, the teacher
worked formatively with a student who was asked to show half of eight. The
teacher clearly contrasted the doubled and halved amounts, comparing them

69
with each other to help the students discern the difference between double and
half, taking advantage of the experiences from the previous lesson.
Excerpt 9, second lesson (T2 = Teacher 2, S = student):
T2: Was he asked to put down half or double the amount?
S: Half!
T2: Half. Is there the half of the amount on that side here? (Points to the
tray and one of the piles with four chestnuts.)
(The student nods.)
T2: How many are here? (Points to the tray and one of the piles with four
chestnuts.)
S: Four!
T2: And how many are here? (Points to the tray and the other pile with
four chestnuts.)
S: Four!
T2: However, that‘s just as much! Where is half the amount?
Here, the teacher made a serious effort to understand which aspects the student
had experienced and which the student had not yet discerned. She evidently
used knowledge gained from the analysis of the first lesson about how the
aspects might be discerned and how to use patterns of variation to make aspects
discernible. She noticed that the student had taken four chestnuts from the
original amount, resulting in two piles with the same amount of nuts, i.e. four
chestnuts in each pile. As she wants the student to discern both eight and half of
eight (four), she puts a question to make it possible for the student to discern the
difference. Because the student had taken four items from the original pile of
eight items when halving, the distinction between four and eight was not visible
anymore; the child ended up with two piles with equal amounts instead. Based
on such mistakes, to draw attention to the difference between the original
amount and half this amount, the teacher repeatedly used simultaneous contrast
in her instruction. There were, however, also examples during the lesson in
which the teacher did not give a clear indication of whether a student‘s answer
was correct.
During the third research lesson, the students were given some direct feedback
during their performance of the tasks, but not after, and often this feedback
neither confirmed nor rejected the students‘ answers. The lack of
correspondence between the intended design of the third lesson and the
teacher‘s actual performance of it could be one reason for this lack of

70
constructive feedback. The communication between teacher and students was

mainly based on giving instruction at a procedural level:
Excerpt 10, third lesson (T3 = Teacher 3):
T3: You also think that 4+4 is 8. Have a look here … I have 8 pieces from
the beginning. That is my quantity, my amount that I am not allowed to
touch. If I am going to put down the double amount, I have to put down
as many as I have here (points at the 8 pieces in the middle area) + the
same amount. This means I have to put down 8+8, and this means?
The feedback in the third lesson thus ends with the teachers telling the students
the correct way to solve the problem, a process, instead of taking into
consideration how the students might have understood the problem and what
aspects had been discernible.
6. Discussion and Conclusions

This study explored teachers‘ development of formative assessment strategies
guided by variation theory during a classroom-based professional development
project. The result of the analysis found three themes for how the teachers used
theoretically informed strategies for assessment in the classroom in their
planning, conducting, and evaluating of lessons. These three themes
demonstrate how formative assessment contributed to the development of the
teaching: theme A concerns the critical aspects for learning the specific content;
theme B focuses on how teachers understand their students‘ knowledge about
the content; and theme C brings A and B together by exploring the teachers‘
formative feedback in the classroom. The results indicate that throughout the
study, the teachers became increasingly aware of how even small changes in
teaching could affect the students‘ learning and how to use this knowledge in
further instruction. For example, after finding out that the students tried to find
a definite figure to use when doubling or halving, such as in all cases choosing to
add one, two, or another number to the initial amount instead of seeing the
relationship between the original amount and what to add, the teachers used
different ways of explaining the concepts to the students. Two of the teachers
realized the importance of varying the initial amount, based on the theoretical
assumptions, and varying this aspect made it easier for the students to
understand that the number added changes when the original amount changes.
However, the third teacher was not aware of the importance of this because she
had not accepted the common theoretical base and thus used this knowledge
neither in her instructions nor in her feedback to the students, resulting in no
significant increase in learning. The learning outcomes of the students‘ tests
strengthened the other two teachers‘ analyses, and they became aware that even
small changes in instruction might have an impact on student learning, such as
what examples to use in the teaching (type and number of different objects),
how these are to be used in the teaching and exercises (simultaneously
contrasted or not), or what expressions to use when explaining the tasks (―same
amount twice‖ or not). Eliciting and clarifying students‘ ideas and
understandings at a fine-grained level based on theoretical assumptions thus

71
seems to have helped the teachers to understand the difficulties students may
have in comprehending the object of learning, and to give relevant feedback to
the students.
The teachers‘ formative assessment, which increased over the course of the
learning study, was informed by the results of the students‘ tests and the video
recordings of the lessons, both of which proved to be powerful tools for the
teachers‘ assessment of the students‘ learning and understanding. Thus, there is
evidence within the study that the learning study design encourages teachers to
work formatively and that the formative assessment is strengthened by the
theoretical framework used. This is in line with the argument put forward by
Black and Wiliam (2009, 2012) and Wiliam (2009) that in order to be able to give
relevant feedback to a student, the teacher needs a theory of how students learn
and the ability to apply this theoretical understanding in a specific context. This
is also in line with the work of James and Pedder (2006) and Pedder and James
(2012), who suggest that the concept of ‗research lesson‘ could be a useful
strategy to promote assessment for learning in classrooms.
The evaluation of the last lesson showed that the simultaneity of double, half,
and the original amount had not been presented in the third lesson as intended.
During the planning meeting before this lesson, the teachers developed a new
design, which was then not implemented during instruction. It was also clear
through subsequent analysis that direct feedback to the students differed
between the teachers and the groups; the teacher conducting the second lesson
gave more frequent and more specific feedback than the other teachers by
explicitly using the concept of simultaneous contrast. It is therefore possible that
the improvement in scores after the second lesson was due to the combination of
increased feedback with a lesson design that followed the assumptions of
variation theory with regard to determining which aspects of the content should
be in focus and which should be kept in the background. On the basis of this, we
can conclude that the outcome of the learning study as a whole might have been
even better had the joint evaluation of the lessons focused more strongly on the
teachers‘ feedback (or lack thereof) to the students, both individually and to the
class as a whole, along with discussing the effectiveness of the lesson design, as
revealed by the students‘ test results and comments and behavior during the
lessons.
7. Limitations of the Study

The small scale of the study—four meetings with four teachers over a limited
period of time and with a limited number of students—means that it is not
possible to make any generalizations from the results. However, these
limitations made it possible to perform an in-depth study, giving some insights
into how a theory of learning can guide teachers in their planning,
implementation and evaluation of a content-specific topic. Since the study is
class-room based we consider it however to have rather high ecological validity
(Brewer, 2000), especially regarding the use of variation theory in designing,
implementing and evaluating teaching, and the impact this is likely to have on

72
the opportunities for children in similar situations and settings to develop

knowledge about doubling and halving.
8. Acknowledgments
We would like to thank the participating teachers, who generously shared their
time and teaching, the Learning Design research team (LeaD) at Kristianstad
University, for their encouragement and support; and the Department of
Pedagogical, Curricular and Professional Studies, University of Gothenburg,
Sweden, for partly financing this study. We are also grateful for valuable
comments provided by the reviewers and for the thorough language review
carried out by Dr. Catherine MacHale, which increased the clarity of the paper.
9. References
Adamson, B., & Walker, E. (2011). Messy collaboration. Learning from a learning study.
Teaching and Teacher Education, 27, 29-36.
Bennet, R. E. (2011). Formative assessment: A critical review. Assessment in Education:
Principles, Policy & Practice, 18: 1, 5 – 25.
Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in
education: Principles, Policy and Practice, 5(1), 7–73.
Black, P., & Wiliam, D. (2009). Developing the theory of formative assessment.
Educational Assessment, Evaluation and Accountability, 21(1), 5–31.
Black, P., & Wiliam, D. (2012). Developing a theory of formative assessment. In J.
Gardner (Ed.), Assessment and learning (pp. 206–229). (2nd ed.) London: Sage
Publications.
Boyatzis, R.E. (1998). Transforming qualitative information: Thematic analysis and code
development. London: SAGE.
Brewer, M. (2000). Research Design and Issues of Validity. In H.T. Reis & C.M. Judd
(Eds.), Handbook of research methods in social and personality psychology (pp. 3-16).
Cambridge, U.K.: University Press.
Elliott, J., & Yu, C. (2008) ―Learning Studies as an Educational Change Strategy in Hong
Kong: An independent evaluation of the ‗Variation for the Improvement of
Teaching and Learning‘(VITAL) Project‖, funded by the Dept for Education and
Manpower, Hong Kong. Published by the Centre for Learning Studies, HK Inst
Educ. http://www.ied.edu.hk/cls/resources.htm
Elliott, J., & Yu, C . (2013) Learning Studies in Hong Kong Schools: A Summary
Evaluation Report on ‗The Variation for the Improvement of Teaching and
Learning‘ (VITAL) Project, Education and Didactique, 7(2), 147-163.
Gustavsson, L. (2008). Att bli bättre lärare. Hur undervisningsinnehållets behandling blir till
samtalsämne lärare emellan. [To become better teachers. How the handling of the
lesson content becomes a topic of conversations between teachers.] Umeå:
University of Umeå.
Hermerén, G. (2011). Good research practice. Stockholm: The Swedish Research Council.
Holmqvist, M. (2002). Lärandets pedagogik. Forskningsansökan till Vetenskapsrådet.
[The Pedagogy of Learning. Research application to the Swedish Research
Council.] Dnr 721-2002-3386
Holmqvist, M., Gustavsson, L., & A. (2007). Generative learning: learning beyond the
learning situation. Educational Action Research, 15(2), 181-208.
Holmqvist, M., Gustavsson, L. & Wernberg, A. (2008) Variation Theory – An Organizing
Principle to Guide Design Research in Education. In Kelly, A.E., Lesh, R., &.
Baek J. (eds) Handbook of design research methods in education, p 111-130. New
York: Routledge.

73
Holmqvist, M. (2011). Teachers‘ learning in a learning study. Instructional Science, 39(4),

497-511.
Holmqvist Olander, M., & Nyberg, E. (2014). Learning study guided by variation theory:
Exemplified by children learning to halve and double whole numbers. Journal of
Research in Childhood Education . 28(2), 238-260.
James, M., & Pedder, D. (2006). Professional learning as a condition for assessment for
learning. In J. Gardner (Ed.), Assessment and learning (pp. 27–43). London: Sage
Publications.
Kullberg, A. (2010). What is taught and what is learned: Professional insights gained and
shared by teachers of mathematics. Diss. Göteborg : Göteborgs universitet, 2010.
Göteborg. http://gupea.ub.gu.se/hdl.handle/2077/22180
Lo, Mun Ling (2012). Variation theory and the improvement of teaching and learning
Göteborg: Acta universitatis Gothoburgensis.
http://gupea.ub.gu.se/handle/2077/29645¨
Lewis, C. (2015). What is improvement science? Do we need it in education? Educational
Researcher, 44(1), 54-61.
Lewis, C., Perry, R., & Murata, A. (2006). How should research contribute to
instructional improvement? The case of lesson study. Educational Researcher,
31(3), 3-14.
Lo, M.L., Pong, W.Y., & Chik, P.M. (Eds.) (2005). For each and everyone: Catering for
individual differences through learning study. Hong Kong: Hong Kong University
Press.
Lo, M. L., & Marton, F. (2012). Towards a science of the art of teaching: Using variation
theory as a guiding principle of pedagogical design. International Journal for
Lesson and Learning Studies, 1, 7–22. Doi: 10.1108/20468251211179678
Marton, F. (1981). Phenomenography - describing conceptions of the world around us.
Instructional Science, 10, 177-200.
Marton, F. (2003). Learning Study - pedagogisk utveckling direkt i klassrummet.
[Learning Study—pedagogical development directly in the classroom]. In
Carlgren, I. (Ed.). Forskning av denna världen: praxisnära forskning inom
utbildningsvetenskap [Research of this world-practice-oriented research in
educational sciences] (pp. 41–46). Stockholm: Vetenskapsrådet [The Swedish
Research Council].
Marton, F. (2014). Necessary conditions of learning. New York, N Y: Routledge
Marton, F., & Tsui, A. (2004). Classroom discourse and the space of learning. Mahwah, N.J.:
Lawrence Erlbaum Associates.
Marton, F., & Booth, S. (1997). Learning and awareness. Mahwah, N.J.: Lawrence Erlbaum
Associates.
Olteanu, C., & Lennerstad, H. (2011). Lesson study och Learning study samt IKT i
matematikundervisningen: en utvärdering av matematiksatsningen [Lesson Study and
Learning Study and ICT in mathematics education: An evaluation of the
mathematics initiative]. Stockholm: Fritzes.
Pang, M. F., & Marton, F. (2013). Interaction between the learners‘ initial grasp of the
object of learning and the learning resource afforded. Instructional Science, 41(6),
pp 1065-1082.
Pedder, D., & James, M. (2012) Professional learning as a condition for assessment for
learning‘. In J. Gardner (Ed), Assessment and Learning (pp. 33-48), (2nd ed.).
London: Sage Publications.
Phelan, J., Choi, K., Vendlinski, T., Baker, E., & Herman, J. (2011). Differential
Improvement in Student Understanding of Mathematical Principles Following
Formative Assessment Intervention. The Journal of Educational Research, Vol 104,
pp. 330-339.

74
Stigler, J.W., & Hiebert, J. (1999). The teaching gap: best ideas from the world's teachers for
improving education in the classroom. New York: Free Press.
Stigler, J. W., & Hiebert, J. (2009). The teaching gap: Best ideas from the world’s teachers for
improving education in the classroom. New York: Free Press.
Swedish National Agency for Education (2011). Curriculum for the compulsory school
system, the pre-school class and the leisure-time centre 2011. Stockholm: Swedish
National Agency for Education [Skolverket].
Wernberg, A. (2009). Lärandets objekt: Vad elever förväntas lära sig, vad görs möjligt för dem
att lära och vad de faktiskt lär sig under lektionerna [The Object of Learning: What
students are expected to learn, what is possible for them to learn and what they
actually learn in class]. Umeå: University of Umeå.
Wiliam, D. (2006). Formative Assessment: Getting the Focus Right. Educational
Assessment, 11:3-4, 283-289.
Wiliam, D. (2009). An integrative summary of the research literature and implications for
a new theory of formative assessment. In H. Andrade & G. J. Cizek (Eds.),
Handbook of Formative Assessment (p.18-40). New York: Routledge.
Wiliam, D. (2011). What is assessment for learning? Studies in Educational Evaluation 37,
3–14.

75
Vol. 11, No. 1, pp. 75 – 98, April 2015
Designing and using interactive e-books in

Vietnam
Ngoc-Giang Nguyen
Doctor’s degree student of The Vietnam Institute of Educational Sciences
101 Tran Hung Dao, Hanoi, Vietnam
Abstract. E-books are now our imperative need. Because of their strong
points, e-books help the process of learning a lot. E-books allow us to
teach an infinite number of students without depending on location and
studying time. In addition, E-books improve independent thoughts,
creative thoughts and informatics thoughts for students.
Keywords: e-books, e-learning, technology.
1. Introduction
In recent years, the research, design, and produce tendencies of ebooks have
been developing a lot. An e-book called Class-book produced by Vietnamese
Education Publishing House has justified the above thing. Students learning
with the e-book can avoid carrying books cumbersomely. If students buy this e-
book, they can use it in all 12 years of learning. Because e-books can interact in
two dimensions with customization ability, multimedia content, they help
students learn enthusiastically. In addition, e-books have other functions that
ordinary books do not have, such as feedback, relating documents, adapting to
the fast changing world, etc. Therefore, e-books were and will be the first
selection in innovating methods, equipment and learning contents.
Although e-books are important, being researched and developed strongly,
some of us still do not know what e-books are, how many kinds of e-books there
are, what characteristics e-books have, what the differences between the learning
with e-books and the learning with e-learning are. Thus, we will refer to these
things in the article. Moreover, we have also built an interactive geometry e-
book, which has many strong points of technology, and is better than current e-
books. We have also designed tools that allow us to divide learners, e-
communication, etc. in order to help the process of learning better.
2. Research content
2.1. The conception on e-books
According to Wikipedia, an electronic book (digital book, e-edition) is a book-
length publication in digital form, consisting of text, images, or both, readable on
computers or other electronic devices. Although sometimes defined as “an
electronic version of a printed book”, many e-books exist without any printed

76
equivalent. Comercially produced and sold e-books are usually intended to be

read on dedicated e-book readers, however, almost any sophisticated electronic
device that features a controllable viewing screen, including computers, tablet
computers, and smartphones can also be used to read e-books. (Wikipedia)
The electronic counterpart of a printed book, which can be viewed on a destop
computer, laptop, smartphone or e-book reader. When traveling, a large number
of e-books can be stored in portable units, dramatically eliminating weight and
volume compared to paper. Electronic bookmarks make refering easier, and e-
book readers may allow the user to annotate pages. (Encyclopedia)
An e-book is the presentation of electronic files on a digital screen. Although the
term e-book has implied its information expressed in a book form, it also shows
that the book content is expressed on some device. These are two basic
characteristics to recognize the difference between an e-book and a traditional
book. Static writing and static image are typical of a book in general, including
traditional books and e-books. However, an e-book also shows animated images
and sound that a traditional book does not have. The files of an e-book can be
provided as sound recorded units (disc), or can be downloaded from an
information store by e-readers (including websites of the Internet).
(Vst.vista.gov.vn)
In our opinion, an e-book is a book software product having an interface and

functions that act on the Internet or without the Internet. An e-book can be
viewed on personal computers, e-readers, tablet computers, or smart-phones.
An e-book can integrate many kinds of advanced and modern technology of
informatics and media in order to serve information transmitting, learning, and
studying best.
2.2. The characteristics of e-books

According to us, there are four characteristics of e-books now, as follows:
First, e-books are book software products, so they have mechanical structures as
normal books. They have e-book pages. We can access their pages randomly as
normal book pages.
Second, e-books are book software products acting on the Internet or without the
Internet. That is, e-books act on both on-line and off-line well (acting on off-line
will restrict some functions of e-books). Indeed, when we refer to this thing, we
want to refer two ideas. The first is that e-books act on the Internet. From this,
we can design e-books having outstanding functions due to the Internet. That is,
e-books can educate an infinite number of students, and educate without
depending on location and learning time. Students can learn with many teachers
at the same time and at various locations. All students who are not of the same
skin color, country, nation, etc. can still learn. E-books are equipped artificially
intelligent which can give predictions and feedback of tests, self-study exercises,

77
etc. immediately. E-books allow us to insert animated images, videos, etc. in

order to make lessons deep and understandable. In addition, e-books allow us to
organize learning acts that improve the activeness of students by using
interactive tools and functions of e-books. The second is that e-books act on the
Internet. From this, we can design e-books as normal packaged software. If we
design like this, there are not the typical functions acting on the Internet such as
the tele-learning function, the function of live images and sound transmitting,
etc. This idea is the basic characteristic when we distinguish learning with the
help of e-learning from learning with the help of e-book. E-learning without the
Internet does not act but e-books without the Internet still act normally.
Third, e-books always accompany with e-readers. Thus, besides transmittable
content, e-books must focus on e-reader devices. These two factors are two close
faces of e-books. The e-reader devices are diversified. E-books can be viewed on
personal computers (PCs, laptops), e-readers (Kindle, Nook, Sony, Reader, etc.),
tablet computers (Multiform: iPad, Android Tablets (Galaxy Tab, Kindle Fire,
etc.), Surface; Specialized: Kno, Class-book) or smart-phones. Some learning
forms (ex. e-learning) do not care about information transmittable devices. The
devices of e-learning are often personal computers.
Fourth, e-books have the abilities to integrate the forms of advanced and modern
technologies and the media in order to serve the best information transmitting.
When we refer to this thing, we focus on the continuous development of e-
books. If the technologies of e-books are advanced, so are e-books. Some
technologies of e-books now are forums, e-dictionaries, text editing, hyperlink,
and hypertext functions, etc. E-books have e-draft notebooks to help us draw
figures, find out the solution to problems, verify, prove, express the
visualization, etc. move and act on animated figures. E-books allow us to mark
tests automatically and objectively, allow learners to do exams in a limited time,
and decentralize marking exams to teacher. E-books give specific self-learning
tasks to each student and allow us to divide students with an infinite number.
After students do divided problems, e-books will give the results, a tree chart as
well as a table of students’ answers intermediately with just one click. E-books
allow us to e-meet, e-teach, e-exchange, to share texts, sound, images, videos,
PowerPoint, MP3, and MP4 files. E-books can also save the activities of users in
e-documents. E-books allow users to improve the best activeness of learning and
studying to develop independent thoughts and motivate users to learn
knowledge for themselves. They also give users an impulse to inner motivation
and creative desires. Some technologies which now we do not have yet and
researchers are studying are artificial technologies. We need to equip e-books
with an intelligent brain to allow them to self-learn, self-develop, and from that,
they can solve problems given by other people or by themselves and solve new
ones. In addition, e-books do not yet have remote technologies such as voice

78
technology, sound technology, etc. In the future, if these technologies are

performed, e-books will develop greatly.
2.3. The classification of e-books

According to us, there are four kinds of e-books now, as follows:
Static e-books are digital text files as PDF files. PDF files are e-books with simple
formats allowing readers to have access easily. The weak point of this kind is
that it has no sound and graphics, and its structures are simple. It does not have
many informatics functions except an only informatics transmitting function. Its
strong point is that it has small capicity, so we can email it easily.
Dynamic e-books without changing content are softwares that are the same as other
learning software using programming language, such as Flash programming
language. This kind is only good for the English subject and multiple choice
software of other subjects because their interfaces are nice and their interactions
are high. However, their weak points are that they do not allow users to change
the contents and the contents given are only the contents in a particular area.
These e-books require users to have high programming levels and nice and
varied graphic design abilities.
Dynamic e-books changing content are the same as e-books that are designed with
PowerPoint software. E-books of this kind can integrate sound, videos and
images and their strongest point is that they allow users to add, correct, change
the contents of e-books as well as allow users to copy data. The weak point is
that their images and interface are not as nice as dynamic e-books without changing
content.
Interactive e-books are book software acting on the Internet or without the Internet
that integrate many kinds of advanced and modern technology of informatics
and media. The e-books are designed in order to serve information transmitting,
learning and studying well as well as remedy the weak points of traditional
information transmitting, learning and studying. The e-books show all the
strong points of the above e-books such as nice and eye-catching interfaces and
e-information sending with real time. In addition, they have other strong points
that the above e-books do not have. They have two structures which are the full
structure and the empty structure. The full structure allows teachers to provide
information for students with a pedagogic intension. The empty structure allows
users to build materials. Students can access the content and the process of
learning. Interactive e-books have clear strong points as mentioned above. As a
result, this kind of e-books improves the best activeness of users. The weak point
is that it has high designed cost.

79
Of all the above e-books, interactive e-books now have the best ability to support
learning in general and learning mathematics in particular at secondary schools
according to the orientation of innovating learning methods.
2.4. The comparison between e-books and traditional books
According to us, the difference between e-books and traditional books is as

follow:
Table 1. The comparison between e-books and traditional books
Traditional books E-books
Strong points Strong points
- They are ready to use. - They are small and light according to
the exploitable equipment, without
- They do not need sub-equipment. depending on the quantity of e-books.
- Their cost is rather cheap. - They can adapt to a fast changing

world.
Weak points
- The new contents can be updated
- They are cumbersome and heavy
easily and timely.
when we carry a lot of books.
- The contents are multi-media.
- The contents are easily outdated in
the context of a fast changing world, - They connect related documents: text
especially in science and technology books, reference books, dictionary
areas. books, and notebooks.
- The contents are restricted due to the - They interact with two dimensions
fees of their print. with customization ability for students.
- The contents are not easy to reuse - They allow users to feedback.
among related books.
- They allow users to approach other
- They do not contain multi-media documents intermediately.
functions (video, sound, or
interaction.) Weak points
- Each document is single and does not - The cost is expensive (due to the need

80
have a connection. of sub-equipment.)
- It’s difficult to read related - They need to use electricity.

documents.
- The longevity depends on the
durability of user’s equipment.
2.5. The comparison between e-learning and e-book
According to us, the difference between e-learning and e-book is as follow:
Table 2. The comparison between e-learning and e-book
Factors e-learning e-book
Resources and learning - The materials mainly - The materials mainly

location depend on documents depend on documents
and graphic techniques. and graphic techniques.
- The contents can - The contents can

change according to the change or are packaged
decentralization of the without changing.
administrator.
- It happens in an
- It happens in an imaginary class or
imaginary class or an happens every time and
imaginary laboratory. everywhere.
Showed interface - It happens on the - It happens on the

Internet. Internet or without the
Internet.
Showed equipment - It does not focus on - It focuses on showed

showed equipment. equipment. Showed
Showed equipment of e- equipment of e-books is
learning usually is personal computers,
personal computers. tablets, e-readers or
smart-phones.
The interaction between - It is via e-mail, chat, e- - It is via e-mail, chat, e-

81
teacher and students forum, or e-class. forum, or e-class.
- The information is - The information is

communicated passively, communicated
and sometimes not intermediately and
globally globally according to the
equipment.
The interaction between - Students interact - Students interact

students – students directly via e-mail, chat, directly via e-mail, chat,
or e-forum. or e-forum.
- Space and time are - Space and time are not

limited. limited.
- The interaction depends - The interaction is 1-1.

on the consciousness of The effectiveness
each member of the depends on each
group. attended member.
To assign a task and a - In the classroom - Everywhere

check
- The time is - At every time and the
predetermined and time is not limited.
limited.
- The check depends on
- The check depends on each individual test.
criteria.
- The tests have a limited
- The tests have a fixed number of questions,
number of questions. depending on the
competence of students
- It is based mostly on
text. - It is based on audio and
animation.
- We can just assign our
tasks if we have the - We can still assign our
Internet. tasks without connecting
to the Internet.
To set an exam and to - Setting an exam is - Setting an exam is

assess students based on theory and based on the reality and

82
practical problems situated problems.
- Giving marks is - Giving marks is based

according to teacher’s on the competence of
marking scheme. each student.
- Tests are based on - Tests are not on printed

printed documents. documents, so not costly.
- We can just set an exam - We can still set an exam

and assess students if we and assess students
have access to the without having access to
Internet. the Internet.
Feedback information - It is almost not - They are both

synchronous. The synchronous and not
feedback happens after synchronous. The
some time. feedback is immediate
and better.
- The feedback happens
mostly according to - The feedback happens
group. according to individual,
group, or divided
learners.
2.6. Building the interactive e-book at the address: http://e-

eduvietnam.edu.vn
The structure of the e-book includes the following parts:
1. E-book title: The e-book title is The interactive geometry e-book.
The bar menu includes following items:
2. E-forum: E-forum is the Adobe Connect Enterprise software embedded in the
e-book. E-forum allows us to organize conferences or e-conferences with a small
or large size or we can apply E-forum to e-teach, etc.
After clicking the E-forum button, the interface of the e-forum is as follows:

83
Share : this button is in the middle of the screen,

allowing us to control the signal of input images.
This interface is of the

conference room; however,
Attendees: this item allows us to see and control the list of the interface is not standard
the television bridges connecting to the conference room. In because we still see this
this conference room system, we have three major object
interface in the Tab of the
groups:
- Host: the chairman and the major controller having the
web browser. If we want to
extreme right in the room. display full screen, we click
- Presenter: the lecturer, the person who is allowed and on the item “Help” at the
decentralized by the chairman to use the functions of the right corner, whose mark is
conference room, for example, using webcam, sharing yellow. Next, we choose the
documents, etc. last item „Install Adobe
- Participants: guests, the people attending a meeting, who Connect Add-in‟. After that,
have no power, but listen, or watch. They can use other we choose „Yes‟ to continue
functions when they are decentralized. to install Add-in. Note that
we should use Mozilla
Firefox, IE web browser
because the installation of
Add-in is simple and causes
Chat: This item less mistakes than Chrome
allows us to chat or Co Rom++….
(to talk) and
control.
Now, let’s study the most basic parts and functions of the e-forum:
a. Images: this item allows us to Switch on or Switch off and control the signals
of Webcam or Camera.
In order to switch on the images we click on the black square Start My Webcam
in the middle of the screen, in the frame Video. After clicking on this square, the
screen displays the announcement: Camera and Micophone Access. We click on
the item Allow and the screen displays the window Preview with camera
images or Webcam. In order to swich on the image signals, we click the button
Start Sharing. After clicking on this button, we will see the image signals. This
time, the official signals of images will be transmitted to all of the television
places.
In order to switch off the image signals or to pause the images, we click on the
Stop My Webcam button at the top right corner of the image frame.
b. Sound: This item allows us to switch on or switch off the microphone or adjust
the sensitiveness of the microphone.
We choose Microphone and Test loudspeaker: this step is forced when we log
in the conference room. This step is performed only one time and we do not
need to do it again. This step allows us to choose the input signal of the
microphone (if we have many input signals) and to test the loudspeaker as well
as the sensitiveness of the microphone. In order to do this, we click on
Meeting/Audio Setup Wizard on the menu bar. After that, we click on Next to
continue the progress of installation.

84
Click on the Play Sound button to test the loudspeaker. If the signal of the
loudspeaker is good, the blue signal runs along as shown above. Click on Next to
install.
After clicking on Next, we see a new table with a list of microphone equipment
available on the computer. We choose each item and test the microphone by
choosing the input signals and then click on Next. To continue, we click on
Record to see whether the microphone works or not. If the microphone is good,
a blue signal runs along. Finally, we click on Next/Finish to finish the process of
sound installation.
On the menu bar, we click on the symbol and then the down arrow
and choose Connect My Audio. After completing the connection, the green
symbol appears like this: . We can speak now. If we want to adjust

better, we click on the down arrow beside the green microphone symbol:
- Mute My Microphone: allows us to pause the

microphone. The microphone is mute when we
click on this button.
- Disconnect My Microphone: is to stop the
microphone.
- Adjust Microphone Volume…: Adjust the
ssensitivity of microphone.
Note: whenever we do not use the microphone to speak, we have to switch it off
by clicking on the button Mute My Microphone or when microphone symbol is
green, we click on the green symbol, which means clicking on Mute My
Microphone. The target of this function is to avoid the phenomena of howls and

85
echoes caused by switching on too many microphones among television bridges

at the same time. At one time, there is only one television place allowed to
switch on its microphone and speak. The others must switch off their
microphones. Next, we adjust the sensitivity of the microphone suitably. We
should not to choose the high sensitivity.
c. resource share : this function allows us to share files such as still images, flash
video files, PowerPoint files, the backgrounds of the computer (desktop) or the
applications that are running on the computer, etc.
If we click on Share My Screen, the system has three specific items as follows:
Desktop (allowing us to share all the applications which are running on the
computer to other television places), Application (Sharing the allowed
applications to suitable television places while the other applications are
invisible), Windows (allowing us to share the applications of the operating
system.)
d. Raise hand: is to ask for the chairman’s permission to speak. We click on the
menu bar, and then we click on the symbol (man icon) or click on the down
arrow and choose “Raise Hand”.
e. Discussion (Chat): is the same as the chat function in Yahoo Messenger.
3. E-dictionary: allows us to look up the meaning of English – Vietnamese

mathematical words.
4. E-check: E-books allow us to test and assess students the same as the
traditional way. First, teacher can test and assess students by observation.
Teacher observes students by the function E-forum. From that, teacher gives the
exact remarks about acts, gestures, and altitudes of students. Second, teacher can
test and assess student by oral. This technique is effective. This technique
assesses the expressed abilities of students such as listening, speaking,
explaining, etc. Teacher can understand and help each student in the process of
knowledge acquirement. Teacher asks students questions and listen to their
answers via e-forum. Teacher and students must have computers and webcam
to make the oral. Third, students can self-check. Students self-assess in order to
develop the responsibility, the self-assessment, the independence, the
confidence, and the creation of themselves. The self-check is done by stocktaking
notes, ranking scales, and the answers. The self-check of students will remedy
the weak points of inferiority. Fourth, teacher can test and assess students by the
performance of students. Teacher allows students to use the functions of e-book
to give solutions, to perform software as well as to give stocktaking notes or
ranking scales on e-book. The assessment is easy and convenient. Fifth, testing

86
and assessment can be done by collecting students’ products. E-books save the
traces of students on them automatically. If we use e-books, the saving is easy.
Teacher uses e-books to set up files of students. Finally, testing and assessment
can be done by using questions and exercises. Questions and exercises can be
used to assess learning, for example, if we define the levels of students, the
compilation needs to satisfy these following requests:
+) Questions and exercises must be suitable for the program, the
standard knowledge of Ministry of Education and Training, and the levels of
students.
+) Questions and exercises must be stated exactly and clearly in order for
students to understand exactly.
+) Besides the questions and exercises oriented to the basic requests, we
need to prepare questions, deeper exercises requiring wider knowledge to
encourage active thoughts.
+) Assessing results is not only giving marks, but also giving comments
on the contents, forms and learning methods as well as giving aid and plans to
help students.
The function e-forum is designed to test and assess using questions and
exercises. The use of e-forum has these strong points as follows:
+) Because e-forum decentralizes giving marks to many teachers, we can
partly avoid subjective factor. E-check allows us to check, assess each student.
We do not need a lot of efforts for checking and assessing writing exams. We
only spend once entering data and use it for the next times.
+) Besides the strong points of objective testing such as independence of
the subjective ideas of teacher, assessing broad knowledge, covering all desired
contents in a short time, and assessing the fast thought abilities of students, e-
books also allow us to self-assess automatically. When students finish the exam,
we will know the result immediately. Using questions and objective exercises to
test students does not require a lot of efforts. We only enter the data once and
use it for the next times.
+) We can combine writing form with oral form to test students easily.
The interface of an objective test is as follows:
The interface of a writing exam is as follows:

87
The check and assessment in general as well as e-check and e-assessment in

particular show that the e-check and e-assessment have more strong points than
the traditional check and assessment such as more objective, more convenient,
independent of the number of students, learning place, learning environment,
and learning time. Moreover, the storage is better. The choice of questions and
answers is also as convenient as the traditional one.
5. E-documents: are PDF files, HTML files as well as video files and others of the
Internet. E-files provide a document resource to discovery learning.
6. Forum: allows us to communicate and discuss strong points, weak points of e-
books as well as gives the remedial solution to weak points.
7. Instruction: instructs in detail the use of e-book.
8. Introduction: introduces the principles and targets of e-books.
9. Connection: connects with the administrator of e-books.
10. Entertainment: includes intellectual games.
Vertical menu includes the following items:
11. E-notebook: E-notebook is the major part of the interactive e-book. Its content
performs two functions. The first is to organize learning acts according to
discovery method. The second is to organize learning acts according to divided
method. In addition, E-notebook has an e-rough notebook (the GeoGebra
software is embedded in the e-notebook) allowing us to draw figures, predict
the result of problem, etc.
a. Discovery learning
Discovery learning includes the following factors:
+) A system of questions about the aware problem and solving problems

through discussion.
+) With the help and instructions of teacher, students give the way of
approach, survey problems and collect data.
+) Students solve problems by individual or group.
+) Finally, students exploit and develop obtained problems, perform

manipulations of science thoughts.
From these factors, we see that discovery learning is divided into three various
kinds:

88
+) Type 1: is called Guided discovery learning. Teacher gives problems

and their solutions. Students find ways to explain.
+) Type 2: is called Modified discovery learning. Teacher gives problems.

Students give the solutions.
+) Type 3: is called Free discovery learning. Students give problems and

the solutions by themselves. (Nguyen, 2007)
b. The levels of discovery learning

There are many ways of approaching the levels of discovery learning. We can
divide the levels of discovery learning based on the content and the way of
approaching the content of a problem. However, we can also divide the levels of
discovery learning according to the learning level of students.
Weak students
The discovery level of weak students is the simplest. Weak students only need to
remember the information and events without needing to understand them.
The discovery learning acts of weak students are usually: identification,
definition, name calling, writing, expressing, etc.
Average students
The level of the discovery of average students is higher than of weak students.
Average students have to not only understand and explain the given
information but also apply the information to new situations and events and
solve the given problems.
The discovery learning acts of average students are usually: classification,
comparison, ordering, reasoning, distinguishing, applying, changing, collecting,
etc.
rather good students
The level of the discovery learning of rather good students is higher than the
two above levels. Rather good students have abilities to divide information into
many components in order to know the inner relations and their structures. In
addition, students must have competence to connect pieces of information in
order to create new ideas, generalize information and infer corollaries.
The discovery learning acts of rather good students are usually: analyzing,
explaining, connecting, asking questions, reasoning, collecting, combining,
constructing, designing, developing, integrating, organizing, etc.
Good students
The level of discovery learning of good students is the highest. Good students
must have competence to identify and evaluate information, problems, things,
phenomena following a specific target. Good students should know how to
create problems and solve them.
The discovery acts of good students are usually: affirming, connecting,
assessing, explaining, finding out the mistakes, predicting, creating, etc.
Applying e-books to teach planar geometry according to the organization of discovery
acts
E-books allow us to teach according to the organization of discovery acts
effectively with an infinite number of students. Teacher easily organizes
discovery acts thanks to the page classification design of e-books. This thing is

89
illustrated by teaching The transformation in the first teaching period Introduction

to transformation of the Vietnamese advanced geometric text book 11th.
Example 1
With a given line a and hyperbola ( H ), for each point M on ( H ), let
M ' be a point symmetric to the point M with respect to a . Give remarks on
the positional points M ' when M moves on ( H ).
Table 3. The discovery learning of the symmetry with the help of e-books
Levels Learning Question content The corresponding figure
aptitude of
a student
1 Very Give remarks
good on the
positional point
M ' when M
is on the
hyperbola.
2 Good Give remarks

on the
positional
points M'
when points
M are on the
hyperbola.
3 Rather Give remarks

good on the
positional
points M'
when points
M are on the
hyperbola.
4 Average Click on the

Interaction
button to
predict the
locus. Find the
locus of points
M '.

90
5 Weak Find the locus

of points M ' .
After organizing discovery acts, teacher has students give remarks:

The symmetry on line a is a transformation that maps each point M onto point
M ' symmetric to the point M with respect to a.
Symbols and terms
The symmetry on line a is usually called S a . The symmetry on a line is simply
called the symmetry.
Line a is called the axis of the symmetry, or simply called symmetry axis.
d. The process of divided learning
Step 1: define learning strategy
The strategy here is the divided learning one.
Step 2: build the process of the divided learning
Build the process of solving problems according to the divided strategy so that
its steps are as close to an algorithm as possible.
Step 3: predict and classify the mistakes of students
Define the expressions and causes of each mistake in order that we set up a
feedback mechanism. Students can make some familiar mistakes such as
calculating mistakes, logical mistakes, etc. And from that, teacher gives remedial
ways as well as supplemental instructions for students.
Step 4: divide the activities of students
Divide the process of learning into knowledge doses to form the program structure
and connect these knowledge doses. The doses are divided into small ones so
that they can predict the thoughts of students happening at these doses based on
the answer results of students. Dividing knowledge doses is related to the
classification of the mistakes of students. The learning effectiveness is much
higher if the process of divided learning is suitable to the thought level of each
student. Teacher needs to give chosen tests and students will be instructed based
on suitable reasoning branches. Good and pretty good students will follow the
shortest branch while average and bad students will follow the longer one.
Step 5: write the chart of divided learning
E-books automatically give the chart of divided learning for each student as well
as the statistical table of correct and incorrect choices immediately. This thing
helps students to self-check their mistakes in solving problems.
e. Divided learning in the environment of e-books
Divided learning in the environment of e-books allows us to remedy the weak
points of traditional learning. It allows us to educate an infinite number of
students, not depending on learning space, learning time as well as learning
place. Divided learning in the environment of e-books happens at every time,
everywhere. Students are provided with feedback as well as instructions
immediately. After students finish their choices, they will get a statistical chart
as well as a statistical table of their choices. This thing helps students to check

91
their mistakes easily, and from that, students can remedy the mistakes in the
next problem.
Example 2
With two given fixed points B, C on the circle (O ; R) and a point A moving
on this circle, prove that the orthocenter of ABC is on a fixed circle.
The divided solution in the interactive e-book is as follows:
Knowledge (TT1)
According to the hypothesis, B, C are fixed on the Figure
circle (O ; R) . Thus, BC is fixed. The directions of line
d are a set of lines that are parallel or coincident with
line d. Hence, the direction of line
(A) AH is (B) AB is (C) BH is (D) OH is
invariant invariant invariant invariant
Branch 1 (Branch ( A ))
If students choose answer ( A ), this are correct. They continue to deal with
knowledge (TT2).
Branch 2 (Branch ( B ))
If students choose the answer ( B ), this is incorrect. The system of the e-book
gives the complement instruction Hd and the situations of the mistakes of
students:
The complement instruction Hd
 A moves on the circle, so the line connecting a fixed point with point A
always has changeable directions.
 A moves on the circle, so the orthocenter H of triangle ABC is also
movable. Thus, the line connecting a fixed point with point H always has
changeable directions.
The invariant line is the line … Figure
( A2 ) AB . ( B2 ) ( C2 ) AO . ( D2 ) AC .
Since AB AH . Since AO Since AC
always Since AH always always
makes BC always makes BC makes BC
a constant makes BC a constant a constant
angle. a constant angle. angle.
angle.
 If students choose answer ( B2 ), this is correct. They continue to deal with the
knowledge (TT2).
 If students choose the answers ( A2 ), ( C2 ), or ( D2 ), they come back to the
knowledge (TT1) to relearn.
Branch 3 (Branch ( C ))
If students choose answer ( C ), this is incorrect. The system of the e-book gives
the complement instruction Hd and the situations of the mistakes of students:

92

movable. Thus, the line connecting a fixed point with point H always has
The invariant line is line … Figure
( A3 ) BH . ( B3 ) CH . ( C3 ) BA . ( D3 ) AH .
Since BH Since CH Since OH Since AH
always always always always
makes makes makes makes
BC a BC a BC a BC a
constant constant constant constant
angle angle. angle. angle.
 If students choose answer ( D3 ), this is correct. They continue to deal with the
knowledge (TT2).
 If students choose the answers ( A3 ), ( B3 ), or ( C3 ), they come back to the
Branch 4 (Branch ( D ))
If students choose answer ( D ), this is incorrect. The system of the e-book gives
movable. Thus the line connecting a fixed point with point H always has
The invariant line is line … Figure
( A4 ) OA . ( B4 ) BH . ( C4 ) CH . ( D4 ) AH .
Since OA Since BH Since CH Since AH
always always always always
makes BC makes BC makes BC makes BC
a constant a constant a constant a constant
angle. angle. angle. angle.
 If students choose the answer ( D4 ), this is correct. They continue to deal with
the knowledge (TT2).
 If students choose answers ( A4 ), ( B4 ), or ( C4 ), they come back to the
Knowledge (TT2)
Since the directions of line AH are invariant, we use a Figure

translation of directions AH . Because there are the
only points A, H relevant to directions AH , we

prove that AH is a constant vector. In order to prove
AH is constant, we prove AH equals to a constant
segment. According to the hypothesis, B, C are fixed,

93
so we will prove that AH equals to B ' B or C ' C and

B ' B , C ' C also have directions AH . Hence, if we
construct a parallelogram AHBB ' , BB ' is a diameter
of circle (O) circumscribing triangle ABC. Since
(A) line (B) line (C) line B ' B (D)
B ' B is B' A is is B ', H , O
perpendicul perpendicul perpendicula are
-ar to BC -ar to AC -r to BC collinear.
(because (because (because
B ' B // AH B ' A // BH B ' B // AH )
) ) and line
B' A is
perpendicula
r to AC
(because
B ' A // BH )
Branch 1 (Branch ( C ))
If students choose answer ( C ), this is correct. They continue to deal with the
knowledge (TT3).
Branch 2 (Branch ( A ))
If students choose answer ( A ), this is incorrect. The system of the e-book gives
 If B  ' AC  B ' BC  900 , B ' C is a diameter of circle ( O ) circumscribing
triangle ABC.
line B ' B  BC , so Figure
( A1 ) ( B1 ) ( C1 ) B ' AEB is ( D1 )

B ' AC  B ' BC  900 B ' BFA is a B ' ACB
a parallelogram is a
(since line B ' A is
parallelo- ( E is the foot prallelo-
also perpendicular
gram ( F of the altitude gram
to AC )
is the foot dropped from
of the B to AC ).
altitude
dropped
from A
to BC ).
 If students choose answer ( A1 ), this is correct. They continue to deal with the
knowledge (TT3).
 If students choose answers ( B1 ), ( C1 ), or ( D1 ), they come back to the
Branch 3 (Branch ( B ))
If students choose answer ( B ), this is incorrect. The system of e-books gives the
complement instruction Hd and the situations of the mistakes of students:

94
 If B  ' AC  B ' BC  900 , B ' C is a diameter of circle ( O ) circumscribing

triangle ABC.
line B ' A  AC it follows Figure
( A2 ) ( B2 ) ( C2 ) ( D2 )

B ' AC  B ' BC  900 B ' AOH B ' AEB is B ' AFB is
is a a a
(since line B ' B is
parllel- prallelogr- parllelogr-
also perpendicular
ogram. am ( E is am ( F is
to BC )
the foot of the foot of
the altitude the altitude
dropped dropped
from B to from A to
AC ). BC )
 If students choose answer ( A2 ), this is correct. They continue to deal with the
knowledge (TT3).
 If students choose answers ( B2 ), ( C2 ), or ( D2 ), they come back to the
Branch 4 (Branch ( D ))
If students choose answer ( D ), this is incorrect. The system of the e-book gives
 If B ' AC  B ' BC  900 , B ' C is a diameter of circle ( O ) circumscribing
triangle ABC.
That B ', H , O are collinear is a wrong statement because Figure
( A4 ) B ', D, E ( B4 ) ( C4 ) B ', O, C are ( D4 )
are collinear B ', D, F collinear ( B ', D, O
are  
B ' AC  B ' BC  900 are
colline- ) collinear
ar
 If students choose answer ( C4 ), this is correct. They continue to deal with the
knowledge (TT3).
 If students choose answers ( A4 ), ( B4 ), or ( D4 ), they come back to the
Knowledge (TT3)
Thus, B ' is a fixed point on circle ( O ) ( B ' is the Figure
intersection point of the fixed line CO with O )) and
// 
B ' B  AH . Translating to vector B ' B , we have:

95
(A) H  F (B) A  H (C) H  A (D) E  O
 If students choose answer ( B ), this is correct. They continue to deal with the
knowledge (TT4).
 If students choose answers ( A ), ( C ), or ( D ), they come back to the
Knowledge (TT4)
Because A moves on circle ( O ), the Figure
locus of points H (when A moves
on circle ( O )) belongs to circle ( O ' )
being the image of circle ( O ) through

vector B ' B.
After answers have been chosen, the e-book automatically gives the branches of
the user’s choice. Its interface is as follows:
The statistical table interface of answers is as follows:
12. Self-learn task: the e-book gives the self-learn request and self-learn task to
each student. This function is similar to the function e-check.

96
13. E-portfolio: allows us to record the whole learning process of users.

14. The functions Register, Password change, information are the same as normal
webs.
3. Results and discussion

We delivered survey forms to 117 teachers of upper secondary schools in Ho Chi
Minh city, Viet Nam in order to check the application of the interactive e-book in
learning mathematics. The result shows that 70 teachers (59.83%) think that it is
very necessary; 37 teachers (31.62%) think that it is necessary; 10 teachers (8.55%)
think that it is not necessary yet; nobody thinks that it is not necessary. It shows
that teachers are interested in using interactive e-book in teaching.
The chart 1.1. The attitude of teacher towards the application
of the interactive e-book in teaching
The attitude of teacher towarsd the application of

the interactive e-book in learning
8.55% 0%
Very necessary
Necessary
31.62%
59.83% Not necessary yet
Not neceassry
We also surveyed the target of using interactive e-book in lessons. The result is
as follows:
Chart 1.2: The target of using interactive e-book in teaching
Percentage %
59.83 62.39
70 55.56
60
50
40 29.91
25.64
30
Percentage %
20
10
0
To ilustrate To stimulate To instruct To practise To
according the interest student to the skills of strengthen
to lesson of learning discover applying e- the practice
knowledge book
Thus, the majority of teachers in the survey asserted that they had used
interactive e-books to stimulate the interest of students. However, there were
still many teachers using the application of the interactive e-book to illustrate
their lessons (70 teachers (59.83%)). Whereas the most important target of using
interactive e-books is to help students find out knowledge. There were only 65

97
(55.56%) teachers applying this. Besides, very few teachers (35 teachers (29.91%))
were interested in applying the interactive e-book to form knowledge, to
practice skills, and to strengthen the practice of students. By observing,
attending the lesson together with interviewing, we realize that there are a lot of
reasons that make teachers fail to use the interactive e-book. Some teachers do
not know how to use the interactive e-book. Some other teachers supposed that
they were not fully equipped with informatics, so they were puzzled about
teaching lessons.
We also delivered survey forms to 251 students on the application of the
interactive e-book in learning. The result is as follows:
Chart 1.3. The effect of the interactive e-book on students
Percentage %
69.72
63.35 64.94
70 59.36
53.79
60
50
40
30
Percentage %
20
10
0
To increase To help To motivate To help To help
the interest of student to students to students to students to be
learning study create self-learn confident
initiatively
Chart 1.3 shows that students are interested in using interactive e-books
in learning. There are 159 students feeling interested in learning (63.65%) and
especially, 175 students (69.72%) feel that this e-book helps them to study. The
interactive e-book improves students’ self-learning and self-study. Students feel
more confident when they learn with the help of the interactive e-book.
4. Conclusion
E-books have many strong points than other kinds of education, which
are easy to use and to learn everywhere, every-time. E-books can act on-line or
off-line, which allows us to e-educate, to e-exchange, to transmit images, sound,
MP3, and MP4 files, or to interact, etc. In addition, e-books can be designed
completely to improve the activeness of students, the classification of students,
the individual learning of students in order to help the process of learning
develop best. Finally, e-books allow us to educate an infinite number of students
without depending on location and learning time, so they are good for learning.
References
Cumaoglu G., Sacici E, & Torun K. (2013) E-Book versus Printed Materials: Preferences
of University Students, Contemporary educational technology, 4(2), 121-135.
Leslie Czechowski (2011), Problems with E-books: Suggestions for Publishers, Journal of
the Medical Library Association, 99, 3, pp 181–182.

98
Dao, T. L., & Nguyen, V. H. (2010). Learning mathematical concept and theorem
according to the divided strategy in e-learning environment, Vietnamese
Educational Journal, (244), 33-34, 51.
Encyclopedia. Retrieved from http://www.pcmag.com/encyclopedia/term/42214/e-
book
Hibbard, L. E. (2014), Ebooks: An Alternative to Paper Books for Online Students?,
International Journal of Learning, Teaching and Educational Research, 46-56.
Dracine Hodges, Cyndi Preston, and Marsha J. Hamilton (2010), Resolving the Challenge
of E-Books, Collection Management, Vol. 35, No. 3 & 4, pp.196–200.
Hanho Jeong (2010), A comparison of the influence of electronic books and paper books
on reading comprehension, eye fatigue, and perception. Retrieved
from file:///C:/Users/ADMIN/Downloads/Students_who_read_print_b
ooks_have_a_better_reading_comprehension_of_the_text_and_prefer_paper
_books_over_e_books%20(1).pdf.
Le, T. T. T (2013). Some activities of forming evaluated competence of learning results for
students at primary educated department, Vietnamese Educational Journal,
(323), 39-40.
Nguyen, V. H. (2007). Applying guided discovery teaching in the process of teaching
mathematics at upper secondary school. Journal of Vietnamese Educational
Science, 28-29.
Nguyen C. T., Nguyen K., Le K. B., & Vu V. T. (2004), Learning and teaching the way of
learning, Vietnamese Pedagogy University Publishing House.
Nguyen C. T., Nguyen K., Vu V. T, & Bui T. (1998), The process of teaching – self-learning,
Education Publishing House.
Owen V., Tiessen R., Weir L, DesRoches D., & Noel W. (2008), E-Books in Research
Libraries: Issues of Access and Use. Retrieved from http://carl-
abrc.ca/uploads/pdfs/copyright/carl_e-book_report-e.pdf.
Tran T., Dang X. C., Nguyen V. H., & Nguyen D. N. (2011), The application of informatics
in teaching mathematics at secondary school, Education Publishing House.
Tran V., & Le Q. H. (2006), The discovery of grade 10th geometry with the Geometer’s
Sketchpad, Eduaction Publishing House.
Tran Vui., & Le Q. H. (2007), The discovery of grade 11th geometry with the Geometer’s
Sketchpad, Eduaction Publishing House.
Walters W. H. (2013), E-books in Academic Libraries: Challenges for Acquisition and
Collection Management, Libraries and the Academy, Vol. 13, No. 2, pp. 187–
211.
Moet.gov.vn. Retrieved from http://www.moet.gov.vn/?page=9.6
Vst.vista.gov.vn. Re trieved from http://vst.vista.gov.vn/home/database/an_pham_die
n_tu/MagazineName.2004-06-09.1932/2004/2004_00004/MItem.2004-09-30.4740
/MArticle.2004-09-30.5116/marticle_view.
Wikipedia. Retrieved from http://en.wikipedia.org/wiki/E-book.

99

Vol. 11, No. 1, pp. 99-116, April 2015
Impact Investigation of using a Digital Literacy

Technology on a Module: Case Study of Tophat
Xue Zhou and Stella-Maris Orim

Coventry University
West Midlands, England
Abstract. With digital technologies replacing the previous source of

entertainment and communication and becoming the primary means by
which we are informed and entertained, there is a need to be informed
and competent with the relevant skills. Also, there is a greater need to
use the digital devices that are employed by students using these
technologies, which tend to keep them captivated at all times. The aim
of this study is twofold: to enhance student performance by improving
the approach to feedback and enhance student engagement by
improving student class participation with the use of their „disruptive‟
devices. We hypothesize that student engagement with the digital
devices lead to better and overall student performance. The areas
identified for research were in relation to student feedback and
engagement through the encouragement of their participation. The
approach adopted in this research was the evaluation of the use of
Tophat as a tool in the creation of an enhanced student-centred learning
experience by creating an active learning environment. The strategy was
to adhere to the usual idea of student tutorial and after an hour the first
research activity started. This involved the use of the disruptive devices
(mobile phones, iPads and Android devices) in a constructive manner.
At the end of the session, Tophat was used to provide additional
feedback and prepare them for their coursework. This was achieved
through the use of „Tournament‟ to find out each week those that
engaged and performed well. The results showed that 74.8% of the
students are of the view that the use of Tophat enhanced their
engagement in the module and 71.9% students perceived that higher
level of feedback was received through Tophat. The impact of digital
technology in higher education was discussed in this study.
Keywords: Tophat; Technology enhance learning; Digital classroom;

Students‟ engagement; feedback.
Introduction
The effect of the use of digital literacy technology on students has become a
topical area for research, which is not surprising as education in itself has to
meet the students in their own arena. It is an acceptable requirement that
@2015 The authors and IJLTER.ORG. All rights reserved.

100
institutions of learning should engage in the development of creativity and

innovation in students (Kärkkäinen & Stéphan 2013). However, this is a tall-
order which is difficult to achieve when the educational systems still function
traditionally and are losing the students to digital media; even in the classrooms.
It is required that higher institutions should engage with providing students

with skills for innovation, however, there is the question of the type of teaching,
learning and assessment that can help students learn and acquire skills for
creativity and innovation. Result from the research carried out by Sander (2000)
suggested that students preferred to be taught by interactive lectures and group-
based activities. Their least preferred teaching and learning methods were
traditional lectures, role-play and oral presentations. This is similar to the
experiences with students in the classroom today. Coursework assessment
preference was for essays, research projects and problems/exercises.
Students’ engagement
Engagement
Student academic engagement involves the willing commitment of the students
to the course of their academic pursuits. Kuh (2009) stated broadly that student
engagement is reflected in the amount of time and effort students put into
achieving college outcomes. This involves the participation in the achievement
of the required learning outcomes before they come into the classroom, in the
classroom and after they leave the classroom. Students‟ engagement could be
enhanced and influenced by instructors‟ pedagogical choices and practices (Lane
and Shelton 2001). Such good practices provide students with prompt feedback,
encourage active learning and communicate high expectations, encourage
interaction between students and faculty, cooperation among students and
respect different talents and ways of learning (Chickering and Gammon 1987).
Performance
The academic performance of students is viewed as a measure of the students‟
ability to show that they have achieved the learning outcomes in a particular
course. This can be measured in myriad ways such as: attendance monitoring,
observation, interview, tracking their online engagement with course content
and participation, self-reporting. Also, there are other traditional means of
checking performance based on the achievement of learning outcomes: essays,
oral and poster presentation, critical reviews, discussions, examination and test.
Digital classroom in higher education
In the past, traditional ways of teaching and learning were upheld in the higher
education institutions. This was viewed as a process of transmitting content to
the students and comprised of the lecturers deciding on the topics, teaching and
assessment methods (Biggs and Tang 2010). However, in today's larger classes
with diversified students, many lecturers could encounter major difficulties in
sustaining academic standards. In relation to these difficulties Biggs (1999) states
that they can be overcome when all components of teaching and learning are
aligned constructively. This will be based on the premise that learning objectives

101
clearly define what is required from students. This teaching context of

constructive alignment encourages students to carry out the learning activities
which will help them attain the required understanding (Biggs and Tang 2010).
Students are thus encouraged to engage in learning activities that are relevant in
achieving these learning outcomes. Biggs and Tang (2015) explained that in this
context, teaching is not topic-based, as is traditional teaching, but focuses on
what students are intended to do after they have learned the curriculum topics.
As a result, several efforts have been placed in developing activities that could
engage students while enhancing the attainment of learning outcomes. Since the
modern classroom is faced with several challenges with student engagement as
one of the key issue, there have been efforts in the use of students‟ digital
devices in fostering engagement while enhancing learning.
Digital tools in education
Due to several technological innovations and the dependence of students on

these innovations, the higher education is undergoing significant changes in
their methods of teaching, learning and assessment. This is because, as Prensky
(2001:1) puts it, „today‟s students are no longer the people our educational
system was designed to teach‟. Prensky (2001) further explained his view that
due to the ubiquitous environment and the sheer volume of students‟ interaction
with it, they process and think through information differently from their
predecessors. He referred to them as „native speakers‟ of the digital language.
This places a lot of requirement on the educational institutions that cater for the
needs of these students.
To meet these needs, the setting of the lecture hall remains the convention in
most higher institutions, but, these are being enhanced by the integration of new
tools, techniques and pedagogies (McAleese et al. 2013) which these „native
speakers‟ are conversant with. This integration has necessitated studies in
relation to the best use of innovative technologies in higher institutions. A
reflection on what Ihde (1993) calls the „active relational pair‟ presents a view on
the ways in which mobile devices have become absorbed into human social
networking practices. Robinson and Hullinger (2008) also found that
asynchronous instructional technology encourage students to achieve higher
order thinking skills such as evaluation, analysis, synthesis, judgement, and
application of knowledge.
In corroboration with this view, Merchant (2012) observes that the mobile
phone, with Twitter, Facebook and YouTube is heavily marketed by a range of
providers due to human reliance in their everyday lives. This is due to the fact
that most customers rely heavily on the use of these applications in their daily
social lives. Also in line with meeting the needs of the digital age by innovative
ways to engage students through the use of various applications, games and
tools in the classroom, Wilson and McManimon (2014) corroborated with
McAleese et al. (2013). However, they argue that best practice session is utilizing

102
the cloud as a tool to bridge the learning gap by providing more useful
instruments for the enhancement of teaching styles.
Student classroom response systems, which allow the instructor to gauge

students‟ understanding of a given topic in real time, have been widely used in
the higher education. The results showed that such fast response system can
improve students‟ in-class attention, attendance, interests and involvement in
the learning process, but also allow lecturer alter the content according to the
students response (Bruff 2009; Terrion and Aceti 2012). Clicker as one of student
response system has been proved to forester student engagement and success in
the module (Hoeskstra 2008; Kaleta and Joosten 2007)
It is arguable that the integration of these new technologies and approaches to

education are already having a clear and positive impact on higher education
provision as seen in the findings of McAleese et al. (2013). They established that
this integration presents the opportunity to design the methods of teaching to
individual students‟ needs, advancing learning analytics which lead to quicker
feedback on students‟ performance.
These technologies afford opportunities to learn anytime, anywhere and from

anyone, provided the learner is motivated. This flexibility is crucial for non-
traditional learners whom Prensky (2001) refer to as „digital natives‟. However,
besides obvious benefits to the students, McAleese et al. (2013) articulated other
benefits in relation to the institutions and governments. These are: a change in
the approach to continuing professional development and lifelong learning; an
important tool to governments in ensuring a diversity of provision within higher
education systems to meet the needs of all learners and provision of the platform
for reaching international markets which will complement existing
developments in cross-border education. Also, there is the benefit identified in
relation to the facilitation of greater collaboration with both global and local
partners.
Regardless of these identified benefits, there are some identified disadvantages

to the move towards digitalising classrooms. In this regard, Merchant (2012: 770)
stated that „as ownership and access to smartphones has spread into the teenage
years, their place in institutions of formal education has been marked by
contention‟. This is a view which is widely held in a number of conventional
institutions of learning. The view that mobiles have no place in the classroom
has recently been contested by educators, such as Parry (2011), who suggest that
mobile learning and literacies should play an important role in education. Parry
argues that it is crucial to recognise that mobile computing power radically
changes the classrooms and more importantly, the „spaces that students inhabit
and the conversations they have outside of our teaching‟ (2011: 17).
Woodcock et al. (2012: 80) observed that Technology is increasingly being

introduced into the classroom, primarily through student-led activities and
typically through the use of smartphones. Some of these activities are web-
based and involve the use of applications such as QuizIt, Socrative and Tophat.
These activities have been found to be useful in a lot of cases, and Liburd and

103
Christensen (2013) suggest that technology, particularly web 2.0, can help
increase the depth of learning by increasing interaction, critical thinking, and
collaboration.
However, there are identified issues in relation to the adoption of innovative

teaching approaches. One of such is students‟ willingness to accept changes
from the traditional way of teaching and learning. Besides the difficulty of some
lecturers in adopting these innovative approaches, they are also a bit
apprehensive about how students would respond to them. Brown et al. (2013:80)
found that students are „in fact able and willing to use a classroom response and
engagement system in order to increase engagement‟. Additionally, findings
from their research suggest that students have an overall desire to use
technology in the classroom. Brown et al. (2013:81) found in their study that
contrary to perceived fears of lecturers that students may not all have smart
devices in the lecture room, which can use for the innovative interactions, all the
413 respondents in their survey had „at least one device that would be capable of
responding via text message or through a browser‟.
Several devices and innovative tools and software have been introduced to
different classroom settings. The results have varied slightly, but what all these
results have in common is a positive disposition of most of the students to the
innovative practices. Some have used clickers as a means of enhancing students‟
classroom engagement. Park and Farag (2015) explored the use of addressing
clicker in a legal studies course. Findings from their studies suggest that both
lecturers and students are more engaged with the course material and in the
process of teaching and learning. He claims that clickers can be used to break-up
the monotony of lecture, assess student understanding of material and difficult
concepts, and identify areas of student misunderstanding and confusion. This
can give the lecturer an idea of where to focus on. Also, they suggested that the
use of clickers give every student, even those who are uncomfortable
participating in class, an opportunity to provide input (Farag 2015).
Common tools used in these innovative approaches include iPad, smartphones,

laptops etc. Ravishankar et al. (2014) investigated the potential impact of the
iPad/Tablet and its applications on teaching and learning in the area of electrical
engineering. Their findings suggest that these devices may transform the
teaching into a collaborative and interactive way.
An example of innovative software is the android classroom response system.

Karakostas et al (2014) carried out a pilot study on the use of QuizIt which is a
new prototype real-time response system for Android mobile devices meant to
enhance active learning methods and assess students' understanding. Their
results from a questionnaire-based evaluation show that the students were quite
positive about the use of QuizIt as a supporting system to their laboratory
course.
Some others have used Socrative as an online Student Response System to

increase in-class student engagement. Dervan (2014) investigated the use of
Socrative cloud-based (Internet) Student Response System (SRS) in improving
student engagement and the learning experience, compared with the traditional

104
lecture setting. The findings from his study were positive suggesting that the use
of Socrative as an online Student Response System increased the in-class
engagement of students.
Overview of Tophat
In the light of the argument by Parry (2011), the use of several tools and
techniques have been explored in educating students (Park and Farag 2015,
Ravishankar et al. 2014, Karakostas et al 2014) but there is not much recorded on
the use of Tophat.
A traditional classroom is based largely on lectures, but with the use of

innovative techniques and tools, students are encouraged to participate in the
teaching and learning process. In most cases, they are encouraged to take
ownership of their learning. With the use of Tophat (Tophat.com) engagement
and participation is encouraged by stimulating the students through discussion
start-ups, asking questions, and gauging understanding with the students‟ own
devices such as iPad, smartphones and laptops. The software makers boast of
providing a command centre where one can run and manage all aspects of the
classroom from one platform.
Research Purpose
Besides the issue of the appropriate type of teaching, learning and assessment,
there is also the issue with the students‟ learning styles which are diverse. An
attempt at developing strategies that will focus on the different students‟
learning styles would result to student engagement and likely lead to enhanced
student performance. Krause et al. (2008) argues that it is imperative to develop
a broader understanding of engagement as a process with several dimensions.
Since technology continues to be increasingly used by educational institutions
(Becta 2009), this implies the need for appropriating pedagogical and education
tools in supporting the enhancement of the quality of student experience.
This need has necessitated several researchers to carry out studies on pedagogy,
digital literacy technologies, student engagement and performance. However,
there has not been any study on the impact of the use of TOPHAT as a digital
literacy technology tool on students‟ engagement and feedback provision.
Consequently, this study will focus on the following questions:
RQ1: Does student class participation improve with the use of their „disruptive‟
devices on the Tophat platform?
RQ2: What is the perceived impact of the use of TOPHAT on students‟
engagement in the module?
RQ3: Does Tophat increase the amount of formative feedback received by
students?

105
Methods
Participants
The participants were postgraduate international students in Engineering

Management at Coventry University in UK. A total of 187 students enrolled in
the course however, 103 students (55%) participated in the survey.
Design and Procedure
Students who enrolled in engineering management course were invited to

participant in this study. Students were informed that their consent entailed the
ability to withdraw out of the study at any time without punishment. The
module lasted for eight weeks, as part of the assessment; students need to
complete the coursework by applying the skills acquired over the period of the
module. As part of the module requirements, students were directed on how to
set up Tophat accounts during the lecture in the first week of the module. At the
end of fifty minutes lecture session, students were asked to use their digital
devices (e.g. smart phones, IPad, laptop) to access different questions,
tournament and discussion forum, which were designed to monitor class
attendance, test their understanding of lecture contents and serve as revision
questions for the piece of coursework. The Tophat accounts were administered
by the two researchers. University student email addresses were used to send
the students invitation to the platform. Educational activities on the Tophat
platform were set up by the researchers and these included questions design,
attendance register, tournament etc.
Instruments
An online survey was designed and administered through Tophat to measure

the students‟ perceptions of the impact of Tophat on their module engagement,
teaching and learning experiences. Two sets of questions were designed and
released to the students though their digital devices. The first set was for testing
the students‟ understanding of the module content, while the second set was for
investigating students‟ opinions and learning experiences.
In the first set of designed questions, different types were used, which included
multiple choices, word, numeric, sorting problem, matching problem and „click
on target‟ questions. The second set comprised 4 Likert-scale question using
four-point agreement level from 1 (Strongly disagree) to 4 (Strongly agree) and
qualitative word questions.
Analysis
Descriptive statistics analysis in the form of percentages and means were used to
analyse the demographic data of the participants. The students‟ gender, age and
technology level were assessed. Four Likert scale survey questions focused on
the impact of Tophat on students‟ engagement, understanding of module
content, feedback, coursework feed-forward. The students were also asked to
evaluate the use of Tophat and its impact on their engagement and feedback

106
received on the module. Additionally, students‟ preferred question type was

observed based on their response. The researchers had access to the data of
students‟ class attendance, correctness of responses, participation weight.
T-test and ANOVA test analysis were carried out to assess the differences in
students‟ engagement and understanding for the different gender and age
groups respectively. PASW (SPSS) Statistics Version 17 was used to analyse the
data exported from Tophat. To evaluate the degree of students engagement in
the module through Tophat, thematic analysis technique was utilised to
determine the efficiency of Tophat in enhancing students‟ engagement and feed-
forward for the integrated coursework.
Results
The presentation of the results is mapped onto the structure of the earlier section
“analysis”. The data showed that reliability cronbach‟s alpha (α) was 0.796,
indicating acceptable reliability.
Description of survey data
The age of 103 participants ranged from 19 to 45 years, with 29% female and 71%
male, which were distributed as follows: Under 21 years old (6.8%), 22 – 25years
old (70%), 26 – 30 years old (12.6%), 31-35 years old (5.8%), over 36 years old
(4.8%). The finding showed that the average score of participant self-appraisal of
their technological competency was seven out of ten. The means and standard
deviations of the responses to the four Likert scale questions addressing the
students‟ perception of Tophat are provided in Table 1. Students‟ perceptions of
the effects of Tophat were distributed as follows: enhanced their engagement
(74.8%), enhance understanding of the topics (67%), enhance level of feedback
(71.9%) and feedforward to complete their coursework (61.1%).
Table 1: Descriptive statistics for Likert-scale questions
Construct Mean Strongly Disagree Agree Strongly Standard

Disagree Agree Deviation
1. The use of Tophat 2.825 6.8% 18.4% 60.2% 14.6% 0.75961
has enhanced my
engagement.
2. The use of Tophat 2.737 7.8% 25.2% 52.4% 14.6% 0.80393
has enhanced my
understanding of the
topics
3. The use of Tophat on 2.835 5.8% 22.3% 54.4% 17.5% 0.78086
this module has
enhanced the level of
feedback
4. The use of Tophat on 2.670 6.8% 32% 48.5% 12.6% 0.78451
this module for
feedback has impacted
positively on my ability
to complete my
coursework.

107
Experience with Tophat based on Gender

Two gender groups exist; male and female. The t-test assesses whether the mean
results of the two gender groups are statistically different. Table 2 below shows
the results of the t-test which examined the impact of gender groups on the
scales of Technological skill, Engagement in the module, understanding of the
module content, level of feedback, and coursework feed-forward. No significant
difference was found between male and female students‟ response to those
components.
Table 2: Demographic differences by gender group (n=103)
Variable Mean t Sig. Multiples

Male Female Differences
Technological skill level 7.1233 7.3000 0.193 0.662 None
Engagement 2.8904 2.6667 1.860 0.176 None
Understanding of module 2.7671 2.6667 0.330 0.567 None
Feedback 2.8630 2.7667 0.322 0.572 None
Coursework Feedforward 2.7123 2.5667 0.731 0.395 None
Experiences with Tophat based on Age
The variable of age was divided into five groups: below 21; 22 to 25; 26 to 30; 31
to 35 and over 36. An ANOVA test was conducted to test how age influences
different students‟ perceptions of Tophat. The results of the ANOVA test are
shown in Table 3. The result showed that there is no significant difference
among different age group of students‟ perception of Tophat.
Table 3: Demographic differences by age group (n=103)
Variable Mean F Sig. Multiples

A B C D E Differences
(<21) (22- (26- (31- (≥36)
25) 30) 35)
Technological 8.429 7.039 7.333 7.250 7.000 0.936 0.446 None
skill level
Engagement 3.143 2.831 2.667 2.750 2.667 0.470 0.757 None
Understanding 3.000 2.740 2.583 2.750 2.667 0.295 0.881 None
Feedback 2.857 2.870 2.583 2.750 3.000 0.388 0.817 None
Coursework 2.857 2.714 2.250 2.750 2.667 1.032 0.394 None
Feedforward
Thematic analysis of students’ engagement
The qualitative data collected was analysed thematically with the initial codes
selected based on the data. These codes were the overriding concepts from the
data, which were presented as negative and positive views (Table 4 and 5,
column 3), where applicable. Presented in Table 4 – 7, are quotes (column 2),

108
which are representative of the students‟ views, as depicted in the codes

(column 1).
In relation to the perception of the students‟ on the way Tophat affected their
engagement on the Module, several views were aired. Some of these quotes are
presented in Table 4 below and were coded broadly as impacting their:
participation; provision of game-based learning, a learning community;
emphasis on learning outcome and revision. A few of these comments presented
in table 4 are:
It has impacted my engagement in this module positively in the sense that it

makes the course look easy by revising what has been taught in the tutorials with
all the students which makes it more interesting, interactive and well understood
through the feedback given after any questions asked.
Tournament is an interesting feature and makes us more engaged to the Tophat
Although a few participating students did not respond to this qualitative

question, 47% of the entire sample believed it impacted on their engagement
while 12% felt otherwise. A few of their quotes were:
It was a total waste of time. I don't learn anything from it
I learned from class lectures not from Top Hat
In terms of the impact of Tophat on feedback provision, students‟ perceptions are

presented in Table 5. The quotations are coded into six main parts: Clarity, feed-forward,
quality, timeliness, approach and accuracy. Some comments by the students which are
presented in table 5.
We get feedbacks immediately after each session
Tophat helps to track our progress throughout the module; it has a user friendly
interface
However, 10% of the entire sample believed that quality of feedback should not be
delivered on Tophat platform. A few of their quotes were:
I can't really say something about it that depends on evaluator if he/she thinks
Tophat is of more value to this course but for me feedback should not be based on
Tophat.
Not much. As sometimes I don't understand why the answer is wrong.
Students were also asked their most and least favourite feature of Tophat. Their
quotations are listed in Table 6 and Table 7. According to students‟ response, the use
friendly interface, game-based learning (tournament) and instant feedback encourage
students to involve in the learning process. However, students also expressed their
concerns in relation to the use of Tophat, such as: predominantly use a surface learning
approach, low quality of the feedback, instability of technical support (wifi, devices,
Tophat did not integrate with the commonly used education platform – Moodle).

109
Table 4: Summary analysis of the impact of Tophat on students‟ engagement
Q: Would you say Tophat has impacted on your engagement in this module?
Codes Positive (48) Negative
(12)
Participation  Easy for sign, it has the code for each lecture.  Nothing. I
it has helped because I cannot be in the class mean when
while I'm not in the class I touch this
is the time I
must to
Game-based  Tournament is an interesting feature and
learning makes us more engaged to the Tophat
Learning  Easy to communication  I learned

community  Active learning from class
lectures not
from Top
Hat
Emphasis on  Being aware of the activity after every class,  It was a
learning made me concentrate fully to the lectures in total waste
outcome preparation of the activity. of time. I
 more engaged better understanding of terms don't learn
through multiple choice layout anything
 Provided a guide line for the study skills from it
 I can find the key point of lecture  Not
impacted
Revision  It has impacted my engagement in this  Help but

module positively in the sense that it makes not
the course look easy by revising what has necessary
been taught in the tutorials with all the
students which makes it more interesting,
interactive and well understood through the
feedback given after any questions asked.
 I remember the material after answering the
fast question
 Improved understanding and rehearsal
 To remind me to finish my work on time
 By getting knowledge from MCQ
 It can help me to understand the module
better
 It helps, but most of understanding is coming
from class.

110
Table 5: Summary analysis of the impact of Tophat on feedback provision
Q: How would you say Tophat has impact on the level of feedback in this module?
Codes Positive (52) Negative (18)
Clarity  Making me understand the meaning of  Not much. As
answer. But, I have a suggestion that I think sometimes I don't
we can finish Tophat at home understand why
 To see how much students learn from the answer is
lecture wrong.
 It help me to familiarize with research
Feed  development of my understanding
forward  I don't know how to say that but its helped
me through questions
 it increased my eagerness in learning
Quality  It quite level of feedback.  Not much
 Increased my satisfaction some, but very feedback received
little.  I'm not sure that
 high impact Tophat impact in
 Very good to get latest information feedback part
 Satisfyingly great
Timeliness  It has good impact because feedbacks are
always given on time
 We get feedbacks immediately after each
session
 The best thing with the Tophat that we get
the feedback immediately
 Get the feedback easy and quickly
 It had been perfect, proving prompt n
accurate feedback
Approach  This way of feedback was a better way  Not significantly
 Top Hat helps to track our progress  I can't really say
throughout the module , it has a user something about it
friendly interface that depends on
 Convenience evaluator if
 Easy way to deliver the feedback he/she thinks
 It's a good way to give feedback, helpful Tophat is of more
 Provided a very complete feedback system value to this
 It has made my learning experience more course but for me
fluid feedback should
not be based on
Tophat.
 The feedback for the module has been very
Accuracy accurate and good
 It has impact on the level of feedback
because of its interactivity amongst students
 It will help by getting different ideas from
different students

111
Table 6: Summary analysis of the students‟ dislike of Tophat
Q: What did you like about the use of Tophat in enhancing your learning experience?
Codes from the Actual quotations
data
Interactive  The fact that it made accessing information about the course
approach online interactive.
 Looked forward to the engagement after every class
 engaged me with e learning
Revision, feedback  Seems like a really good learning tool
and feed-forward to  Choice question
the coursework  Help me refresher course. Friendly interface.
 It can help me to enhance my understand of course and if I
have any question that I can ask there
Tournament  More interactive than writing on paper and creates
competition amongst fellow students
 Increased competitiveness
 Competition
 Competitive with others at the end.
Convenient  Was able to use device anywhere
 It could be convincible approach but during in dissertation
time.
 Everyone can answer
 Can use with phone
Easy to use and  No feedback on answered questions
user friendly  Easy to use
 Different types of questions in a good learning way
Test the  It made me question my understanding of the course because
understanding, I'm unable to answer questions in time
summaries the  It helps to understands the notes from the lectures
learning outcome  Improve understanding of lecture
 Giving me some choices, then ask which is correct.
 Asking questions about topics thought in class served as a
refresher

112
Table 7: Summary analysis of the students‟ dislike of Tophat
Q: What did you not like about the use of Tophat in enhancing your learning
experience?
Codes from the Actual quotations
data
Surface learning,  No feedback on answered questions
in-depth learning
 Tophat is superfluous, why do not use our Moodle?
Technological  The fact that the questions are done live in lectures. Sometimes
infrastructure (Wi- I did not have my tablet or laptop with me and it is annoying
Fi, devices, and to complete questions on a mobile device
classroom,  Sometimes no signal
integrated with  Sometimes it gets reset, keeps showing the same question,
Moodle) what I experienced during the questionnaire
 network problem
 I didn't like its complex procedure, everybody can't afford
android phones and this software is working on assumption
that everybody has android phones and that also in working
condition every time, please keep learning procedure simple,
so that we learn what is required not the technology for its
usage.
Time to response  Not very useful. When I can use this program, the time,
question, answer are limited. On other time, is just a picture on
my iPad and nothing in it
 Not enough time cope the word answers
 Not enough time to complete all the questions
 Access through phone which is difficult to type
Fully explanation  A bit complication to register
on how to use it  The fact that it wasn't introduced in the beginning of the
and the purpose of course, If It was introduced earlier, I feel it would have really
using it enhanced my learning experience
 Never use it before
 Open questions are not clear enough
 Not fully prepared and used when the class was done so to
motivation from the class was lacking
 I should use it more and know an effective way to get use it
 It doesn't work well on my phone and the time frame before
the next question pops in is too short have experienced
difficulties to use it and don't know where to find feedback for
my answers
 A bit hard to understand at the beginning how it works

113
Discussion
RQ1: Does student class participation improve with the use of their „disruptive‟
devices on the Tophat platform?
The data demonstrated that the students‟ participation in the module with the
use of Tophat on their digital devices increased from week to week. User
friendly interface and active learning approach resulted to increase in the
student excitement and motivation to participate in the module activities.
Variation in the types of questions used served to encourage student
participation and held their attention for a longer period of time.
On the other hand, some challenges were identified. Some of these were that it
took around three weeks to encourage all the students to register on Tophat.
Also, the students struggled with the technological infrastructure; these were in
relation to access to internet connection in that classroom (wifi), affordability of
smart mobile devices; proficiency in the use of these devices; desire of working
on a familiar existing platform (Moodle). When the students did get involved
with the activities, the data suggested that some of them (11%) were of the view
that the time given for them to respond to the questions was too short.
RQ2: What is the perceived impact of the use of TOPHAT on students‟

engagement?
Quantitative analysis showed that digital attendance register through Tophat

improved students‟ classroom attendance. The quick test at the end of each
session gives students a clear idea of the expected learning outcome and the
results informs them of their sense of mastery of the topic covered. The
availability of the revision questions with correct answer also allows students to
reinforce their learning, conduct constructive revision and further study.
Furthermore, the use of tournament fosters students‟ behavioral engagement
and affective engagement as competition with peers was found to increase
students‟ enthusiasm and motivation towards the module. Discussion forum
generated learning communities, in which students interacted with lecturers and
peers easily through their digital devices. This close interaction between lecturer
and students on the platform was found to improve students‟ critical thinking,
knowledge acquisition and overall learning experiences. Most importantly,
Tophat provide a platform that allows students who are shy and deliberate to
express themselves. Such anonymous voting and discussion system maximize
students‟ psychological adjustment and knowledge attainment.
RQ3: Does Tophat increase the amount of formative feedback received by

students?
Both quantitative and qualitative results showed that the use of Tophat as part
of the module plays an important role in providing instant and clear feedback.
Easy access to the feedback could help students develop their understanding of
the knowledge and feed-forward to complete their integrated coursework.
However, using Tophat as a means of focusing on providing prompt and
summative feedback results to a lack of detailed explanation and as such limits

114
students‟ in-depth knowledge exploration. As most of the management module

request students to think outside the box, vague feedback may restrict students‟
creativity and ability to formulate a broad understanding.
Conclusion
This study aims to determine whether the use of Tophat could enhance students‟
engagement and provision of feedback/feedforward in a higher education
module. The findings showed that the integrated features of Tophat such as
tournament and quiz increase students‟ engagement across all the age range and
gender. Prompt feedback received by the students enables the process of their
revision and ability to apply the knowledge of concept acquired (feedforward).
These results are important for educational practitioners as there is university
emphasis on a move towards the application of digital technology in teaching,
learning and assessment. The impact of Tophat was found to transform
disruptive digital devices to efficient tools for pedagogical interventions.
Furthermore, in relation to the classroom application, the students‟ digital
devices were used to constructively impact the lecture sessions. They were
found to engage with the learning through the activities set up on the Tophat
platform. The implication from the findings of this research is that the use of
digital devices as an innovative tool can enhance students‟ learning experiences
by providing instant and quality feedback.
Limitation and Future work

Not all the students that participated in the study had sufficient understanding
of the use of the technological infrastructure. The Tophat system was not used
repeatedly as a means of teaching and engaging the students fully during the
lectures, but at the end of the 50minutes lectures. Students were provided with
revision questions, which they attempted after lecture session. These questions
were designed to engage them for 30 minutes and feed-forward for their
coursework. These questions could have been more comprehensive and engaged
them for a longer period of time. The research focused on one master level
module, further study can be expanded on undergraduate level.
Reference
Biggs, J. (1999) What the Student Does: teaching for enhanced learning. Higher Education
Research & Development, 18(1), 57-75.
Biggs, J. & Catherine, T. (2010). Applying constructive alignment to outcomes-based
teaching and learning. Training Material for “Quality Teaching for Learning in
Higher Education” Workshop for Master Trainers, Ministry of Higher
Education, Kuala Lumpur.
Biggs, J., & Tang, C. (2015). Constructive alignment: An outcomes-based approach to
teaching anatomy. In L. K. Chan, & W. Pawlina (Eds.), Teaching anatomy: A
practical guide (pp. 31-38). Switzerland: Springer International Publishing.
Brown, E. A., Nicholas, J., Thomas, & Lisa, Y. T. (2014). Students‟ willingness to use
response and engagement technology in the classroom. Journal of Hospitality,
Leisure, Sport & Tourism Education, 15, 80-85.

115
Bruff, D. (2009). Teaching with Classroom Response Systems: Creating Active Learning
Environments. San Francisco, CA: Jossey-Bass.
Chickering, A.W., & Gamson, Z.F. (1987). Seven principles for good practice in
undergraduate education. AAHE Bulletin, 39(7), 3-6.
Dervan, P. (2014). "Increasing in-class student engagement using Socrative (an online
Student Response System)." AISHE-J: The All Ireland Journal of Teaching and
Learning in Higher Education. 6(2), 1977-1983
Hoekstra, A. (2008).Vibrant student voices: exploring effects of the use of clickers in large
college courses. Learning, Media and Technology.33, 329–341.
Ihde, D. (1993). Postphenomenology: essays in the postmodern context. Evanston:
Northwestern University Press.
Jones, I., & Day, C. (2009). Harnessing technology: New modes of technology-enhanced
learning action research. Becta report Online available from:
<http://www.sero.co.uk/assets/capital/ht_new_modes_action_research.pdf> .
Kaleta, R. & Joosten, T. (2007). Student response systems: a University of Wisconsin
system study of clickers. EDUCAUSE Center for Applied Research: Research
Bulletin, 10 (1), 1-12.
Karakostas, A., Adam, D., Kioutsiouki, D., & Demetriadis, S. (2014, November). A pilot
study of QuizIt: The new android classroom response system. In Interactive
Mobile Communication Technologies and Learning, 2014 International Conference on
(pp. 147-151). IEEE.
Kärkkäinen, K., & Vincent-Lancrin, S. (2013). Sparking Innovation in STEM Education
with Technology and Collaboration.
Krause, K.L.; Hamish, C. (2008). Students‟ engagement in first‐year university.
Assessment & Evaluation in Higher Education, 33(5), 493-505.
Kuh, G. D. (2008). Advising for student success. In V. N. Gordon, W. R. Habley & T. J.
Grites (Eds.), Academic advising: A comprehensive handbook (2nd ed., pp. 68-
84). San Francisco, CA: Jossey-Bass.
Lane, D. R. & Michael W.S. (2001). The centrality of communication education in
classroom computer‐mediated‐communication: Toward a practical and
evaluative pedagogy. Communication Education, 50(3), 241-255.
Liburd, J. J. & Inger-Marie F. C. (2013). Using web 2.0 in higher tourism
education. Journal of Hospitality, Leisure, Sport & Tourism Education,12 (1), 99-108.
McAleese, M., et al. (2013). Report to the European Commission on Improving the quality of
teaching and learning in Europe’s higher education institutions. Luxembourg:
Publication Office of the European Union.
Wilson, L., & McManimon, S. (2014, October). Bringing Back the ICEAGE: Interactive
Cloud-based Engagement Activities Globalizing Education!. In World Conference
on E-Learning in Corporate, Government, Healthcare, and Higher Education. 2014(1),
2065-2068.
Merchant, G. (2012). Mobile practices in everyday life: Popular digital technologies and
schooling revisited. British Journal of Educational Technology, 43(5), 770-782.
Park, J. D., & Farag, J. D. (2015). Transforming the Legal Studies Classroom: Clickers and
Engagement. Journal of Legal Studies Education, 32.
Parry, D. (2011). Mobile perspectives: on teaching mobile literacy. British Journal of
Educational Technology, 43(5),770-782.
Prensky, M. (2001). Digital natives, digital immigrants part 1. On the horizon, 9(5), 1-6.
Ravishankar, J., Epps, J.; Ladouceur, F., Eaton, R., & Ambikairajah, E. (2014). Using
iPads/Tablets as a teaching tool: Strategies for an electrical engineering

116
classroom. In Teaching, Assessment and Learning, 2014 International

Conference on (pp. 246-251). IEEE.
Robinson, C. C., & Hullinger, H. (2008). New benchmarks in higher education: Student
engagement in online learning. Journal of Education for Business, 84(2), 101-109.
Sander, P., Stevenson, K., King, M., & Coates, D. (2000). University students'
expectations of teaching. Studies in Higher education, 25(3), 309-323.
Terrion, J. L., & Aceti, V. (2012). Perceptions of the effects of clicker technology on
student learning and engagement: a study of freshmen Chemistry students.
Research in Learning Technology, 20.
Woodcock, B., Middleton, A., & Nortcliffe, A. (2012). Considering the Smartphone
Learner: an investigation into student interest in the use of personal technology
to enhance their learning. Student Engagement and Experience Journal, 1(1), 1-15.

117

Vol. 11, No. 1, pp. 117-127, April 2015
Implementation of the 2006 Education Amendment Act

on Indigenous Languages in Zimbabwe: A Case of the
Shangaan Medium in Cluster 2 Primary Schools in the
Chiredzi District
Webster Kododo
Great Zimbabwe University,
Masvingo, Zimbabwe
Sparky Zanga
St. Josephs Tongoona High School,
Jerera, Masvingo, Zimbabwe
Abstract. This research investigated the extent to which the 2006

Education Amendment Act was being implemented in selected
Zimbabwean schools. The amendments were intended to usher into
education most of the formerly neglected indigenous languages with the
hope of propping up learner performance in schools. The research sought
to establish the attitudes of teachers, school heads, pupils and parents
towards the use of Shangaan as the medium of instruction in schools
where the language is commonly used as L1 in the community. Attitudes
of language users were seen to be a significant factor in education
language innovation. The research recognizes that there is a gap between
policy enactment and policy implementation. Policy implementation will
either succeed or fail depending on the attitudes of implementers and the
user community.
Keywords: Shangaan, Shona, indigenous, attitudes, user community,

Zimbabwe.
Background
Like most of the African states, Zimbabwe is a multilingual country. As such,
choice of language for use in the promotion of literacy and basic education for
citizens has been debated for quite some time. This has been due to the view that
the learner should be educated for his/her own benefit and ultimately for the
benefit of the society (McNab 1989). Therefore, in an attempt to strike a balance
between these two ends, Zimbabwe embarked on language policy innovation in
2006 (Education Amendment Act 2006). After independence in 1980, Zimbabwe
had fashioned its first Education Act in 1987 meant to address the perceived (by
some) negative dominance of the English Language where only Shona and
Ndebele (the two main indigenous languages) were allocated inferior status
© 2015 The authors and IJLTER.ORG. All rights reserved

118
with the rest of the indigenous languages having no recognised role in education
(Chimhundu 1984). Chimhundu (in Roy-Campbell & Gwete 2000; see also
Royneland 1997) sadly notes that in the 1987 Education Act, the Zimbabwean
government had failed to honour the proposal presented by the Minority
Languages Committee in 1985 that in areas where there were dominant,
specified indigenous languages, these should be taught in addition to Shona and
Ndebele. The proposal was in response to debates and experiments on
suitability of L1/L2 as medium of instruction in schools. The debate had been
exacerbated by the UNESCO (1953) claim that L1 is the best medium for infant
education.
Owing to heightened debate and accusatory complaints to the effect that poor
performance in schools were partly a result of the use of an L2 as medium of
instruction in education, the Zimbabwean government embarked on language
innovation that culminated in the 2006 Education Amendment Act. In the
amendments, a new section on language use replaced the old Section 62,
Chapter 25:4. Part of the new section reads;
1. Subject to this section, all the three main languages of Zimbabwe, namely,
Shona, Ndebele and English shall be taught on an equal time basis in all schools
up to Form Two level,
2. In areas where indigenous languages other than those mentioned in Section (1)
are spoken, the Minister may authorise the teaching of such languages in schools
in addition to those specified in Section (1),
3. The Minister may authorize the teaching of foreign languages in schools,
4. Prior to Form One, any of the languages referred to in Sections (1) & (2) may be
used as the medium of instruction, depending upon which language is more
commonly spoken or better understood by the pupils.
One can note that pronouncing policy in education is one thing but
implementing that policy is another. In spite of good intentions in policy
formulation, there can be various factors that may affect the implementation of
the said policy. One of such factors is language attitudes. As such, these
researchers got inspired to investigate attitudes of language users towards the
implementation of the language policy amendments in the sampled school
communities. Attitudes have the capacity to affect policy implementation
(Kadodo et al 2012). Research elsewhere shows that language choice for
individuals tends to be influenced by culture, politics and economics (Diamond
1993). Language attitudes raises the question of what language users prefer
when confronted by an array of competing interests ranging from social to
economic? Choices are arrived at after serious balancing acts for individual users
be it learners, parents, teachers, school managers or education managers. The
dilemma implied here makes it necessary to investigate and ascertain language
users‟ choices and the reasons they attach for such language choices. The
implementation of the language policy as directed by the 2006 Education
Amendment Act in Zimbabwe is subject to users‟ attitudes.
Research Question
What are the language attitudes of pupils, parents, teachers and school heads at

119
Nandi, Mwenje and Nyahanga primary schools towards the use of Shangaan as
medium of instruction as provided for in the 2006 Language Amendment Act?
Conceptual framework
1. Language and education
Language is one of the most essential asserts gifted to humans, indeed a miracle
that defines their existence (Aitchinson 2008). Being a key to communication,
language can bring human beings together as much as it can set them apart. One
primary cause of division among communities is differentiation of languages
roles in education. Defining language, Finocchiaro (in Brown 1987:5) says that it
is “a system of arbitrary … symbols which permit …people in a given culture, or
other people who have learnt the system of that culture, to communicate or to
interact”. Thus, apart from both the vocal and visual, “a language has a dual
character… [as] a means of communication and a carrier of culture” (Roy-
Campbell & Gwete 2000:7). Durckeim (in Blackledge & Hunt 1985) defines
education as an influence exercised by a generation on those not yet ready for
social life [or those who wield power over those without]. In other words,
communicating through language is one of the channels through which
particular norms and values of a society can be transmitted from one generation
to the next. Undermining the language of a people, therefore, devalues their
dignity and leads to, unfortunately, a painfully slow death of such languages
(Open Space 2008; waThiongo 1993). As McNab (1989:11) notes, “education is
perceived as the terrain for excellence where language related inequalities and
discrimination are manifested”. In other words, education must navigate
through this terrain of language use to ensure that all cultural groups are catered
for. Language is one of the significant factors in education that may lead to
either the educability or miss-educability of learners. It can have a telling effect
on the achievement of learners defining the quality of (or lack of) learning and
teaching in educational institutions. It is on this basis that some researchers
advocate for mother-tongue education seen as more effective for mastery of
educational concepts (See Open Space 2008; Adegbija 1994; Bamgbose 1991;
Mupande 2006; Brock-Utne 1993).
This research argues that every language is (or can be) an effective and efficient
tool for its users in education so long those users have firm control of that
language. Firm control of language implies speaker capacity in both the
linguistic and social nuances of the given language. The negation and relegation
of a language from an education system in a country is tantamount to excluding
the speaker community from national and developmental activities. For this
reason, the United Nations propagated the universal declaration of linguistic
rights (Open Space 2008) as paragon for the existence of even the so called
minority communities. Language is a kaleidoscope that unlocks various
meanings of existence for its users. Where learners, participants or community
members are in firm control of the means of engagement (language as one key),
they are able to display their abilities and contributions. Undermining a people‟s
language is equally undermining their confidence, ability and contributions. Use
of unfamiliar language leads to bad results (Prah 2000). This research was
guided by these beliefs regarding the intricate relationship between language

120
and education, thus seeking to find out to what extent the 2006 Language
Amendment Act was being implemented in selected schools. One can note that
language users‟ attitudes are key to the implementation of language innovation
(Kadodo et al 2012). This research sought to examine language attitudes of
pupils, parents, teachers and school heads at Nandi, Mwenje and Nyahanga
primary schools towards the use of Shangaan as medium of instruction as stated
in the 2006 Language Amendment Act.
2. Attitudes to policy
An attitude is “an organized predisposition to think, feel, perceive, and behave
toward referent or cognitive object … an enduring structure of beliefs that
predisposes the individual to behave selectively toward attitude referent”
(Taylor et al., 1997: 130; Ajzen, 1988:4; Kerlinger, 1986: 453; Kosslyn &
Rosenberg, 2006: 738). In fact, attitudes are the “very general evaluations that
people hold of themselves, other people, objects and issues” (Tesser, 1995: 196).
Beyond the basic functions of language, the roles that we subtly assign to
various languages at our disposal reflect what attitudes we hold of each of these
languages (Adler & Rodman 2001). The undertones in this move are the transfer,
in dosages, of the said attitudes (positive or negative) to the unsuspecting
language users. This is done through subtle insinuations that learning through
language A leads to employment and better life whilst learning through
language B may lead to lack of this and that. This method is borrowed from
most colonised worlds where it has been successfully used to shape oppressed
people‟s preferences. What we think about a language (cognitive attitude), what
we feel about it (affective attitude) and what we actually do with that language
(behavioural attitude) (Taylor et al 1997; Child 1993) are clear attitudinal
demonstrations of the values we attach to each of the languages that are at our
disposal. Consequently, the said attitudes will shape how we use language in
the various activities of our lives. In the same manner, this also has visible
influence regarding whether language policy innovation will or will not be
successfully implemented.
These researchers note that issues of language use are always bounded in power
struggles as demonstrated during the colonial processes in various parts of the
world. In the then Rhodesia (Zimbabwe now), there was incessant tutelage that
English was „the language‟ whilst the local ones were of no consequence. As
Diamonds (1993) at http://www.pre.org (accessed 20/08/2011) notes, “when a
people have been told for many years that their cultures [so their languages too –
our emphasis] are worthless, they come to believe it”. Consequently, this created
positive attitudes (in most colonised people) towards colonisers‟ languages
whilst conversely creating negative ones for indigenous languages that had been
disempowered by lack of economic rewards for them (Kadodo et al 2012).
Ironically, at independence, there has been unequal empowerment of
indigenous languages in Zimbabwe. The 1987 Education Act tended to raise
some languages (notably Shona and Ndebele) to national languages at the
exclusion of the rest of other local languages. Regrettable to say is that the same
power struggles hitherto stated regarding the colonial times came to haunt the
language use in an independent state. Linguistic imperialism, hence likewise

121
cultural oppression, can be practiced even in democracies with the pretext of

national unity which can be used to impose some languages over others. This, in
our view, is no different from the ways used by former colonisers. Clearly, this
explains how Shangaan, like other indigenous languages, was left in the cold for
a number of years in an independent Zimbabwe without any meaningful role in
education. It was not until the „noise‟ was getting louder and louder for
recognition of other indigenous languages that the 2006 Education Amendment
Act was gazetted. However, it is one thing to gazette policy but quite another to
implement it. It was the intention of this research to establish the extent to which
the Shangaan medium was being used (2006 Education Amendment Act) in
selected schools in a Shangaan speaking community given the long history of
linguistic and cultural oppression both in the pre and post independent
Zimbabwe. The question begging answer is, „what measures were put in place to
incentivise its use for users to have embarked on counter-attitudinal processes?‟
Factors that may raise or impede language policy implementation

Successful implementation of language policy is dependent on various factors
which explain why some policies fail but others succeeding. Policy without
stringent implementation measures has slim chances of succeeding. In short, for
policy to succeed, the operating environment must be permissive and
supportive creating a „want‟ in the users to see that policy working.
One key factor for language policy to work is the job market that should act as
an incentive for the said policy to be favourable to the user community (Open
Space 2008). Unless there are promises for the products of that policy being
employable, then that policy may not be supported by the user community.
Whatever language is economically incentivised will tend to attract positive
attitudes from the user community. In close proximity is the role of media (both
print and electronic) which can play a supportive role or become the devil‟s
advocate (Ndinde & Kadodo 2014). Media is key in shaping people‟s attitudes
by the way how it will campaign for or against a view or language. For instance,
the nature of programmes and the language employed to air them could have an
influence on people‟s language preferences. The points noted above are, in turn,
dependent on the commitment of the leadership of a country (Roy-Campbedll &
Gwete 2000). A language policy gazetted by leadership not politic enough will
hardly succeed. The policy itself must be outlined in succinct language leaving
no room for speculation or debate as to the meaning of statements. In other
words, tentative and speculative language must be avoided so that each
instruction is understood for what it is. An astute leadership will first ensure
that relevant teaching staff and appropriate teaching materials are in place prior
to gazetting education policy. It is no point pronouncing an education policy
when appropriate preparations have not been done because that is tantamount
to pronouncing its failure before implementation starts. Banda‟s 1968 Chewa
only medium in Malawi and Ratsiraka‟s 1972 Malagachisation in Madagasca are
cases in point. In short, it is important to look at both facilitating and debilitating
factors within the operating environment so as to take corrective measures for
policy implementation to succeed.

122
Methodology
The research was guided by the descriptive survey design in an effort to
understand the attitudes of pupils, teachers, headmasters and parents at the
three selected schools towards implementation of the 2006 language innovation.
The descriptive survey was seen as suitable for measuring user‟s attitudes
(Chikoko & Mhloyi 1985) regarding the implementation of the 2006 language
innovation. This research employed mixed methods (Maree 2010) where both
qualitative and quantitative data were collected. Use of the mixed methods
helped in the triangulation to increase reliability. Questionnaire, interview and
observation data collection methods were employed. These collection methods
were chosen for their versatility in data collection of users‟ language preferences.
The questionnaire is a self-report instrument that guarantees anonymity of
research participants (Best & Khan 1993) thus increasing chances for participants
to reveal their deep-seated feelings regarding attitude referent. On the other
hand, face-to-face interviews increased rapport with participants allowing
elicitation (Brenner 2006) of information. These researchers also felt they needed
to observe a couple of lessons at the research site to ascertain visible reactions of
learners when what language (or combination of languages) was/were used.
This was a useful method for learning learner behaviours (Sapsford & Jupp
2006) that betrayed learners‟ language preferences.
Selection of participants
Purposive and random sampling techniques were used in sample selection.
Whereas the three school heads (of the three schools) were purposively selected,
ninety pupils, thirty teachers (from the three schools) and ninety parents (of
learners at the three schools) were randomly selected using the lottery method.
Data collection procedures

Four sets of short questionnaires were designed (heads‟, teachers‟, pupils‟ &
parents‟) and these were distributed and collected by the researchers.
Participants were allowed ample time to complete these each on his/her own.
Interviews were also organised and carried out by these researchers (the three
school heads, ten teachers and ten parents) whereupon the data was recorded in
field notes. Six lesson observations were done by the researchers to get a close
feel of what was happening at classroom level. Data was also collected in form
of field notes on what transpired. The data collected was organised into
contingent tables for analysis with interview data thematically factored into the
discussion.

123
Results and discussion
Table 1: Participants’ language choices
Question Language Heads Teachers Pupils Parents Summaries of

theme choice reasons
Preferred English 3(100%) 18(60%) 63(70%) 52(58%) -pass exams;
medium -most books
for education are in English
Shona - 10(33%) 15(17%) 18(20%) -cannot

understand
English
Shangaan - 2(7%) 12(13%) 20(22%) -understand

Shangaan
Best medium Shona - 6(20%) 15(17%) 12(13%) -teachers &
for learners
education understand it
English 3(100%) 14(47%) 63(70%) 70(78%) -language

used in exams
& for
employment
Shangaan - 10(33%) 12(13%) 8(9%) -we can

understand it
Learners‟ Shangaan 16(18%)
favourite English 45(50%)
language Shona 29(32%)
Language used English 8(27%) -it is the
by language of
teachers to examinations
explain
concepts Shangaan 6(20%) -pupils offer
correct
answers
Shona 16(53%) -pupils

participate
Learners‟ L1 English 1(1%)
Shona 40(44%)
Shangaan 49(55%)
The „global‟ picture of results from the presented data in Table 1 above and
Table 2 below reveal the respondents‟ attitudes to the three languages that are at
their disposal. In Table 1, 100% of participating school heads, 60% of teachers,
70% of learners and 58% of parents preferred that English continue to be the

124
medium of instruction in their schools. Key to their decisions as revealed in their

reasons are issues to do with the current situation where examinations for all
content subjects are in English as well as employment opportunities that still use
the English Language as gate-keeper. After all, English remains the granary for
key information and technology that teachers and learners require for their
teaching/learning activities in content subjects. For that reason, it becomes
logical for them to prefer a language medium that they perceive as giving
learners better life opportunities and that language in their view is English. In
spite the fact that 55% were L1 Shangaan and 44% were Shona L1 speakers, the
majority of learners were very pragmatic thus, preferred English because in their
view it promises employment opportunities.
There were other issues that were cited by teachers during interviews where
they felt that Shangaan as an education medium did not have the terminology
range to adequately convey the scientific and commercial „worlds‟. In
conjunction with this perceived problem is the non-availability of reading
materials even for Shangaan itself as a subject, let alone reading materials
written in Shangaan for content areas. These researchers, however, note that
languages can grow depending on what range of uses we assign them. Likewise,
Shangaan can develop to cover the range that the users so desire it to cover so
long we avoid the pitfalls of purism where we want languages to be what they
were centuries ago. Regarding materials and manpower development, these are
issues that can be resolved say in phases provided the policy-makers,
implementers and user communities are all agreed on the necessity for such
move. Simply put, policy development should not be conceived in the top-down
form which, more than often, leads to tissue rejection (Obanya 1987). There is
need for extensive positive consultations.
As implied above, respondents also noted that none of the teachers on the
ground at the time of data gathering was ever trained to teach in Shangaan. This
was seen to be a handicap to the implementation of the 2006 policy. This,
coupled with the reality of lack of materials on the ground, questions the
sincerity of policy-makers in gazetting the 2006 Education Amendment Act. The
ground surely was not „flattened‟ for indigenous languages to be used as media
of instruction in schools. Scales have always, and sadly remains so, in favour of
the English Language. There has not been any attempt to incentivise the
indigenous languages as a measure for users to commit counter-attitudinal
processes and develop positive attitudes towards them.
Table 2 below give results of observations made by these researchers when they
attended some of the lessons in the research schools.

125
Table 2: Observed effects of language use in 6 lessons
Areas & behaviours noted

Les
son
No
Subject Language used

Code Pupil’s visual Participation
taught switching expression level
/mixing
1 Religious & More English Low Look-warm Moderate
Moral Ed. than Shona mostly
2 Mathematics English None Indifferent Very low
3 Mathematics More Shona & High Enthusiastic & Very high for
Shangaan than confident Shangaan, high
English for Shona &
lower for English
4 Social Studies English but High Enthusiastic & Very high in
with more excited either Shona or
Shona & Shangaan than
Shangaan English
5 Mathematics English & a Low Look-warm Low
little bit of mostly
Shona
6 Environmenta English & a Moderate Look-warm Moderate
l Science little bit of
Shangaan
Lesson observations, as shown in Table 2, allowed these researchers a small

window to observe language use and learner responsiveness in language
ecology. The researchers note that overall teachers maintained English Language
as the focal but with varying degrees of permissibility of Shangaan and Shona.
Apart from Lesson 2 as shown in the table where the teacher enforced all
English, the other 5 lessons had varying degrees of code switching/mixing. The
pattern emerging from the lessons seemed to drive to a conclusion that the more
permissible the teacher was to code switching/mixing (Lessons 3 & 4 in Table 2)
the more the classroom activities were liberated allowing learner hype and
participation (Freire 1972). As Freire notes, such environment leads to the
traditional teachers‟ and learners‟ roles mutating allowing the teacher and
learner each to be both an educator and learner at the same time. This would
create classroom partnerships where learners are carefully and subtly moulded
into mature critical learners. On the contrary, the teaching/learning
environment in Lesson 2 presented a contrast to Lessons 3 and 4. It is not
unreasonable, in the case of Lesson 2, to be worried that the teacher may be
tempted to „tell‟ the learners his/her own knowledge than creating enabling
environment for learners to engage issues leading to their own learning. Else

126
how are these learners engaging issues in this handicapped and ominously quiet
environment? Based on the observations above, we conclude that the question of
what language of instruction in education matters.
Looking at the sets of findings in this research one notices some internal
contradictions. Significant to note is that leaners seem to enjoy, and possibly
learn more meaningfully when they are in control of the language of education.
This should, supposedly, see them opt for their L1 as medium of instruction.
However, when learners consciously make a choice they still opt for a language
they may not have efficiency in. The matrix in the maze is that their conscious
choice of language is driven by what they yen for in their future lives, more of a
wish-driven choice. This is what holds sway to their language attitudes.
Unfortunately, a number of education systems today have become so
mechanistic and examination-driven that teachers can just coach learners in their
quiet environments to pass the national examinations in spite of their low
proficiency. Is this not possibly the reason why the industry sector is perennially
complaining of raw graduates from some of our institutions?
Conclusions
Based on the findings of this research one concludes that choice of language of
education is not always a rational process but is more often emotional.
Notwithstanding that learners may not be efficient in a language they may still
opt for that language as medium of instruction owing to their perceived life
opportunities. For that reason, before legislating any language policy there is
need to ascertain users‟ language attitudes. If these are not in tandem with the
proposed language we would rather incentivise the intended language for users
and implementers to prefer such language. This research also concludes that
contingent planning should precede gazetting of any education policy. This is
possible in situations where there are open and well-intended consultations. The
fact that this was not meaningfully done as precursor to gazetting of the 2006
Education Amendment Act in Zimbabwe, the innovation has not been embraced
by the user community and therefore has not succeeded.
Declaration
The researchers wish to declare that there was no research grant attached to this
research by any organization.
References
Adegbija, E. (1994). Langauage Attitudes in Sub-Saharan Africa: A Sociolinguistic
Overview. Clevedon: Multilingual Matters.
Adler, R.B. & Rodman, C. (2001). Communication. London: Routledge.
Aitchinson, J. (2008). The Articulate Mammal: An Intorduction to Psycholinguistics 5e
(5th ed.). London: Routledge.
Ajzen, I. (1988). Attitudes, Personality and Behaviour. Milton Keyness: OUP.
Bamgbose, A. (1991). Language and the Nation. The Language Question in Sub-Saharan
Africa. Edinburgh: EUP.
Best, J.W. & Khan, J.V. (1993). Research in Education (7 th ed). Boston: Allen & Bacon.
Blackledge, D. & Hunt, B. (1985). Sociological Interpretations of Education. London:
Groomhelm.
Brenner, M.E. (2006). “Interviewing in Educational Research” in American Educational

127
Association, Handbook of Complementary Methods in Education, pp. 357-

370.
Brock-Utne, B. (1993). Education in Africa, Vol. 1 Report No. 3. Stockholm: Bindern.
Brown, H.D. (1987). Principles of Language Learning and Teaching (3 rd ed.). Englewood
Cliffs: Prentice Hall.
Chikoko & Mhloyi (1985) at htt://www.tandfonline.com (accessed 06/10/11)
Child, D. (1993). Psychology and the Teacher. London: Holt Rinehart.
Chimhundu, H. (1984). “Towards a Policy on the Teaching of African Languages in
Zimbabwe” Paper Presented at the Linguistic Conference of SADAC
Universities, Malawi, 19-21 November 1984.
Diamond (1993) at http://www.prel.org (accessed 20/08/11)
Freire, P. (1972). The Pedagogy of the Oppressed. New York: Herder & Herder.
Kadodo, W., Kadodo. M., Bhala, T. & Bhebe, C. (2012). “The influence of teachers‟ and
students‟ attitudes towards the use of Shona as medium of instruction in
secondary schools” in International Journal of English and Literature,
February 2012, 3(2), pp32-39.
Kerlinger, F.N. (1986). Foundations of Educational Research (3 rd ed.). New York: Holt,
Rinehart & Winston.
Kosslyn, S.M. & Rosenberg, R.S. (2006). Psychology in Context (9th ed.). Delhi: Pearson
Education Inc.
Maree, K. (ed.) (2010). First steps in research. Pretoria: Van Schaik Publishers.
Mupande, I. (2006). Silent Voices: Indigenous Languages in Zimbabwe. Harare: Weaver
Press.
NcNab, C. (1989). Language Policy and Language Practice: Implications and Dilemmas
in Ethiopian Education. Stockholm: University of Stockholm.
Ndinde, S. & Kadodo, W. (2014). “The role of community-based information centres in
development: Lessons for rural Zimbabwe” in International Journal of
Learning, Teaching and Educational Research, 2(1), pp. 44-53, February
2014.
Obanya, P. (1987). A Curriculum Development‟s Manual. Switzerland: WCOTP.
Open Space (2000). Language Inclusion Participation and Expression, 2(1), November
2008, Johannesburg: Open Society Institute for Southern Africa.
Prah, K.K. (ed.) (2000). Between Distinction and Extinction: The Harmonization and
Standardization of African Languages. Cape Town: Centre for Advanced
Studies of African Societies (CASAS).
Roy-Campbell, Z.M. & Gwete, W.B. (2000). Language Policy and Planning. Harare:
University of Zimbabwe.
Royneland, U. (ed.) (1997). Language Contacts and Language Conflict. Voldo College,
The Iva Aasen Institute: University of Oslo.
Sapsford, R. & Jupp, V. (eds.) (2006). Data Collection and Analysis. Los Angels: Sage.
Taylor, S.E., Peplau, L.A. & Sears, D.D. (1997) Social Psychology (9 th ed.). New Jersey:
Prentice Hall.
Tesser, A. (1995). Advanced Social Psychology. New York: McGraw-Hill Inc.
The 1987 Education Act. The Government of Zimbabwe.
The 2006 Education Amendment Act. The Government of Zimbabwe
UNESCO (1953). The Use of Vernacular Languages in Education. Paris: UNESCO.
waThiongo, N. (1993). Decolonizing the Mind. Harare: ZPH.

128

Vol. 11, No. 1, pp. 128-140, April 2015
The Concept of In Situ Lecturing
Joachim R. R. Ritter and Ellen Gottschämmer

Karlsruhe Institute of Technology, Geophysical Institute
Karlsruhe, Germany
Abstract. Traditional teaching at university level takes place mainly in

lecture rooms without any direct linkage to the real subject. Thus even
during well taught lectures the human, in our case students’, multi-
sense receptivity is stimulated only in a limited way. This means that
just a part of the actual subject matter can be transferred. Therefore,
students will not learn to observe the whole range of the circumstances
and environmental parameters involved in a specific subject. This
problem arises especially in natural sciences and, partly, engineering
teaching in which the environmental setting is often a key to
successfully understand complex processes, proportions and scales as
well as human (counter) actions. In our concept “in situ lecturing” we
teach at the place which is being studied, hence “in situ”. In situ
lecturing is a valuable pedagogical concept to develop student’s
understanding of basic concepts, to enable them to transfer concepts and
theory to local conditions, to train practical skills and to promote a
comprehensive understanding of processes. This approach is achieved
by an integrative combination of pre-courses and practicals in the
classroom, followed by in situ lectures, practicals and seminars as well
as a final reporting. Examples are presented from geophysics but these
may be transferred to many other disciplines.
Keywords: university teaching; field work, teaching method.
Introduction
The curricula of most degree programmes in natural sciences and engineering
predominantly involve classroom lectures, practicals for solving exercises,
seminars with student presentations and, rarely, excursions or actual fieldwork.
Classroom lectures, practicals and seminars account for the by far most part of
the higher education. Classroom lecturing is vital for setting a solid base for
primary skills such as understanding of theoretical concepts (e.g. mathematics),
first principles (e.g. basic physics), taxonomy and laboratory working
techniques. However, classroom lectures are mostly passive events for students
with a one way communication, although there are many concepts to stimulate
the audience (e.g. Laws, 1991, Powell, 2003, Reiber, 2006). There are also many
applied courses which concentrate on specific themes, e.g. monitoring of
environmental parameters, assessment of hazardous natural processes,
construction of buildings, instruments and machines etc. Teaching in some

129
subjects requires indispensably out of classroom instruction, e.g., geosciences

(for a review see Butler, 2008 and references therein). In a subsequent
professional career such activities require skills which cannot be trained
adequately by only classroom lecturing (Lonergan & Andresen, 1988, Kastens et
al., 2009).
During lectures the students should learn and understand the necessary
theoretical and conceptual basis of a subject. Yet, when examples are displayed
in the classroom, students experience only a limited aspect of a study or
situation which is presented in a slide or movie. Often a detailed and
comprehensive description of a situation or object and its environmental
circumstances and parameters is missing (even the lecturer may not know under
which circumstances a photo or film was taken). The true scale of objects is
often not clear to students, if they do not experience the real objects themselves.
In addition, visual material for class room teaching is often biased by well-
chosen factors or circumstances: optimum light conditions to illuminate an
object, undisturbed environmental conditions etc.
Students should participate actively during practicals where they present their
solutions to questions and exercises distributed by the lecturer and where they
apply the principles and concepts which were taught during the lectures. Such
solutions are mainly developed outside the classroom, sometimes within a small
group of students. In the majority of practicals, the students are given real data
e.g. measurement curves such as seismograms, rock samples, electronic modules
etc. to work on. Such material is often carefully selected to avoid complications
due to noisy data, unclear interpretational options etc. The problem of scale
arises also in many subjects, e.g. a complete bridge cannot be examined in the
lab during an engineering course.
Seminars are conducted in such a way that students learn to present, discuss and
think about case studies from articles in the literature (which they read typically
in the late evening just before the seminar is be held). Again, links to a realistic
situation are given in parts only and experience of the real world is missed.
During many excursions the students are carried from one point of interest to
the next one without any active participation. Sometimes even the connection
between basic theoretical principles and objects of an excursion are poorly
explained and, in this way, the link between them remains obscure. Students
may also be confused if there are different lecturers in the classroom and during
the excursion who use different ways and concepts to describe the same object
or process. Only rarely is actual fieldwork is done by students (e.g. mapping,
collecting, measuring in the field or assembling of an instrument). However,
well prepared lecturing outside the classroom trains students for their
professional career and widens their perspective (Hursh & Borzak, 1979) due to
the inherent interdisciplinary nature of outside teaching (Claiborne et al., 2014).
In the following section we propose ways to better link theoretical concepts,

passive and active learning and real situations by adding to the theory given in

130
lectures to include practicals and seminars where the subject is actually based.
This concept can be compared to the idea of learning a language in the country
where it is actually spoken. In that case, the language will not only be adopted
during the language classes, but also during the rest of the day. In a similar
way, during an in situ class, students are confronted with the matter all day
long. This stimulates the students to think about the subject in more detail.
Such an approach leads to questions (and answers) which would not have been
asked (and answered) during a typical classroom lecture. Thus, students get a
much deeper and more comprehensive understanding of the subject. An
example can be seen in Figure 1, where students give in situ a seminar on the
flanks of Stromboli volcano, Italy. The background of our examples is described
in Box 1.
Figure 1: Student giving an in situ presentation (step E) about volcanic processes at

the slope of Stromboli volcano, Italy. The used screen is the backside of a poster
which was displayed earlier for another student presentation. A notebook and a
beamer with battery power are utilised for the projection in the open field; volcanic
ash plumes from small eruptions could be observed from this place during a pleasant
evening lecture (left picture courtesy of J. Käufl).
Concept and Examples

The term “in situ” is the Latin expression for “in the original place” or short “in
place”. We use in situ to express that teaching is done outside the classroom at a
place which is directly linked with the subject matter. Therefore, in situ can be
related to a lecture, practicals, exercises or a seminar with student presentations.
Our experience from in situ lecturing at bachelor’s and master’s degree level
shows that a balanced combination of lecture presentations, practicals with
exercises and seminar-type elements is a meaningful way to educate students in
many disciplines. Of course, the subject or theme must be suitable for successful
in situ lecturing. Geosciences are obviously very suitable for in situ lecturing,
because the study objects such as volcanoes, mines, observatories etc. are
obvious targets which should be studied outside the classroom (Thompson,
1982, Butler, 2008). Education in many other disciplines may also benefit from in
situ lecturing, e.g. biology (zoos, wilderness), physics (particle accelerators),
history (historic sites) and many others.

131
Table 1: Overview of the concept of in situ lecturing with its main elements A - F.
Step Type Main Functions

A preparatory  introduction of first principles
lecture  theoretical basis
 outline and organisation of in situ part
B preparatory  reading of relevant literature
student work  preparation of presentation, e.g. for classroom
presentation or in situ seminars
 preparation of handouts or lecture notes for
fellow students
 organisation of in situ part
C in situ lecture  repetition of first principles and basic concepts
 presentation of subject matter with direct link to
the subject
D in situ  solving exercises directly linked with the subject
practical  observation, identification and description of
elements related to the real subject
 onsite training
E in situ seminar  presentations directly related with the subject,
in the field or during evening seminars
F post-trip students:
documentation  writing a final report
 reflecting on the subject and preparation for
exam
lecturers:
 reflecting on the course for further improvement
 reflecting on students’ comments
Figure 2: Students discussing and working on a short-time exercise during a

preparatory lecture (step A). Such exercises increase their attention to the lecture
and help the students immediately reflect the lecture subject.

132
Our pedagogical concept for integrated in situ teaching includes the following
elements or steps (Table 1):
A) preparatory classroom lectures on basic concepts and theoretical work,
B) preparatory student work which includes the repetition of the basic concepts,
reading of relevant literature, preparation of classroom presentations, in
situ seminars and lectures, as well as getting familiar with the specifics
(e.g., geology, geophysics, eruption style, vegetation, accessibility) of the
location of the field study,
C) in situ lectures in order to reinforce the theoretical background and to transfer
the basic principles to real situations,
D) in situ practicals to actively apply the basic theory learnt during the lectures,
within the actual situation,
E) in situ seminars with presentations by the students to explain observations in
the field, review the background, explain the local history or add details
as well as discuss uncertainties and other problems related to the present
in situ state of the subject, and
F) post-trip documentation.
The succession A) – F) is arranged in order to achieve a high transfer of

knowledge and skills to the students. Especially the active parts foster the
training of competences with a special focus on real professional situations and
working environments. Details and examples for steps A) to F) include:
A) Preparatory classroom lecturing: higher education must include instruction
in the solid theoretical background of the subject matter. Thus our geophysics
teaching includes the presentation of basic physical concepts and equations.
These theoretical parts are taught best in a traditional class course where one can
concentrate on physics theory and the related mathematics but it should also
include practical applications, case studies and quantitative parameter
descriptions. For instance during a course on induced seismicity (see Box 1), we
explained the theory of tectonic stress with applications to Earth materials and to
earthquake fault zones as well as the influence of stress changes generated by
humans. In order to overcome their passive status, we introduce student
activities such as brief exercises (Figure 2) or pop quizzes or provocative
questions which they solve in small teams. By these means, the students are
continually motivated. (Powell, 2003, Handelsman et al., 2004). The preparatory
classroom lecturing includes also a first introduction to the sites of the in situ
lecturing as well as organisational issues.
Figure 3: Students giving a poster presentation on borehole logging methods during

preparatory student work (step B). Each student had to prepare and explain a part of
the presentation to get familiar with the subject matter discussed in the field.

133
Figure 4: Local field engineer (left) during an in situ lecture (step C) in a quarry with
volcanic rocks, Vogelsberg region, Germany.
B) Preparatory student work: active participation of students is a key element

for understanding the subject matter. For the preparation of the in situ part we
place a special emphasis on reading the relevant literature. We select mainly
articles in scientific journals to force the students to read original science reports
and get familiar with scientific approaches and the appropriate language style.
Reading well-selected journal articles helps students deepen their knowledge
and improve their understanding of the theoretical concepts. Students also learn
typical linguistic expressions and the appropriate use of terms used in their
fields. Besides, our German students are thus forced to train their skills in
English, the main language in science. Another focus is on the region of the in
situ part – this means getting familiar with the geography, regional and local
geologic framework as well as geophysical field studies and models in our case.
For an in situ lecture course on the volcanoes in Southern Italy (see Box 1), each
student had to write a chapter of the excursion guide book based on a literature
study and prepare a presentation to be given later in situ (Figures 1 & 8). For a
course on the volcanic complex of Vogelsberg volcano, Germany, the students
had to prepare both, a short chapter for the lecture notes, and a 10-15 minute-
long poster presentation about methods used in borehole logging. The posters
were presented in the preparatory classroom lecture before the in situ part
(Figure 3). Additionally, the students prepared a presentation given at an in situ
seminar, either in the field, or during an evening seminar held in the hotel
(Figures 9-10).
C) In situ lectures: in situ means that university lecturers or external experts

present the learning matter at the specific place where the subject is relevant. In
situ lectures can be done in several ways and at different kinds of locations
(Figures 4-6). Mobile equipment such as a notebook and a beamer can be used
in a hotel, on a ship or even in the open field (Figure 1). Concepts presented

134
earlier during the preparatory classroom lecturing can be repeated and directly
linked with the real world. Of course the main point should be the direct
inclusion of the local phenomena (e.g. a geoscientific site, major machinery or
instrument, a building…). A classic example is the explanation of rock types
and the genesis of these rocks and their usage in a quarry (Figure 4). One may
also visit a seismic reflection field survey and explain the implementation of
recording arrays, the data acquisition and preliminary data interpretation. Such
a lecture can be given also by an external expert, e.g. a field engineer of a
company, a local geology expert etc. Generally, parts of the in situ lecturing can
be done by local experts who have specialised local expertise (Figure 5), specific
experience with a production machine or research infrastructure (Figure 6) etc.
It is useful to explain to such external lecturers in advance what is the aim of the
lecture, what is the state of knowledge of the students and what the students
could do as possible practicals.
Figure 5: In situ lecture (step C) at about 700 m depth inside a potash mine.
A group of geophysics students is instructed by a local geologist.
D) In situ practicals: during in situ practicals students have the opportunity to

experience things that cannot be done in a class room (Figure 7). This may be
the exploration of the underground using geophysical equipment, the handling
of real production machinery or a realistic field study e.g. for hazard assessment.
Especially in geoscience training, it is important to learn techniques for
successful fieldwork (Kastens et al., 2009). Compared to laboratory studies, in
the field, one has to cope with completely different working conditions: rough,
partly extreme weather conditions, varying light conditions (which can cause a
different appearance of rocks or minerals for example), identification of often
hidden objects (e.g. a mineral vein underneath a vegetation cover), systematic
spatial mapping or collection of data or a working environment with an
unknown cultural and language background (e.g. archaeological work in a
remote area in the Himalayas). It is important that students learn such realistic
situations in situ and gather first experience for a later professional career
(Kastens et al., 2009).

135
E) In situ seminars: students present a seminar talk in situ while incorporating

the study object directly in their presentation. As media posters or computer
presentations can be taken along and displayed at the hotel or in the open field
(Figures 8-10). Subsequent to the preparation of such a presentation, the
students are well prepared for the in situ subject, because they are forced to
actually read the required literature and to think about the main relevant topics
for the presentation. The shorter the allowed time for the presentation, the
higher the pressure for focussing on the main facts and background information.
This is a valuable exercise for students to learn to concentrate on the basic
principles.
Figure 6: In situ lecture (step C) in the museum of historical seismometers

of the University of Strasbourg, France. The main principles of seismometry
are explained to the students and the operating mode of the mechanical
components can be demonstrated at real objects.
F) Post-trip documentation: After the in situ phase, students should write up

what they saw, experienced and learnt in a final report. Due to our experience,
we recommend giving the students a clear limit for the length of the final report
(the shorter the better: students learn to concentrate on relevant parts and
lecturers are protected from reading numerous endless essays) and we tell them
which main points should be covered. Individual final reports may be
combined to manuscript-like lecture notes which cover different aspects of the
subject matter and may serve e.g. as preparation material for an examination or
for a future in situ course.

136
Box 1: Case Studies

Our in situ lecturing experience is based on the following geophysical topics
and examples:
a: (Geo-)Physical Volcanology and Hazard Assessment: volcanic eruptions are a

major hazard to society in many regions of our planet. We travelled to the
volcanic islands of Lipari, Vulcano and Stromboli in the Tyrrhenian Sea and
to Mt. Vesuvius (Italy). In these places we studied volcanic edifices, volcanic
rocks, volcanic activity, monitoring concepts and monitoring instruments.
We determined and estimated hazards related with the volcanic activity
including the identification of vulnerabilities and the estimation of values at
risks.
b: Geophysical Exploration of Volcanic Fields: deep geophysical exploration

(down to several hundred kilometres depth) of volcanic regions is necessary
to understand the origin and history of magmatic processes inside the Earth
and for understanding future volcanic activity. We went to the Eifel volcanic
field in Germany to explain such approaches. Geophysical models were
presented and discussed for a thorough understanding of the deep magmatic
processes underneath the Eifel. In addition many surface expressions of
volcanology can be studied including rock types, gas emissions or current
vulnerable infrastructure.
c: Induced Seismicity: induced seismicity embraces man-made earthquakes

and related processes which are mainly due to mining operations and water
injections into the Earth. We travelled to geothermal power plants, deep
mines and water reservoirs which are potential sources of induced
seismicity. Monitoring concepts were explained to the students as well as
the destroying impact of induced seismicity to infrastructure.
d: Historic Seismicity and its Use for Seismic Hazard Analysis: historic seismicity
deals with earthquakes and their impact on society and nature in the past
(mainly the time before instrumental seismicity started at around 1900). We
explained the relevance of historic seismicity for estimating the hazard and
risk by future earthquakes in the preparatory course and then visited a
museum with historic seismic instruments as well as a town near to our
university which suffered from destructive earthquakes in the past.
e: Geophysical Investigations at a Complex Miocene Volcanic Structure: this

lecture deals with questions such as how can we use geophysical exploration
and measurements in order to investigate a complex volcanic structure. We
went to the extinct Vogelsberg volcanic complex near Frankfurt, Germany
and visited quarries, geotopes and borehole sites. There we explained the
use of geophysical measurements which are needed to understand the
volcanic structure at depth. In the preparatory lectures, students worked on
poster presentations about borehole logging methods, and in the field we
discussed the results of geophysical measurements conducted in the region.
Students conducted geomagnetic measurements and had to identify
anomalies of the local magnetic field.

137
Discussion
We introduce an extensive concept in which a part of the traditional classroom
teaching is transferred to the actual places where the subject matter can be
studied directly and under realistic conditions. Compared to other teaching
approaches such as excursions or fieldwork, we propose to lecture and practise
comprehensively in situ (Table 1). Whereas students often follow passively the
visited sites during an excursion, we motivate and force them to actively
contribute to the lecturing. This active contribution goes beyond typical
fieldwork, as the students are involved e.g. in preparing the lecture notes or
giving presentations. Basic concepts and skills are taught and learnt ahead of
the in situ part during preparatory classroom activities. During the in situ part,
lectures are done to repeat these basics and deepen the students’ knowledge. In
addition new learning matter is introduced by including a direct link to local
specialities, some of which can never be presented in a realistic manner inside a
classroom (Figures 4-7). Active application of the freshly trained skills will
admit students an even deeper insight to the subject during the in situ practicals
and in situ seminars. This comprehensive learning cycle helps students acquire
a wide range of competences, even exceeding the main subject.
When the complete preparatory and in situ lecturing is prepared and executed
by the same lecturer or lecturer team, the subject matter can be presented in a
coherent way to the students. This avoids confusing students due to different
descriptions or parameter abbreviations of the same object as it can happen
when different lecturers use their own teaching material.
Figure 7: Students exploring hot fumaroles inside the Fossa crater on

Vulcano island, Italy. The students measured the temperature of the emissive
gas and liquid sulphur (up to 270 °C), analysed rock samples and identified
endangered infrastructure during the in situ practical (step D).

138
The in situ lecturing is suited for bachelor’s as well as master’s level teaching, we
also applied it to mixed groups (from first year bachelor’s to second year
master’s students). Of course, exercises and themes for seminar presentations
should be adjusted individually to the different students according to their
background and experience.
The in situ lecturing requires significant input and active contribution from the
students: both, the preparatory and in situ phases include practicals which can
be quite time consuming. Especially the preparation of the in situ seminar
presentation, including the generation of hand-out material for fellow students,
may take some time. An example for a work plan can be outlined as follows: we
plan about 40-60 working hours for the active preparatory phase and 10-20
hours for the preparatory lecturing. Depending on the subject and site, the in
situ part can last another 30 hours (3 days) to 120 hours (12 days). For the final
report about 20-40 hours may be required. This corresponds to an overall work
load of 200-300 hours or 7-10 credit points of the European Credit Transfer
System (ECTS, with 1 ECTS credit point equivalent to 30 hours of student work).
An important point is to make clear to the students what is expected from them.
Especially presentation material (posters, handouts, computer presentations …)
prepared for the in situ part must be done thoroughly by the students, because
missing background material may not be available during travel. If the students
prepare material for in situ lecturing, then flaws must be avoided as it may be
also difficult to conduct revisions during travel.
The students can be included in the organisation of the in situ part in order to
learn the organisational side of their subject. For instance students may organise
the travel to a starting point of the in situ part. We told our students that the in
situ part of a volcanism-related lecture series will start at the port of Naples
(Italy) at a specific pier, day and time. It was their own responsibility of get to
this place in time which is about 1000 km away from their usual classroom.
Figure 8: Students giving an in situ seminar presentation on the mechanism

and volcanic hazard of the 1944 lava flow at Mt. Vesuvius, Italy. The group
stands on this specific lava flow and the city of Naples with one million
inhabitants is seen in the background (step E).

139
Figure 9: Students giving an in situ poster presentation (step E) on a magnetic

anomaly at the Geiselstein volcanic intrusion, Vogelsberg region, Germany.
The group stands on top of the magmatic rocks which were also studied with
geomagnetic measurements during an in situ practical (step D).
Most students are highly motivated during our in situ lecturing. The students’
seminar presentations, hand-out material and their final reports were prepared
predominantly in an excellent way. The average grades assigned to the in situ
courses were better compared to other classroom lectures. We interpret this
positive outcome to be the result of high motivation due to our concept of in situ
lecturing.
Likewise our in situ courses were evaluated very positively by the students
within the regular anonymous evaluation procedure which is conducted at our
university (KIT) (Craanen, 2010). The overall grade of the students’ evaluation
was always very high (between 1 and 1.5 with 1 as the best grade on a scale
between 1 as excellent and 5 as deficient). Furthermore, the students provided
helpful comments to improve this kind of lecturing method. Students
commended for example “that they liked to talk to local experts”, “that they
could go to sites which are not publicly accessible”, “that they were
demonstrated the subject in accordance with practical needs” or “that they could
evidently realise the relationship between theory and real measurement
instruments”. We are encouraged by this positive feedback to further conduct
and develop in situ lecturing as well as recommend this concept to other
lecturers.
Acknowledgements
Our development of the in situ courses benefitted much from the response of the
students who gave valuable comments in their official evaluation sheets for the
courses. In two cases our geophysics lectures were complimented with a
valuable geology part by Geologierat Bernd Schmidt (Mainz) during the
preparatory and the in situ phases. Such interdisciplinary input improved the
quality of the teaching. Prof. Norman Harthill (Karlsruhe) kindly helped
improve the manuscript. The in situ lecturing in Southern Italy was financially
supported by a teaching grant (Fakultätslehrpreis) to E. G.

140
References
Butler, R. (2008). Teaching Geoscience through Fieldwork. GEES Subject Centre,
Learning and Teaching Guide, http://www-new1.heacademy.ac.uk/
assets/Documents/subjects/gees/GEES_guides_rb_teaching_geoscience.pdf
(accessed 1 March 2015).
Claiborne, L., Morrell, J., Bandy, J., & Bruff, D. (2014). Teaching outside the classroom,
http://cft.vanderbilt.edu/guides-sub-pages/teaching-outside-the-classroom
(accessed 1 March 2015).
Craanen, M. (2010). Fakultätsübergreifendes Monitoring der Veranstaltungsqualität am
Karlsruher Institut für Technologie (KIT) [Title in English: Inter-faculty
monitoring of teaching quality at KIT]. Qualität in der Wissenschaft (QiW) -
Zeitschrift für Qualitätsentwicklung in Forschung, Studium und Administration, 4
(1/2010), 2-11.
Handelsman, J., Ebert-May, D., Beichner, R., Bruns, P., Chang, A., DeHaan, R., Gentile, J.,
Lauffer, S., Stewart, J., Tilghman, S. M., & Wood, W. B. (2004). Scientific
Teaching. Science, 304 (5670), 521-522.
Hursh, B. A., & Borzak, L. (1979). Toward cognitive development through field studies.
Journal of Higher Education, 50, 63-78.
Kastens, K. A., Manduca, C. A., Cervato, C., Frodeman, R., Goodwin, C., Liben, L. S.,
Mogk, D. W., Spangler, T. C., Stillings, N. A., & Titus, S. (2009). How
geoscientists think and learn. Eos, Transactions American Geophysical Union, 90
(31), 265-266.
Laws, P. (1991). Calculus-based physics without lectures. Physics Today, 44 (12), 24-31.
Lonergan, N., & Andresen, L. W. (1988). Field-based Education: Some Theoretical
Considerations. Higher Education Research and Development, 7, 63-77.
Powell, K. (2003). Science education: Spare me the lecture. Nature, 425, 234-236.
Reiber, K. (2006). Wissen – Können – Handeln. Ein Kompetenzmodell für
lernorientiertes Lehren [Title in English: Knowledge – Ability – Action: A
conceptual model for learning-oriented teaching]. Tübinger Beiträge zur
Hochschuldidaktik, 2/1.
Thompson, D. B. (1982). On discerning the purposes of geological fieldwork. Geology
Teaching, 7, 59-65.

141

Vol. 11, No. 1, pp. 141-160, April 2015
The Mathematics Problem and Mastery Learning

for First-Year, Undergraduate STEM Students
Layna Groen, Mary Coupland, Tim Langtry, Julia Memar,

Beverley Moore and Jason Stanley
School of Mathematical and Physical Sciences,
University of Technology Sydney, NSW Australia
Abstract. In the 2014 academic year Mastery Learning was

implemented in four first-year mathematics subjects in an effort to
address a lack of preparedness and poor outcomes of increasing
numbers of undergraduate students in science, engineering and
mathematics programs. This followed partial success with the use of
diagnostic testing and pre-teaching, active learning, and a greater
emphasis on problem solving in context - under-prepared students were
still more likely to fail the pre-teaching subject and to struggle with
subsequent mathematics subjects. This paper describes the learning
design used, and the outcomes achieved, with implementing Mastery
Learning – the positive: improved academic success, time management,
and attitudes towards learning and Mathematics, an increased sense of
independence, confidence and retention of content, and reduced stress
and anxiety; and the negative: students having a sense of being taught
how to pass a test rather than having a deeper understanding of the
content. It will be seen that this negative is a consequence of a small but
important difference in implementation.
Keywords: The Mathematics Problem, Mastery Learning, first-year

undergraduate experience, competency-based assessment.
1. The Challenge – the Mathematics Problem

First-year undergraduate mathematics education of students in science,
technology, engineering and mathematics (STEM) programs currently face
many challenges. In Australia, failure rates are unacceptably high for some
intake cohorts (Groen, Beames, Coupland, Stanley & Bush, 2013), and attrition is
higher than desirable. These problems can be traced back to high school where
around 40 per cent of junior secondary mathematics classes are taught without a
qualified mathematics teacher (McKenzie, Rowley, Weldon & Murphy, 2011).

142
Additionally, enrolments in elementary 1 mathematics subjects are increasing

(Barrington & Evans, 2014). This is a problem as hard prerequisites for entry to
most STEM programs in most Australian universities do not exist, though
advanced mathematics is usually „recommended‟ and intermediate mathematics
is „assumed‟. These recommendations and assumptions are often ignored, and
students enter university with elementary mathematics or no mathematics at the
senior high school level. In 2013, students with these backgrounds amounted to
56% of the University of Technology Sydney (UTS) science intake, 31% of the
engineering intake and 28% of the mathematics intake. (Information technology
students have no mathematics subjects in the cores of their programs.) The lack
of preparedness and declining enrolments, collectively referred to as the
Mathematics Problem (Hawkes & Savage, 2000), are problems world-wide (Hoyt
& Sorensen, 2001; Smith, 2004; Luk, 2005; Heck & van Gastel, 2006; Brandel,
Hemmi & Thunberg, 2008; Rylands & Coady 2009; Varsavsky, 2010).
More than a decade ago, UTS, and many other Australian universities,
introduced diagnostic testing of assumed knowledge (at UTS called the
Readiness Survey). The diagnostic tests worked (and work) in conjunction with
the pre-teaching subject, at UTS, Foundation Mathematics. Students failing the
Readiness Survey take Foundation Mathematics prior to their first core
mathematics subject to ensure they have the assumed knowledge of their
program.
Over this same period, small changes to the first-year undergraduate

mathematics curriculum were made, active learning 2 was incorporated into
learning designs of all first-year mathematics subjects, and changes were made
to assessment schemes to reflect the more diverse learning activities undertaken.
There was also an increase in emphasis on problem-based learning. Though
improvements in pass rates were observed, unfortunately, under-prepared
students were still more likely to fail Foundation Mathematics and later subjects.
Our response to a lack of mathematical preparedness of some STEM students
needed to be revisited.
2. A Solution – Mastery Learning

2.1 Background
Mastery Learning endorses the belief that all students can learn and achieve the
same level of content mastery when provided with the appropriate learning
conditions (including time) (Bloom, 1971). Mastery is defined in terms of a
subject‟s objectives and achievement of a prescribed level of performance, or
competency, in (criterion-referenced) tests. This level of performance is usually
75 or 80% of the marks available for the „mastery‟ test. There is little or no delay
between marking and feedback, and students failing to meet the mastery level
1 Using the classification system of Barrington and Brown (2005), for New South Wales
high school mathematics subjects, Mathematics Extensions 1 and 2 are classified as
“advanced”. Mathematics (2 unit) is “intermediate” and General Mathematics is
“elementary”.
2 See Prince (2004) for a brief description of active learning and problem-based learning.

143
are provided with remedial activities before a second attempt at achieving

mastery. The process may be repeated (theoretically, as many times as required
to demonstrate mastery, but practically, and in the current implementation,
three times).
Mastery Learning is not a recent innovation, but it is ideally suited to

undergraduate mathematics education given its frequently hierarchical
structure. It is particularly suited in the case of under-preparedness as it has the
capacity to deal with individual learner differences. The research literature
indicates positive effects of mastery learning on students, especially in the areas
of achievement, attitudes toward learning, and the retention of content (Guskey
& Pigott, 1988; Kulik, Kulik, & Bangert-Drowns, 1990; Anderson, 1994; Trigg,
2013). Why then isn‟t the use of Mastery Learning widespread? The answers lie
in its intensive resource use and the (usual) requirement to allow students to
work at their own pace. The issue associated with resource use is now very
effectively addressed by online learning systems where immediate marking and
feedback are available, remedial activities can be flagged and supported, and
textbooks are not merely ebooks. The second issue is one that requires a more
creative approach to learning activity (including assessment) scheduling. The
implementation chosen for UTS first-year mathematics students utilised both
these answers.
Mastery Learning was trialled at UTS in the first (Autumn) semester of 2013.
The results were promising enough to suggest a further trial of two subjects in
Autumn semester of 2014. Analysis of the results of these two subjects
confirmed Mastery Learning as a solution to the problems facing first-year
STEM students in their mathematics subjects, and Mastery Learning was
implemented in another two subjects in second (Spring) semester of 2014. This
paper examines the success, or otherwise, of this Mastery Learning initiative.
2.2 Implementation
After subdividing the subject curriculum into learning units and further into a
logical sequence of smaller objectives, learning materials, instructional strategies
and activities were identified, sequenced and executed over the teaching period.
Criterion-referenced tests were administered. These were supervised, online,
summative tests of just under an hour‟s duration, undertaken approximately
two weeks after the completion of a unit. They assessed the „fundamental‟
knowledge and skills objectives of the unit, that is, the knowledge and skills that
provide the basis for further development. In the UTS implementation,
„mastery‟ was set at 80% of the marks available on the mastery tests.
Students were provided with multiple opportunities for formative assessment

prior to the mastery tests. Marking and feedback for both the formative and the
summative assessments were provided online immediately. Students used the
feedback to feed-forward, engaging in remedial activities individually or
collaboratively (where necessary). Students were then given a further formative
assessment before being given a second opportunity to sit the test, the „second-
chance‟ test - same concepts, different questions (Bloom, 1971). Students already

144
exhibiting mastery, could choose to undertake the second-chance test. The best
mark of the attempts was used in determining the final mark for the subject.
While there was slight variation between subjects as to the finer details of the
implementation, four mastery tests were scheduled over the course of the
semester. Each test was worth 16% of the final mark for the subject. The final
exam was then worth 36% of the final mark. Passing at mastery level in all four
mastery tests ensured that students achieved a Pass in the subject. Perfect scores
ensured an upper range Pass was achieved. Students achieving mastery in all
tests could sit the final examination to improve their result beyond the Pass
grade. Students were required to earn 13/16 on at least one of their attempts of
each mastery test. Students not achieving mastery on the first attempt were
given a choice of remediation activities to participate in. This remediation was
conducted outside scheduled class time. Where mastery was not demonstrated
on the second attempt, more structured activities were made available
(primarily small group and peer-assisted learning).
The third attempt was conducted at the end of the semester, allowing students
additional time to acquire the knowledge and skills they needed. (The third
attempt was only available to students who had not already demonstrated
mastery.) To facilitate a successful third attempt, the last third of the semester
consisted of enrichment activities which were examined in the final exam. The
emphases in these enrichment activities were the application of modelling and
problem-solving using the tools developed in the earlier units. These enrichment
activities also provided students who were yet to demonstrate mastery with the
opportunity for reinforcement and further exposure to content in context.
Students requiring a third attempt (approximately 5% of a class, on average)
were not eligible to sit the final exam. This demonstrates the primary trade-off
made to implement Mastery Learning in semesters of fixed length, and is similar
to some of the implementations reported by Twigg (2013).
UTS is not the only Australian university to implement Mastery Learning, the
University of Canberra implemented Mastery Learning in 2014. Twigg (2013)
reports on a number of US tertiary institutions that are also using Mastery
Learning as part of a learning design called the Emporium Model.
2.3 The subjects

The subject Foundation Mathematics is offered to any UTS student, but
primarily targets students who fail the Readiness Survey. This quick, online
diagnostic test is only compulsory for engineering students. Engineering
students who fail the test are enrolled in Foundation Mathematics. For other
students failing the survey, enrolment in Foundation Mathematics is
recommended. Success of this combination has been mixed. For example,
failure rates in Foundation Mathematics by engineering students over the 2012-
2013 semesters averaged around 59%. Mastery Learning in this subject was
introduced in the Autumn semester of the 2014 academic year.

145
The standard program for engineering students includes Mathematical

Modelling 1, followed by Mathematical Modelling 2. Mastery Learning was
introduced into Mathematical Modelling 2 in Autumn semester of 2014, while it
was introduced into Mathematical Modelling 1 in Spring semester. This was a
consequence of the fact that the Autumn Mathematical Modelling 2 cohort is
smallest in first semester, and a final decision regarding the roll-out of Mastery
Learning to other subjects had yet to be made.
Science students undertake Mathematical Modelling for Science, followed by

Mathematics and Statistics for Science. Mastery Learning in these subjects was
introduced in the 2015 academic year (following success in the other subjects).
Mathematics students undertake Introduction to Linear Dynamical Systems,
followed by Introduction to Analysis and Multivariable Calculus. Mastery
Learning was introduced into Introduction to Multivariable Calculus in Spring
semester 2014, while it was introduced into Introduction to Linear Dynamical
Systems in Autumn 2015. These sequences of subjects are broadly similar in
content, learning and assessment design, and learning environment. Students in
information technology, biological, medical and environmental sciences have no
core first year mathematics in their programs, though some take Foundation
Mathematics.
Active learning was introduced into first-year mathematics subjects

approximately ten years ago. In these subjects this takes the form of interactive
activities as well as collaborative learning in what were once traditional lectures.
Minor changes were also made to assessment schemes at this time to encourage
participation in all learning activities. However, for most first-year mathematics
subjects, the majority of marks were still allocated to the traditional closed-book
final exam. Weights for this varied, but the typical weight for the final exam was
65% of the marks available for the subject. (As previously mentioned, this
changed with the introduction of Mastery Learning.)
3. Data
Data was collected concerning program, tertiary entrance rank (ATAR),
Mathematics subject(s) studied in high school and mark(s) in the Higher School
Certificate (HSC) or other background, as well as final mark and grade in
Foundation Mathematics, Mathematical Modelling 1, Mathematical Modelling 2,
Introduction to Linear Dynamical Systems and Introduction to Analysis and
Multivariable Calculus for the 2012-2014 academic years (six semesters).
Information about the sample sizes can be found in Table 1.
Table 1 – Size of the database for Foundation Mathematics (FM), Mathematical

Modelling 1 (MM1) and Introduction to Analysis and Multivariable Calculus (IAC)
Subject Background (2012, 2013, 2014)
Adv. Int. Elem. No Tertiary Other*

(HSC) studies
Semester Ext. 2 Ext. 1 Math. Gen. senior
Math. math.

146
FM Autumn 3, 6, 9 9, 18, 41, 21, 34, 1, 3, 7 217, 146, 22, 23,

24 31, 50 63 141 27
Spring 2, 0, 3 5, 8, 3 7, 8, 9, 13, 9 0, 0, 3 128, 92, 18, 19,

18 86 31
MM1 Spring 1, 1, 3 11, 3, 21, 5, 10, 5 0, 0, 0 77, 68, 72 23, 30,

12 26, 17 33
IAC Spring 25, 36, 15, 0, 0, 1 0, 0, 0 19, 13, 10 4, 6, 4

22, 16 34, 19 20, 13
The “Other” category includes students with partially complete qualification,

and students where no information on their background could be found.
The class sizes of the six semesters of Mathematical Modelling 2 were 286, 560,
235, 574, 274 and 483 respectively.
Students are assigned grades based on their final marks – A Fail (Z) is 0% to
49%, a Pass (P) is 50% to 64%, a Credit (C) is 65% to 74%, a Distinction (D) is 75%
to 84%, and a High Distinction (H) is 85% to 100%.
To examine qualitative aspects of the student experience, responses to open-

ended questions on the end-of-semester student surveys (Subject Feedback
Surveys) were examined. Representative responses were included in this paper.
Focus groups were also conducted by staff not involved with the teaching or
administration of the subjects included in the study. Students self-selected to
participate, in line with ethics approval. Groups consisted of up to 10 students,
and responses to six set questions (on attitudes, confidence, stress and
assessment structure and nature) and a concluding open-ended, catch-all,
question.
4. Methodology
Mixed methods were used to assess the impact of Mastery Learning -
quantitative techniques were used to assess student achievement, though
students‟ perceptions of this achievement were also examined. Qualitative
techniques were used to assess the impact of Mastery Learning on things such as
confidence, anxiety, attitudes, and behaviour.
In order to examine the statistical significance of any improvements in mean

final marks that may be consequent to the implementation of Mastery Learning,
one-sided t-tests comparing sample mean final marks, under the assumption of
unequal variances, were used. Where the mark distribution wasn‟t normally
distributed, a chi-square test was used (Levine, Berensen & Krehbiel 2008).
Pairwise comparisons of semester results from 2012 with 2014 and 2013 with
2014 were undertaken. A 5% level of significance was used for all statistical
tests. Where the level of significance was something other than this, p-values are
reported. Autumn outcomes were compared with Autumn outcomes, and

147
Spring outcomes with Spring outcomes (representing the different intakes into
the programs).
For pairwise comparisons of failure rates z-tests were used (Stat Trek, 2015).
Where normality couldn‟t be assumed, chi-square tests were used. For cohorts
where the expected numbers of failures or successes were too small (<5), Fisher
Exact tests (McDonald, 2014), as implemented at Preacher (2015), were used.
Comparison of medians was also undertaken as a means of examining the

impact of any skewness in the mark distributions. For pairwise comparison of
medians, z-tests or the Wilcoxon Rank Sum Test were used (where an
underlying normal distribution could be assumed), otherwise a Mann-Whitney
U-test was used. These were implemented using the Mann-Whitney U-test
Calculator of Stangroom (2015) and the implementation of the Wilcoxon Rank
Sum Test in Excel using Zaiontz (2015). These techniques were selected in
preference to transforming the data using a logarithmic transformation as some
of the cohort sizes were too small to assume the transformed variables would be
normally distributed.
5. The outcomes
5.1 Quantitative measures
5.1.1 All programs

35010 Foundation Mathematics. In Table 2 we can see an overall reduction in
failure rates for the Autumn semester cohorts – the improvement between the
Autumn 2012 rate and the Autumn 2014 rate is significant at the 10% level (p =
0.087), and the improvement between the Autumn 2013 rate and the Autumn
2014 rate is significant at the 1% level (p = 0.00002). There is also reduction in
the failure rate for the target background cohort, General Mathematics students.
Using Fisher‟s Exact Tests to compare Autumn 2012 with Autumn 2014, the
reduction in failure rate is significant at the 10% level (p = 0.092), while the
reduction between Autumn 2013 and Autumn 2014 is not significant (p = 0.219).
A one-sided t-test assuming unequal variances on the mean overall final mark
finds that the improvement in mark is significant at the 5% level when Autumn
2013 is compared with Autumn 2014, but is not significant when Autumn 2012 is
compared with Autumn 2014. These results are duplicated for the mean final
mark for the Mathematics and General Mathematics cohorts.
Median final marks also improve for nearly all background cohorts with
Mastery Learning. For the cohorts of primary interest, the General Mathematics
and Mathematics cohorts, the improvement in median using the z–test for the
General Mathematics cohort when Autumn 2012 is compared to Autumn 2014 is
significant at the 10% level (p = 0.051), and the increase in median when Autumn
2013 is compared to Autumn 2014 is significant at the 1% level (p = 0.00082). For
the Mathematics cohort, the Autumn 2012 comparison is not significant, while
the Autumn 2013 comparison is significant at the 1% level (p = 0.00036).

148
Table 2 - Autumn semester results by high school background

Back- Autumn 2012 Autumn 2013 Autumn 2014
ground
mean median %fail mean median %fail mean median %fail
(of) (of) (of)
Ext. 2 97.0 98 0% 77.50 83.5 0% 87.00 90 0%

(3) (7) (8)
Ext. 1 85.11 87 0% 66.78 78 0% 80.71 92.5 8%

(9) (15) (24)
Math. 73.59 81 33% 56.00 57 25% 72.88 81 14%

(12) (24) (50)
General 49.57 50 48% 30.97 19 45% 52.97 57 28%

Math. (21) (11) (63)
No HSC - - - - - - 55.5 55.5 0%

math. (3)
Overall 66 76 21.16 53 60 28.81 66 70 18.75

% % %
(314) (261) (321)
In Table 3 we see the results for the Spring offering – the reduction in failure
rates on pairwise comparisons overall are significant (p = 0.00002 and 0.027
respectively). We can also see improvement in the failure rates of students with
backgrounds in Mathematics and General Mathematics. The improvement in
failure rates for students with a Mathematics background is significant (p =
0.004) when Spring 2012 is compared with Spring 2014 (using the Fisher Exact
Test). However, when Spring 2013 is compared with Spring 2014, the
improvement is not significant. The comparisons of the failure rates for students
with a General Mathematics background between Spring 2012 and Spring 2014,
and Spring 2013 and Spring 2014 are both statistically significant (p = 0.009 and
0.049 respectively). Here we again see favourable outcomes for the main target
group (students with a General Mathematics background).
Using a one-sided t-test with unequal variances on the mean overall Spring final
mark, we find that this improvement is significant at the 5% level when Spring
2012 is compared with Spring 2014, but is not significant when Spring 2013 is
compared with Spring 2014. For the students with Mathematics backgrounds,
the increase in mean and median were significant when Spring 2012 - Spring
2014 comparison, but not when Spring 2013 was compared to Spring 2014. For
the General Mathematics cohort, the increases in the mean for both comparisons
were significant, but there were no significant improvements in the medians.

149
Table 3 - Spring semester results by high school background

Back- Spring 2012 Spring 2013 Spring 2014
ground
(of) (of) (of)
Ext. 2 91 91 0% - - - 88 85 0%
(2) (3)
Ext. 1 78 81 0% 88 94 0% 84 85 0%
(5) (8) (3)
Math. 30 30 63% 66 70 13% 73 75 6%

(8) (8) (18)
General 30 56 63% 45 54 38% 59 55 0%

Math. (8) (13) (9)
No HSC - - - - - - 11 2 100%
maths (2)
Overall 58 68 21.43 65 73 12.21 66 68 9.61%

% % (153)
(169) (140)
To ensure that significant improvement in most of the statistics measuring

achievement could not be attributed to more capable students electing to take
General Mathematics, ATAR was examined for the Autumn and Spring cohorts.
For the Autumn comparisons, mean ATARs were 74, 80 and 67, with the
Autumn 2012 to Autumn 2014 reduction significant at the 10% level, while the
Autumn 2013 to Autumn 2014 reduction was significant at the 1% level (using a
t-test with equal variances). For both pairwise Spring semester comparisons,
there was no significant difference at the 5% level between the mean ATARs.
We can conclude that there is no evidence in the mean ATARs of both the
Autumn and Spring cohorts that suggest more capable students are electing to
do General Mathematics, and hence that there is no significant increase in
capability of the 2014 cohorts indicated.
Foundation Mathematics data were also broken down by program. In Table 4

we can see that though numbers of Engineering students in the Spring offering
of Foundation Mathematics have increased significantly, the failure rate for this
cohort is now 0%. Comparisons of the failure rates for Engineering students
reveal that the reduction in failure rates for both Autumn and Spring cohorts for
all years compared are significant at the 5% level – in three of the four
comparisons the improvements in failure rates are significant at the 1% level.
Science students, also show a 67% decrease in failure rate in the Autumn cohort
and a 50% decrease in failure rates in the Spring cohort. Comparisons of the
failure rates for Science students reveals that the reductions in failure rates for
both Autumn and Spring 2013 cohorts are significant at the 5% level, while
Fisher‟s Exact test yields no significant reduction in comparing the Autumn 2012

150
cohort (p = 0.123), though in comparing Spring 2014 with Spring 2012 the
improvement is significant (p = 0.047).
Students in Information Technology programs, have the lowest failure rates.

This can be explained by the fact that they have higher participation rates in
intermediate and advanced mathematics subjects at high school.
Table 4 - Failure rates by STEM course

Year 2012 2013 2014
Semester Autumn Spring Autumn Spring Autumn Spring

(of) (of) (of) (of) (of) (of)
Science 0.2 (25) 0.333 (6) 0.633 0.364 (11) 0.205 0.105 (19)
(30) (39)
Engineering 0.697 0.760 (46) 0.580 0.317 (41) 0.377 0

(76) (69) (69) (102)
Information 0.05 (20) 0.091 (33) 0.043 0 0.25 (4) 0

Technology (23) (12) (13)
5.1.2 Engineering programs.
33130 Mathematical Modelling 1. Using a one-sided t-test assuming unequal

variances (and a 5% significance level) on the mean overall Spring final mark,
we see the improvement is significant when Spring 2012 is compared with
Spring 2014, but is not significant when Spring 2013 is compared with Spring
2014. These results are repeated for the mean final mark for the Mathematics
cohorts. The analysis couldn‟t be conducted for the General Mathematics
cohorts as the sample sizes were too small.
Overall, we can see an approximately 50% reduction in the failure rate overall
compared with previous Spring semester results. This result is significant at the
5% level for both the Spring 2012 and Spring 2013 comparisons to Spring 2014.
Considering background cohorts, for those students with a Mathematics
background using Fisher‟s Exact test yields a significant reduction in failure rate
at the 5% level when the Spring 2014 Mathematics cohort is compared to the
Spring 2012 cohort (p = 0.0005), but the reduction is not significant when the
Spring 2014 cohort is compared to the Spring 2013 cohort (p = 0.185) . Other
background cohorts from the target groups are too small to perform significance
tests.
The failure rates for students who had undertaken Foundation Mathematics
prior to taking Mathematical Modelling 1 has seen a reduction from 25% down
to 6% for a similar sized cohort. Pairwise comparisons using a z-test
demonstrates that both reductions are significant at the 5% level (p = 0.00004 and
0.015 respectively). No strong conclusions can be drawn about the failure rates

151
for those students who did HSC mathematics and who did not take Foundation
Mathematics as the background cohorts are too small. However, when students
with other backgrounds are included, we see a reduction in failure rate from
20% to 12%. Pairwise comparisons using a z-test demonstrates that only the
reduction from 2012 to 2014 is significant at the 5% level (p = 0.00002), while the
reduction from 2013 to 2014 is not significant (p = 0.116).
Using a one-sided t-test with unequal variances on the mean overall Spring final
marks, finds that this improvement is significant at the 5% level when Spring
2012 is compared with Spring 2014, but is not significant when Spring 2013 is
compared with Spring 2014.
For the cohort that hadn‟t undertaken Foundation Mathematics (the top
groupings of Table 5), we see that a one-sided t-test with unequal variances
demonstrates that there is a significant improvement (α=0.05) between Spring
2012 and Spring 2014 mean final marks. However, the increase in mean marks
for the Spring 2013 and Spring 2014 cohorts is not significant. Similar results
hold for the comparison of mean final marks for students who did undertake
Foundation Mathematics prior to enrolling in Mathematical Modelling 1.
In comparing medians little can be said about individual background cohorts as

the samples sizes are quite small. In comparing the overall Foundation
Mathematics and non-Foundation Mathematics cohorts, and the subject overall
cohort, the increases in median are significant at the 5% level when Spring 2012
is compared to Spring 2014 (p = 0.027, p = 0.00076, p = 0.00012). When the Spring
2013 median is compared with the Spring 2014 median for the non-Foundation
Mathematics cohort and for the subject overall, we find significant
improvements (p = 0.015 and p = 0.03673 respectively). However there is no
significant improvement for this yearly comparison for the cohort who had
previously taken Foundation Mathematics.
Table 5 - Spring semester results by background

Spring 2012 Spring 2013 Spring 2014

(of) (of) (of)
No FM
Ext. 2 0 0 100% 86 86 0% 47 59 33%

(1) (1) (3)
Ext. 1 50 42 55% 72 72 0% 64 67 8%
(11) (3) (12)
Math. 39 40 57% 55 54 19% 60 59 6%

(21) (26) (17)

152
General 24 30 100% 51 54 30% 35 55 40%

Math. (5) (10) (5)
Cohort 51 52 41% 58 57 20% 61 64 12%

overall (117) (121) (114)
With FM
Ext. 2 90 90 0% 84 89 0% 60 60 0%
(1) (3) (1)
Ext. 1 0 0 0% 66 63 0% 63 63 0%
(0) (3) (1)
Math. 43 52 0.5 59 72 20% 57 57 0%

(6) (5) (1)
General 40 43 54% 49 55 22% 55 58 7%

Math. (13) (9) (15)
No HSC - - - - - - 54 54 0%
Maths (0/1)
Cohort 47 50 44% 55 62 25% 58 60 6%

overall (43) (48) (49)
Total 50 51 41% 57 58 21% 60 62 10%

Overall (138) (138) (142)
33230 Mathematical Modelling 2. For the Autumn 2014 Mathematical

Modelling 2 cohort in Table 6 we see an increase in mean and median final
mark, and associated reduction in failure rate for students who achieved a Pass
in Mathematical Modelling 1.
The improvements in overall mean final mark for both Autumn semester
comparisons are significant at the 5% level, and are in fact significant at the 1%
level, with the implementation of Mastery Learning. The improvements in
median for both Autumn comparisons were significant at the 1% level using the
Wilcoxon Rank Sum Test.
For the cohorts who achieved a Pass grade or better in Mathematical Modelling
1, it is not universally true that the mean and median final marks improved. The
improvements in mean and median final marks are significant (at the 1% level)
for the two, year comparisons of students who obtained Passes in Mathematical
Modelling 1, and when Credits and Distinctions for Autumn 2012 are compared
with Autumn 2014, but are not significant at the 5% level for Credits and
Distinctions when Autumn 2013 results are compared with Autumn 2014
results.
The reduction in failure rates overall for both Autumn comparisons are
significant at the 1% level. For students who received a Pass in Mathematical

153
Modelling 1, both comparisons to Autumn 2014 resulted in significant

reductions at the 1% level. For students with Credit grades, the Autumn 2012
comparison to Autumn 2014 was significant at the 1% level, though the Autumn
2013 comparison was not (p = 0.105). For students with Distinction grades, the
Autumn 2014 reduction in failure rate was not significant at the 5% level (p =
0.0571) when compared to Autumn 2012, nor was the Autumn 2013 comparison
(p = 0.270). This improvement in outcomes for students earning a Pass grade is
further support for a Mastery Learning approach to first-year Mathematics
subjects for Engineering students.
Table 6 - Autumn semester Mathematical Modelling 2 results by Mathematical

Modelling 1 result
MM1 Autumn 2012 Autumn 2013 Autumn 2014
grade
(of) (of) (of)
Pass 38 40 72% 51 53 35% 58 59 20%

(74) (93) (88)
Credit 50 52 35% 60 64 16% 61 61 0%

(46) (32) (20)
Dist. 53 51 32% 65 68 16% 65 63 6%

(28) (19) (16)
High 63 64 0% 84 87 0% 74 77 0%
Dist. (16) (7) (9)
Overall 47 50 45% 52 53 35% 59 60 11%

(286) (235) (274)
A similar pattern can be seen in the Spring Mathematical Modelling 2

failure rates (Table 7). Again, there is not universal improvement in means and
median final mark for all cohorts, though the improvements are significant for
the Pass and Credit cohorts from Mathematical Modelling 1. Comparisons of
means overall (under an assumption of unequal variances), medians, as well as a
comparison of failure rates, find that improvements in overall outcomes are
significant (α=0.01) for all Spring pairwise comparisons.
Comparisons of mean final mark (unequal variance) and failure rate were also
undertaken for each Mathematical Modelling 1 grade cohort for Spring classes.
The improvements for the Mathematical Modelling 1 Pass and Credit cohorts
are significant for all years compared (means, medians and failure rates at the
1% level). For the Spring Distinction cohort only the 2013 to 2014 failure rate
showed significant improvement, though this was at the 10% level.

154
Table 7 - Spring semester Mathematical Modelling 2 results by Mathematical

Modelling 1 result
MM1 Spring 2012 Spring 2013 Spring 2014
grade
(of) (of) (of)
Pass 48 50 47% 48 52 43% 59 60 6%

(222) (206) (140)
Credit 58 59 15% 59 60 16% 64 65 2%

(124) (109) (84)
Dist. 68 70 7% 70 73 9% 67 68 2%
(91) (90) (100)
High 79 80 2% 82 83 0% 77 77 1%
Dist. (51) (78) (89)
Overall 56 58 27% 59 60 26% 65 64 4%

(560) (574) (483)
Overall then, the improvements in achievement with Mastery Learning are very
encouraging and would appear to ameliorate the previous observation that
students who obtained a Pass grade in Mathematical Modelling 1 were not as
likely to achieve a Pass or higher in Mathematical Modelling 2.
5.1.3 Mathematics programs.
35102 Introduction to Analysis and Multivariable Calculus. Table 8 presents

the mean, median and failure rate of the subject Introduction to Analysis and
Multivariable Calculus based on mark in the prerequisite subject, Introduction
to Linear Dynamical Systems.
Using a one-sided t-test with unequal variances and a significance level of 5% on

the mean of the overall Spring final marks, we find that this reduction is not
significant when Spring 2012 is compared to Spring 2014. It is also true that the
small increase in median is also not significant for this comparison. However,
the Spring 2013 improvement in mean is significant at the 10% level, while the
improvement in median is significant at the 1% level. Little can be said about
the Credit, Distinction and High Distinction cohorts from Introduction to Linear
Dynamical Systems (the prerequisite subject) because of the small size of the
cohorts. However, there is a statistically significant increase (at the 10% level) in
mean final mark for student who achieved a Pass in Introduction to Linear
Dynamical Systems for both pairs of years compared.

155
There is a slight reduction in the failure rate of Pass students, though this is not
significant at the 5% level (p = 0.270 and 0.391 respectively). The reductions in
failure rates overall are not significant (p = 0.221 and 0.187 respectively).
Table 8 - Spring semester Introduction to Analysis and Multivariable Calculus results

by Introduction to Linear Dynamical Systems result
35101 Spring 2012 Spring 2013 Spring 2014

grade
(of) (of) (of)
Pass 50.62 51 33% 50.21 50 26% 51.14 63 24%

(39) (38) (41)
Credit 70 72 8% 60.79 63 14% 60 60 50%

(13) (14) (2)
Dist. 79.5 80 0% 66.29 70 14% 64.25 77.5 25%

(8) (7) (4)
High 90.6 90 0% 79.5 80.5 12.5% 91 91 0%

Dist. (8) (8) (2)
Overall 55 55 32% 52 50 33% 57 64 26%

(99) (95) (63)
Results in Table 9 (results by background) exhibit an improvement in median

result, along with an approximately 25% reduction in failure rate for students
with a Mathematics background. Statistical analysis could not be undertaken on
the Extension 2, Mathematics and General Mathematics cohorts due to the small
sizes of these cohorts (Stat Trek, 2015). There is a significant (α=0.01)
improvement in mean and median final mark in both year comparisons for
students with an Extension 1 background. There is a significant improvement at
the 1% level in failure rate for this cohort when Spring 2012 and Spring 2013 are
compared to Spring 2014 using a Chi-square test.
Table 9 - Spring semester results by high school background

Back- Spring 2012 Spring 2013 Spring 2014
ground
(of) (of) (of)
Ext. 2 61 59 24% 54 51 27% 51 58 38%

(25) (22) (16)
Ext. 1 47 45 56% 54 51 29% 60 67 16%

(36) (34) (19)

156
Math. 56 56 13% 50 50 40% 56 67 31%

(15) (20) (13)
General - - - - - - 69 69 0%
Math. (1)
5.2 Qualitative outcomes - student perspectives

Accommodate individual differences. Mastery Learning by its very nature
seeks to accommodate differences in background and time students require to
develop mastery of learning outcomes. Smaller, targeted tests with immediate
feedback, learning support and multiple attempts to demonstrate mastery are
more accommodating than one-chance testing. The majority of students report
liking the mastery tests:
I like the mastery testing scheme. I think it‟s fair, and I particularly
like that [the lecturer] gives a pool of questions that can be repeated
and practiced. (SFS – Mathematical Modelling 1 Spring 2014)
Achievement. By providing timely feedback, supporting remediation and

allowing multiple test attempts, students are able to identify and address their
weaknesses, leading to an overall improvement in academic performance:
I thoroughly enjoyed the subjects re-structure using the mastery tests. I
previously failed the subject ... The mastery tests have enabled me to
consistently and thoroughly cover the material better preparing me for
my … future career as an engineer. (SFS – Mathematical Modelling 1
Spring 2014)
Having the mastery testing system was really helpful … I found
that being able to view your results and having second attempts
encouraged me to study and learn much more. (SFS – Introduction
to Analysis and Multivariable Calculus Spring 2014)
Attitudes towards Mathematics and Learning. The motivating characteristics of

Mastery Learning and applied problem solving resulted in an improvement in
attitude towards mathematics:
… the Mastery Tests … helped me ease into [the subject and] … I
have a greater appreciation for maths now. (Focus group –
Foundation Maths Autumn 2014)
The second chance in the test really helped me to learn where I had
made my mistakes and to learn from them and to come back and
achieve a great mark the second time around. (SFS – Mathematical
Modelling 1 Spring 2014)
Retention of content. Mastery tests foster knowledge and skills retention:

… the mastery tests … helped retain the concepts longer. (SFS
Foundation Mathematics Autumn 2014)
[Proprietary online learning system] is a fantastic tool, the
instantaneous feedback and the ability to read the text online with
worked examples is great. The ability to continually practice the

157
alternate problems solidifies the knowledge gained. (SFS –

Mathematical Modelling 2 Spring 2014)
Reduced stress and anxiety. Mastery Learning can reduce stress and anxiety by
having more frequent, lower stakes tests that students can also sit multiple
times. Final exam stress is also reduced – students know that they are walking
into the exam knowing that they have already passed the subject:
I liked the fact that … the final exam wasn't as stressful knowing
that you have passed the subject after passing the mastery tests.
(SFS – Mathematical Modelling 2 Spring 2014)
I enjoyed doing the mastery tests as it helped me to stay on top of
the work and made it much more relaxed when preparing for the
final exam (SFS – Introduction to Analysis and Multivariable
Calculus Spring 2014)
However, a small number of students felt that there was too much testing
and pressure:
Just thinking about getting 80% on a mastery test is too [much]
pressure. (SFS – Mathematical Modelling 2 Spring 2014)
Improved time management. Through a structured approach to the timing of

assessment tasks, with Mastery Learning students learn to manage their time
more effectively, and so cramming for the final exam is also discouraged:
The mastery tests were a great way in keeping up to date with
content. (SFS – Mathematical Modelling 2 Spring 2014)
The mastery tests made you learn continuously instead of
cramming at the end of the semester. (SFS Foundation Mathematics
Autumn 2014)
Increased independence. Mastery Learning encourages self-correction and

independence:
I liked … the Mastery tests which ensured that your learning and
understanding of the subject matter were reinforced and that you
kept up to date with your work throughout the semester. The
online [proprietary learning system] was an excellent resource for
study at home. (SFS – Introduction to Analysis and Multivariable
Calculus Spring 2014)
Increased confidence. Mastery Learning can build confidence:

The „2nd chance‟ class test. … It is not the same test but similar and
this does wonders to a student‟s confidence … (SFS Mathematical
Modelling for Science Autumn 2013)
The assessment of 4 mastery test was good - made me feel more
confident approaching finals. (SFS – Mathematical Modelling 2
Spring 2014)
The above aspects combine to empower students so that they are not just
achieving better marks and retaining what they have learned but are also

158
developing skills and attributes that can not only be translated to the rest
of their studies but to their later professional life.
For some students the experience was less positive – they felt that
they were learning how to do the tests not master fundamentals of the
subject:
I was able to pass because all I had to do was learn how to do a
specific set of questions for each mastery test. (SFS – Mathematical
Modelling 2 Spring 2014)
As a consequence of this feedback from students in Autumn 2015 more
variation in questions was included in Mathematical Modelling 2.
Online testing. There are significant advantages to using online testing for the
mastery tests. These include:
[Proprietary online learning system] is a fantastic learning tool … I find
that my ability to learn through tools like this is greatly enhanced due to
the instantaneous feedback and the practice alternate problems. (SFS –
Mathematical Modelling 2 Spring 2014)
However, online testing was not universally popular, especially at the
beginning of the semesters when students were learning how to navigate the
environment:
I did not like the way assessments were conducted online. I spent
far too long trying to work out how to enter the answers on the
computer correctly rather than focusing on the actual material. (SFS
– Foundation Mathematics Spring 2014)
The … [online] assessment need[s] to be more reliable, the system is
unpredictable. (SFS – Mathematical Modelling 2 Spring 2014)
6. Conclusion
The aim of the paper was to contribute to the current debate on the ways to
address the lack of preparedness of some first-year students of STEM programs
(the Mathematics Problem). At the University of Technology Sydney, a form of
Mastery Learning has shown itself to address the lack of preparedness
successfully for many STEM students as well as for first-year mathematics
students overall – it has advantages over one-chance testing and heavily
weighted final exams. These advantages include improved academic
performance. Students also report increased independence and confidence, and
improved time management and retention of content. For many students the
learning experience is positive with less stress and anxiety. While some students
report poor experiences with the online learning and testing environment, most
appreciate the central role it plays in facilitating Mastery Learning. The poorer
experience of students in Mathematical Modelling 2 in 2014 resulted in the fine-
tuning of content in the sequences of formative and summative assessments.
Overall, Mastery Learning appears to afford a sustainable solution to the

increasing lack of mathematical preparedness of some students in STEM
programs.

159
Acknowledgements
Some funding for this project was received under the UTS First Year Experience
Grants.
We would like to thank Dr Kathy Egea for conducting and organising the
transcription of the focus groups. Thanks also go to Stephen Bush for his
assistance with Figure 1 and his encouragement and interest.
References
Anderson, S. A. (1994). Synthesis of research on mastery learning, US Department of
Education, Office of Education, Research and Improvement, Educational Resources
Information Center (ERIC). Retreived 13 October 2014 from
http://files.eric.ed.gov/fulltext/ED382567.pdf
Barrington, F. & Brown, P. (2005). Comparison of year 12 pre-tertiary mathematics
subjects in Australia 2004-2005, Australian Mathematical Sciences Institute/
International Center of Excellence for Education in Mathematics (AMSI/ICE-EM)
report.
Barrington, F. & Evans, M. (2014) Participation in Year 12 Mathematics 2004 – 2013,
Retrieved 14 October 2014 from http://amsi.org.au/publications/participation-year-
12-mathematics-2004-2013/
Bloom, B. S. (1971). Mastery learning, In J. H. Block (Ed.), Mastery learning: Theory and
practice.
Brandel G, Hemmi K & Thunberg H (2008), The widening gap - a Swedish perspective,
Mathematics Education Research Journal 20, 38-56.
Groen, L., Beames, S. Y., Coupland, M. P., Stanley, P., & Bush, S. (2013). 'Are science
students ready for university mathematics?', Proceedings of the Australian
Conference of Science and Mathematics Education (2013), University of Sydney,
Sydney.
Guskey, T. R. & Pigott, T. D. (1988). Research on group-based mastery learning
programs: A meta-analysis, Journal of Educational Research 81, 197–216.
Hawkes, T. & Savage, M. (eds.) (2000). Measuring the Mathematics Problem (London:
Engineering Council). Retrieved 28 October 2014 from www.engc.org.uk/about-
us/publications.aspx
Heck, A. & van Gastel, L. (2006). Mathematics on the threshold, International Journal of
Mathematical Education in Science and Technology, 8 (15), 925-945.
Hoyt, J. E. & Sorensen, C. T., (2001). High School preparation, placement testing, and
college remediation, Journal of Developmental Education 25, 26-34.
Kulik, C. C., Kulik, J. A. & Bangert-Drowns, R. L., (1990). Effectiveness of mastery
learning programs: A meta-analysis, Review of Educational Research 60, 265–299.
Levine, D. M., Berenson, M. L. & Krehbiel, T. C., (2008). Statistics for Managers Using
Microsoft Excel, Retrieved 11 March 2015 from:
http://www.prenhall.com/behindthebook/0136149901/pdf/Levine_CH12.pdf
Luk, H. S., (2005). The gap between secondary school and university mathematics,
International Journal of Mathematical Education in Science and Technology 36, 161-
174.
McKenzie, P., Rowley, G., Weldon, P. R. & Murphy, M., (2011). Staff in Australia‟s
schools 2010: main report on the survey. Retrieved 24 October 2014 from
http://research.acer.edu.au/tll_misc/14

160
McDonald, J. H., (2014). Handbook of Biological Statistics, Retrieved 11 March 2015

from: http://www.biostathandbook.com/fishers.html
Preacher, K. J., (2015). Calculation for Fisher‟s Exact Test. Retrieved 11 March 2015 from:
http://quantpsy.org/fisher/fisher.htm
Prince, M., (2004). Does Active Learning Work? A Review of the Research [Electronic
version], Journal of Engineering Education, 93, 223-231.
Rylands, L. J., & Coady, C. (2009). Performance of students with weak mathematics in
first-year mathematics and science, International Journal of Mathematical Education
in Science and Technology 40 (6) 741-753.
Smith, A., (2004). Making Mathematics Count (London: HM Stationery Office).
Retrieved 28 October 2014 from
www.mathsinquiry.org.uk/report/MathsInquiryFinalReport.pdf
Stangroom, J., (2015). Mann-Whitney U-test Calculator, Social Sciences Statistics.
Retrieved 13 March 2015 from
http://www.socscistatistics.com/tests/mannwhitney/Default2.aspx
Stat Trek.com, (2015). Hypothesis Test: Difference Between Proportions. Retrieved 9
March 2015 from http://stattrek.com/hypothesis-test/difference-in-
proportions.aspx
Twigg, C. A., (2013). Improving Learning and Reducing Costs: Outcomes from
Changing the Equation [Electronic version]. Change: The Magazine of Higher
Learning, 45, (July-August 2013)
Varsavsky, C., (2010). Chances of success in and engagement with mathematics for
students who enter university with a weak mathematics background, International
Journal of Mathematical Education in Science and Technology 41 (8) 1037-1049.
Zaiontz, C., (2014). Wilcoxon Rank Sum Test for Independent Samples, Real Statistics
Using Excel. Retrieved 16 March 2015 from http://www.real-statistics.com/non-
parametric-tests/wilcoxon-rank-sum-test/

161

Vol. 11, No. 1, pp. 161-179, April 2015
Teaching Culture through Language: Exploring

Metaphor and Metonymy in Chinese Characters
Hu, Ying-Hsueh
Tamkang University, English Department
New Taipei City, Taiwan
Abstract. Learners of Mandarin Chinese often find reading and writing

Chinese characters extremely challenging. This study proposes a holistic
approach that is anchored in the theoretic framework of Cognitive
Linguistics to teaching Chinese characters for reading/writing by
explicitly heightening learners‘ awareness of the cultural knowledge
encoded in the radicals and characters. Radicals are keys to learning
characters as they are a vital clue to the meaning of a word and help
compose compound characters. Traditional ways of organizing and
teaching radicals are in accordance with the number of strokes they have.
However, the proposed approach organizes radicals by way of concepts
that reflect the folk categories speakers of Chinese share. Learners were
also taught how conceptual metaphor and metonymy motivate the
formation of radicals and compound words. Twenty-nine international
students at a university in Taiwan participated in the study. A survey
was administered after 6 weeks of treatment yielding results that are
favorable to the approach. It was also found that such an approach may
not suit all learners depending on their prior knowledge.
Keywords: Chinese radicals, semantic cues, cognitive linguistics,

metaphor and metonymy, language and culture.
.
1 Introduction
The Sapir-Whorf Hypothesis claims that one‘s language, depending on whether
it‘s the strong or weak hypothesis we are referring to, shapes or influences our
world view. After the hypothesis was formulated in the early 20th century, a
series of heated debates ensued, which have been followed by countless research
and experiments to refute or support the hypothesis. Evidence so far has
suggested that both the lexicon and the grammatical structure of a language do
seem to influence certain key conceptualizations such as color, space, time,
gender, and the event structure of various motions (Casasanto & Boroditsky,
2007). The evidence, in turn, supports the notion that language is also the product
and manifestation of human conceptualization faculties that have been
influenced by the physical, social and cultural environment humans live in.

162
This interrelationship is best encapsulated in the Embodiment Hypothesis

proposed by Lakoff and Johnson (1980, 1999). The hypothesis posits that primary
human cognition, which is mostly concrete, is anchored in embodied experiences
such as using one‘s body to navigate space, motions, balance, and the senses of
enclosure that create boundaries and dimensions. These in turn give rise to more
abstract cognitive concepts such as time, causality, and container schemas that
help in the comprehension of complex phenomena. In this light, language does
not merely passively label objects or abstract concepts humans have, but
participates in creating concepts at the same time. This understanding has
formed the backbone of cognitive linguistics (CL) as it is known today (Dirk,
2006, pp. 1-20).
This linguistic insight opens up an exciting avenue for foreign language teaching
and learning, particularly with respect to the role of culture, which comprises the
values and beliefs a speech community shares. It also highlights the necessity of
teaching language and culture simultaneously. One fruitful area is the
vocabulary itself. Works on concept transformation in the naming of lexical items
establish that the lexicalization process of a given language should consist of that
which is ―tantamount to category formation at the level of a whole culture.‖
(Györi, 1998, p.99) In other words, the formation of a cultural category inevitably
involves linguistic coding, as there is no other way for conceptual categories to
spread in a culture and for it to become explicitly part of cognitive structures of
the individual members of that culture. In this light, a closer look at the 214
radicals that structure over thousands of Chinese characters frequently used
today reveal a rich conceptual system of categorization. It groups experiences of
various interactions with the natural, social and cultural worlds ancient Chinese
lived in. This conceptual system, as Lakoff (1987) and Johnson (1987) argue,
converges on cognitive mechanisms of prototype, image schema, metaphor and
metonymy. They in turn helped create more vocabulary through these radicals to
form numerous compound characters and words as the language continued to
evolve. Such insight forms the basis of a Chinese e-learning course, CRILL
(Chinese Radical Integrated Language Learning) the researcher developed,
which aims to explore its pedagogical validity.
The course consists of 15 units introducing 15 basic radicals denoting three

groups of concepts encoded in Chinese vocabulary and idioms: body parts,
natural phenomena and plants. Its design was originally to raise the awareness of
Chinese social/cultural values for adult beginners learning Chinese as a Foreign
Language (CFL) so as to increase language retention, cultural understanding, and
enthusiasm. To explore the last two aspects, a small, preliminary study was
launched with a five open question survey designed and distributed to 29
subjects of pre-intermediate level after over 6 weeks of training between 2012 and
2013. A survey was designed to address the following main questions: 1) Can a
Chinese language course based on illuminating the metonymic and metaphorical
concepts in the radicals and characters motivate the learning of reading/writing
for pre-intermediate learners? 2) Can such a design enhance and motivate
cross-cultural understanding? And finally 3) Does a learner‘s language
background affect their perception of pedagogical efficacy and thus enthusiasm
in such a course? For example, would Japanese learners or heritage learners who

163
had learned Chinese characters in school or at home find CRILL as useful as

speakers of Indo-European linguistic heritage?
The findings of the study may have significant pedagogical implications for
teaching CFL, particularly in terms of writing, reading and cultural learning. This
study also provides some insight into the merit of holistic teaching by
expounding on the metonymic and metaphorical concepts encoded in lexical
items, and thus lending increasing support to the practical application of CL in
modern language classrooms.
2 Background of the Study

Embodied Cognition and Cognitive Linguistics
The Embodiment Hypothesis proposed by Lakoff and Johnson (1980, 1999)
claims that ―Reason is not disembodied, as the tradition has largely held, but
arises from the nature of our brains, bodies, and bodily experience.‖ (Lakoff and
Johnson, 1999, p.4) Hence, the functioning of our bodies is crucial for the
structure of our conceptual system. Our conceptual system is, they argue,
mirrored in language patterns, as in systematic use of metaphors. This view gives
rise to their Conceptual Metaphor Theory (CMT) which has been extensively
adopted by cognitive linguists to investigate a wide range of issues from lexicon
meanings such as polysemy (Sweetser,1990) to grammar patterns (Talmy, 1988).
Some linguists argue that metaphorical concepts may have emerged from
metonymic ones (Barcelona, 2000; Radden, 2000). Because of this connection ,
metaphor and metonymy are often intertwined to form ―metaphtonymy‖
(Goossens, 1990). Consider this example : ―She could read my mind ,‖ given by
Ruiz de Mendoza Ibáñez, (2000, p. 121). He explains that ―read someone‘s mind‖
combines a metaphor of MIND IS A BOOK and the metonymy of MIND
STANDS FOR THOUGHT, giving rise to the eventual understanding of ―She
understands me.‖ Metaphors and metonymies are often found on phrasal or
sentential level; however, they also help form lexical items.
Metaphors and Lexicons

Each language family utilizes different ways of encoding concepts in the lexical
items, for example, Proto-Indo-European languages (PIE) employ morphemes
which are mainly consisted of roots, prefixes, and suffixes, and the extension of
meaning can be achieved through compounding, derivation, borrowing, the
creation of neologisms, acronyms, etc. An equally efficient if not more productive
way of extending lexical items, as Dirven (1985, p. 96) points out, is through the
processes of metaphor. He uses the term ‗metaphor‘ in its broad sense which also
includes metonymy. Györi‘s (1996, 1998) work on concept transformation of
naming lexical items in the course of their semantic changes in several major
European languages illustrates these processes, which, he argues, are deeply
anchored in culture. These studies of diachronic semantic changes strongly
suggest that a word structure not only encodes semantic but also conceptual
information. The primary motivation for these changes, as argued by Györi, is
functional because it is based on a speech community‘s adaptation to its
environment, which is not merely biological but, more importantly, a
socio-cultural one.

164
Although Chinese has an entirely different writing system, it also extends its
lexicon following similar rules as those of PIE. A Chinese character consists of
one or more components put together in various ways in a typically square-shape
format. As in most PIE, it is not an arbitrary process how certain components are
combined to form new words or new meanings. Based on a printed posthumous
work of a Chinese scholar Xu Shen (86 BC), the Shou Wen Jie Zi, (―Explains Simple
Characters and Compounds‖) that was published in cir. 121 AD, there were10516
characters arranged under 534 to 544 primitive symbols which are the origin of
the 214 radicals used today. The most common way of forming characters is to
combine a radical component that stands for meaning and a component that
stands for sound. This phono-semantic principle created nearly 95% of
commonly used characters in modern Chinese (Dictionary of Chinese Character
Information, 1988). This high rate reveals the significance of radicals in Chinese
characters. Most of them are of pictograms, which as indicated in Shou Wen Jie Zi,
can be divided into 1) Cosmology and Geology, 2) Plants, 3) Zoology, 4) Human
Body Parts, 5.) Artifacts and Other Man-made Objects, 6) Clothing, and 7)
Housing and Shelters. The radicals found in the current frequent words from
both Taiwan and mainland China, of course, exceed these pictograms. Some
radicals may fall under other categories not mentioned in Shou Wen Jie Zi such as
Colors and Shapes, and different scholars may come up with slightly different
groupings. (Zhou, 2012). Such groupings do not represent arbitrary divisions of
the world; they in fact converge on the cognitive capacities of the human mind.
These concepts are all based on cognitive salient prototypes the speakers of a
community; these are folk categories rather than scientific ones (Ungerer &
Schmid, 1996, p.19).
The radical, together with other components of a character, through metaphor or

metonymy, give rise to the meaning of the character. Take the radical which is
also a character, 心 xin, as an example. It is a pictogram of a human heart. The
semantic analysis of the word (Hu and Fong, 2010) supports a prevalent
metaphor that is HEART IS THE SEAT OF EMOTION, which in turn gives rise to
numerous characters denoting various emotions. Consider these words: 怒
―anger‖,悶 ―pent-up anger, depressed, ‖愁 ―sadness‖, and 恨 ―hate‖ as well as
憤 ―anger‖ (radical of the last two words is 忄, a stylized 心), which are all
composed of a semantic part and a phonetic part. However, the phonetic part is
often necessary for meaning to emerge (Ma, 1997). The phonetic part, 奴 means
―slave‖, so together with the metonymic concept of XIN STANDS FOR
EMOTION, it requires the understanding of this concept, SOMEONE IS NOT IN
CONTROL OF HIS/HER EMSOTION, through a metonymic process so as to
arrive at the interpretation of anger. Another case in point is the word 恨 hèn. It
consists of 艮 which means ―tough, leathery, or blunt‖ for sound and 忄 for
meaning, and it is necessary to blend these two concepts to arrive at the
understanding that TOUGH EMOTION IS HATE. A further example that the
phonetic component of such compounds is necessary for meaning to emerge is,
for example, when the phonetic part 艮 gěn (the sound hèn might have evolved
from gěn) was used to compose new words. A case in point is 狠 hěn,
―cruel,‖(compared with 恨 hèn) where the radical 犭 stands for four-legged

165
animals. Underlying the overall meaning is the metaphor: HUMANS ARE

ANIMALS, and the understanding that four-legged animals are usually fierce, so
when this feature is mapped onto a human, we come to the understanding of
―cruelty.‖ These examples also transpire the process of lexicalization of written
Chinese through metaphors and metonymies.
Radicals Teaching in TCFL

Several studies have examined the role of radicals in teaching writing and
reading Chinese characters to SL and FL learners of Chinese (cf. Shen, 2000, Shen
and Ke, 2007, Wang and Koda, 2013). They have invariably demonstrated the
effectiveness of teaching radicals through the semantic cues embedded in them to
help recognize and retain words over more traditional methods such as rote
learning. It was also established that semantic cues work best when the meaning
of the word has direct connection with the radical; these characters benefit most
from explicit teaching of semantic cues. (Dunlap et al. 2011)
In short, there are characters that are more transparent than others in terms of the
predictability of the radical. The less transparent characters are those whose
radicals are not directly related to their overall meaning. For example, Dunlap et
al investigated a cluster of words that have the radical 禾 ‗grain, rice plant,‘ and
they listed some characters that are not directly related to ‗grain‘ which include
稅 ‗tax‘, 稱 ‗to weigh, to call‘, and 稍 ‗a little bit‘; they are supposedly more
difficult to learn and recall. However, as Zhou‘s (2013) study of the
radical/character 土 ‗earth, soil‘ in Shou Wen Jie Zhi demonstrates, it was
through the principles of CMT (and semantic field) that words related to 土 had
emerged. Hence, on a closer inspection, applying CMT in examining the
etymology of those words of 禾, one would find that they are still related to rice
grain through various degrees of metaphorical and metonymic extension. For
instance, 稅 is a combination of 禾 and 兑 ‗exchange‘, so considering the
importance of agriculture in ancient Chinese society, using rice grain for tax
payment was probably practiced in those days. In this case, 禾 stands for money
or commodity used to pay for taxes—a metonymic principle made sense in a
cultural context.
Shen (2004) espouses deep learning which is using semantic cues in teaching
CFL; therefore, it would be of interest to investigate the role of metaphor and
metonymy in even deeper learning. The challenge is how to make metaphorical
and metonymic clues accessible to learners so that they can become teachable and
learnable? The study discussed below explores this issue so as to answer the
research questions raised earlier.
3 The Study
Experiment Material: CRILL
Based on the theoretical framework of embodied cognition, folk categories,
metaphor, and metonymy as discussed above, an on-line, self-learning,
asynchronized course CRILL (Chinese Radicals Incorporated Language
Learning) for learning the Chinese writing system and culture for English
speakers was designed between 2008 and 2010. Since September 2010, it has been
made accessible to students who enrolled at the university in northern Taiwan

166
where the researcher works. It is accessible through an intranet platform,

Campus (http://campus.viainno.com/campus/Homepage.aspx), that was
developed and maintained by the digital learning center of the university who
was also a partner in the development of CRILL. CRILL was designed to
introduce Chinese culture through the writing system to novice adult learners in
college. As most FL learners find Chinese writing difficult, they tend to find
learning it rather daunting. Even while learning they find the experience
frustrating as there are few rules for them to follow. CRILL therefore aims to help
overcome these experiences by illustrating the universal as well as certain
cultural-specific concepts in the formation of Chinese characters. Hence,
characters are not grouped together by the number of strokes or sound. In
tandem with the folk taxonomy found in the category system of Chinese radicals
discussed previously, the characters are organized in accordance with the
concepts the radicals share, starting from universal ones such as human body
parts, to concepts such as nature and plants which can be regional and culturally
specific, having been derived from Chinese geography and climate.
In view of this principle, there are fifteen units with the first seven units dealing
with human body parts (outer and inner organs), followed by four units with
nature, and the last four with plants. The figure (Figure 1) below shows the table
of contents of CRILL as found on the website.
Figure 1 Page of Contents for Unit 3 in CRILL
Under the heading ―Body‖, learners will find the radicals for eye 目, nose 耳,
mouth ロ, hand 手, foot 足, heart 心, and flesh 肉/(月?), whereas radicals for sun
日, moon 月, mountain 山 and water 水/氵 are under ―Nature‖, and radicals for
bamboo 竹, wood 木, grass 草, and rice 米 are under ―Plant‖. These radicals were

167
chosen as they represent the most common concepts and thus have generated a
rich vocabulary in Chinese with many of it suitable for beginners.
Each unit comprises of 12 to 13 sections. The following figure (Figure 2) shows

one of the units, Unit 3 – Foot, as an example. The menu on the left gives learners
a clear idea what the contents are.
Figure 2 Lead-in Activities for Unit 3 in CRILL
Each unit is divided into three phases of learning: lead-in activities (Section 1 to
5), core learning materials (6-10), and post-learning exercises (11-13). As can be
seen, the lead-in activities include 1) a list of learning goals, 2) warm-up activities
that ask learners to think iconically about a body part or a natural/artificial
object, 3) matching pictograms with the radical/character of the unit, 4)
animation of the evolution of the radical/character, and 5) the recognition of the
radical/character among various characters. The figure below (Figure 2) shows
what learners see when one clicks on Section 4 for the historical evolution of 足
(the foot) and 走 (to walk), two radicals which are characters as well for the same
body part, ―foot‖, in Unit 3.

168
Figure 3 Evolution of Chinese Characters of 足 and 走
The second phase of learning (Sections 6 to 10) involves several compound words
as well as fixed expressions that are commonly associated with the radical(s) of
the unit. These phrases, with some being polysemous, are a mixture of concrete
and abstract meanings so that learners can see the role metaphor and metonymy
play in meaning extension. For example, in Unit 3 as seen in Figure 2, two
radicals which are characters as well are introduced:足 and 走 with the former
representing the physical body part, foot, while the latter represents the motions
that are in tandem with the foot. In short, the same body part gives rise to two
related concepts represented by two slightly different icons. When 足 (the
physical foot) functions as a radical that helps create further semantic items, it is
written as 𧾷, which can be seen in many motion verbs that involve various
actions involving the foot such as 踢 (to kick), 跑 (to run), 蹲 (to squat), and 跳
(to jump). Nouns such as 路 (the road), 跡 (track or trace) are semantic items
extended from various interactions of the foot with certain objects. All these
words are phono-semantic compounds encoding concepts that are, according to
CM theory, metonymic, namely BODY PART STANDS FOR ACTION, and
ACTION STANDS FOR CONCEPT. They are relatively concrete. However, 足 is
polysemous, like most words in all languages. One of its senses which is more
abstract in fact means ―satisfied‖, deriving from the metaphor BODY IS A
CONTAINER, so expressions such 足够 (enough, sufficient) and 滿足
(satisfied, content) capture this metaphorical sense.
Together with these phrases, there are also sentence patterns and sentence
building activities included in these sections that help to provide some kind of
context for association. Both phrases and sentences have all been controlled in
terms of frequency and familiarity for the beginner‘s level. The final phase
consists of post-learning exercises which usually uses songs, poems, or nursery
rhymes that are associated with the radical/character of the unit (Section 11).
Section 12 provides exercises with feedback for learners to gauge their own
learning outcome. Finally, each unit ends with an idiom that contains the
radical/character of the unit with a story explaining the origin of the idiom.

169
Procedure
In order to ascertain the perceived efficacy of CRILL and answer the questions
raised earlier, a survey consisting of five open-ended questions was designed and
distributed from 2012 to 2013 to twenty-nine international students who were
studying at the university where the researcher works. They all enrolled in the
Fall-semester course entitled ―Cross-cultural Learning‖ the researcher offered.
CRILL was an integral component, among other course materials, of the course
syllabus, and it was assigned as self-study homework over six weeks each
semester after the mid-term exam. Prior to that, participants had been taught
about the concepts of metonymy and metaphor existing in all languages and they
were assigned to specific tasks in identifying those found in their own language.
They would present their findings in class so that they could also make some
cross-cultural comparisons. Following that, the organization of Chinese radicals
and the metaphorical and metonymic clues in Chinese radicals/characters were
incorporated. These exercises were meant to prepare them for CRILL. Once they
started with CRILL, they could decide when and how long they wanted to spend
on CRILL. They were encouraged to raise questions in class should they have
encountered any issues during their self-study. The website is equipped with a
log recording the frequency and time they actually spent on CRILL, although this
data was not taken into account in the final analyses.
At the end of the course, which lasted sixteen teaching weeks in total,
participants would take a test on various course materials and CRILL. At the end
of this test, a five open-question questionnaire was administered to investigate
the efficacy of CRILL. By completing these questions, participants would receive
extra points for the test.
Participants
The students that enrolled in the ―Cross-cultural Learning‖ course from 2012 to
2013 came to Taiwan either as exchange students staying six months to one year,
or as international students pursuing an undergraduate degree at the university.
Their Chinese proficiency would be considered pre-intermediate at the time of
enrolment, although speaking Chinese fluently was not a prerequisite for
attending the course since the course was mostly conducted in English.
Nonetheless, they all had previous Chinese writing experience before the course
started.
Their writing experience differed according to the region they grew up in and the
language they speak at home. Among the twenty-nine students, thirteen of them
speak Indo-European languages coming from Europe and the Americas (N=13),
nine of them speak Japanese coming from Japan (N=9), while seven of them come
from other Asian countries (OACs, N=7) including Korea, Malaysia, and
Vietnam. Such grouping is of particular interest when considering that Japanese
learners learn Kanji –a script based on Chinese characters – at a young age.
Would Japanese learners also find the radical based learning, as put forward in
CRILL, beneficial to them? In short, would early and long exposure to Chinese
writing make it easier or more difficult in understanding the explicit knowledge
for the formation of Chinese characters compared with speakers of different

170
linguistic backgrounds? Although the number of participants for each group in

this study was few, it still gives us some preliminary indications.
Students’ Perceptions: Questionnaire

In order to address the research questions for the study, a questionnaire with the
following five questions were designed and administered: A. What did you not
know about Chinese language before starting with CRILL? B. What did you not
know about Chinese culture before starting with CRILL? C. What do you like
most about CRILL, for example, in what ways has it been helpful for your
learning/reviewing? D. What do you NOT like about CRILL? In what ways has it
been confusing and hindered your learning/reviewing? And E. Any suggestions
you may have in order to make this platform better from a learner‘s point of
view?
All questions address the research questions one and two, namely whether the
design of CRILL can benefit and motivate learners learning Chinese characters
and culture, with questions D and E focusing on their critical comments on
CRILL. As for research question three, regarding learners‘ language background
and their evaluation of CRILL, data elicited from the five questions (A, B, C, D,
and E) in the questionnaire were further analyzed according to the participants‘
region of origin. The responses for each question were categorized and coded to
be calculated in terms of percentage, so as to yield an overview of the
participants‘ experience and evaluation of CRILL. For question A regarding what
they hadn‘t known about Chinese language before they started with CRILL,
participants‘ answers were able to be grouped into the following three codes: 1.
No familiarity with the radicals, 2. Having familiarity with the radicals, and 3.
Having some familiarity with the radicals. For question B, addressing what they
hadn‘t known about Chinese culture before they started with CRILL, the coding
was as follows: 1. Culture and idioms, 2. Culture and characters, 3. Neither of the
above, 4. Festivals/culture, and 5. None. For Question C, which asked what they
liked most about CRILL, for example, in what ways it was helpful for their
learning/reviewing, their responses were categorized into: 1.
Sentence/grammar, 2. Characters, 3. Idioms, and 4. Games/songs.
Responses from participants for question D and E received similar coding as they
address related issues. Hence, the coding for question D is as follows: 1.Too easy,
2.Confusing translation, pinyin, or pronunciation, 3. Repetition, 4. Insufficient
feedback, reading and composition input, 5. Insufficient examples (for the lexical
items and sentence patterns taught in each unit), 6. Silly, 7. Technical issues, and
8. No problems. Similarly, the coding for question E were: 1.Too easy-should
have more levels, 2. Correct or Confusing translation/pinyin/pronunciation, 3.
Reduce repetition, 4. Give feedback; more reading and composition input, 5.
Create more linguistic examples, 6. No songs, 7. Improve technical issues, 8. No
problems, and 9. No change. With these codes, it was possible to measure some
tendency in terms of percentage.

171
Analysis and Results

Overall results. Firstly, most of the participants (66%) were unfamiliar with how
Chinese radicals are organized before they started with CRILL, despite the fact
that they had all learned Chinese for several years. 10% of them reported some
familiarity, while 24% good familiarity. This finding is further corroborated by
the results of question B, in which nearly half of the subjects (45%) did not have
any idea about the connection between culture and the formation of characters.
This number could be higher when some subjects (7%) who claimed not to have
any idea about the cultural meanings contained within idioms or characters
before starting CRILL are also included. Nearly one third of them had no
knowledge of the connection between festivals and culture (31%), and less than a
quarter of them did not know the connection between culture and idioms (17%).
Although most participants were not familiar with the way radicals and
characters were presented in CRILL, they liked and considered such design
helpful to their learning with 69 percent of the participants expressing positive
perception to its design, in contrast to other components (See table 1).
Table 1
Overall Results for Question C: What do you like most about CRILL?
Coding N Percentage
1 29 11%
2 29 69%
3 29 9%
4 29 11%
Note: 1= Sentence/grammar 2=Characters 3=Idioms 4=Games/songs
The results of questions D and E (Tables 2 and 3) provide some further insight
into participants‘ criticisms and suggestions. The results of question D (Table 2)
show that the level of difficulty in materials ranks as the highest complaint (28%),
followed by ―inconsistent translation and pronunciation‖ (17%). ―No feedback
/reading/composition input‖ and ―insufficient examples for the lexical items
and sentence patterns‖ (10% for each respectively) were also among the major
complaints with ―technical issues‖ being another one (10%). A small number of
participants did find the design somewhat ―boring and childish‖, specifically
referring to the songs and nursery rhyme parts (7% and 3% respectively).
However, 10 percent of them found no major problems in the design of CRILL.
As for suggestions for improvement (Table 3), participants ranked having ―more
feedback/reading/writing practices‖ (29%) as the most important, followed by
having ―technical issues corrected‖ (24%). A small number of them (7%) gave the
suggestion of ―adding higher levels to existing CRILL curricula‖ in the future.
Participants considered ―reducing repetition and improving consistent
translation and pronunciation‖ more important than ―adding higher levels‖ (17%
and 14% respectively). A very small number of participants (3%) would have
liked to see ―some more linguistic examples‖ to be added to either fixed
expressions or sentence patterns.

172
Table 2
Overall Results for Questions D: What do you NOT like about CRILL?
Coding N Percentage
1 29 28%
2 29 17%
3 29 7%
4 29 10%
5 29 10%
6 29 3%
7 29 10%
8 29 17%
Note: 1=Too easy 2=Confusing translation/pinyin/pronunciation 3=Repetition
4=Insufficient feedback, reading and composition input 5= Insufficient examples
6=Silly 7=Technical issues 8=No problems
Table 3
Results of Question E: Suggestions to improve CRILL
Coding N Percentage
1 29 7%
2 29 14%
3 29 17%
4 29 29%
5 29 3%
6 29 0%
7 29 24%
8 29 3%
9 29 3%
Note: 1=Have more level 2=Correct confusing translation/pinyin/pronunciation
3=Reduce repetition 4=Give feedback; more reading and composition input 5=
Create more linguistic examples 6= No silly songs 7=Improve technical issues
8=No changes
Results by Group. When the results presented above are broken down into
region, with participants‘ language background taken into consideration, the
individual picture for each region resembles somewhat that of the overall results.
However, there are some minute differences which can help answer research
question three, regarding whether participants‘ language background would
affect their evaluation of CRILL and their motivation to learn Chinese.
Firstly, although Japanese learners are exposed to Chinese characters in Kanji at

an early age, the number of participants unfamiliar with the metonymic and
metaphorical nature of Chinese character formation and the cognitive
categorization of radicals is nearly the same as those speaking European
languages such as French, German, and Spanish (69% vs. 67%). Even the
participants from other Asian countries (OACs), some of whom were heritage
learners, reported a lack of knowledge in a percentage similar to those from
Europe/Americas (57% vs. 69%). However, a small number of participants from
Europe/Americas and OACs said they were very familiar with the principles
behind radicals and characters (15% and 14% respectively) before they started

173
with CRILL, whereas none from Japan claimed so. On the other hand, about one
third of the participants from Japan (33%) said that they were somewhat familiar
with radicals/characters, slightly more than those who claimed so in the OACs
group (28%). Only 15% percent of the participants from Europe/Americas
claimed to be somewhat familiar with the radicals and characters.
In terms of cultural knowledge encoded in the radicals/characters, most

European language speakers (69%) and a good number of participants from
Japan (44%) claimed what they learned in CRILL was new to them. However, if
we also include the 15% of European languages speakers who claimed not to be
familiar with the connections between idioms and characters with culture, the
number of participants who were ignorant of the connection between language
and culture in this group is much higher than that of the Japan group. Only a
small number (14%) from OACs claimed they were unaware of the connection
between culture and radicals/characters. Furthermore, about half of the
participants from Japan and OACs said they were not familiar with the cultural
meanings of the festivals introduced in CRILL, while a fairly small number of
participants from Europe/Americas reported such a lack of knowledge (7.5%).
Not surprisingly, all participants from Europe/Americas (100%) reported a

favorable perception of the presentation of radicals/characters in CRILL and
found this design beneficial to learning, followed by a high percentage of
participants from OACs who said so (85%). About half of the participants from
Japan (55%) also liked and were motivated by such a design, with the other half
split between grammar practice (22%) and games/songs (33%). A relatively
smaller number of participants from the OACs group liked and were motivated
by grammar practice (14%) and games/songs (14%), while a very small number
to none of the participants from Europe/Americas considered these two
components beneficial and motivating (7.5% and 0% respectively).
As for criticism, most of the Europe/America and Japan groups thought the
skill-level taught in CRILL was too easy for them (38% and 33% respectively),
whereas none of the participants from OACs said so. Similarly, the OACs group
had the highest number of participants who did not find any major issue with the
methodology of CRILL (42%). About a quarter of the participants in the Japan
group (22%) also found no serious issues with CRILL. However, none of the
participants in the Europe/Americas group expressed such an evaluation.
When we consider the ranking of the criticisms by group, it becomes clear that
both the Japan and OACs groups had different emphases on what was missing
when compared to the Europe/Americas group. The latter considered ―not
having feedback for exercises and no reading/composition input‖ (coding 4) a
big drawback, whereas the former two groups did not share this criticism at all.
They instead thought that the ―number of examples for the phrases and sentence
patterns were insufficient‖ (coding 5) and more of a hindrance to their learning.
Lastly, all three groups believed that there should be ―more feedback with their
exercises with additional reading/composition input and practices‖ (coding 4)
along with ―improving technical malfunctions‖ (coding 7). They also seemed to

174
agree that ―repetition needed to be reduced‖ (coding 3). The suggestions for
other issues are less unanimous.
4 Discussion
Benefit and Motivation
Although the analyses presented here are confounded by the small sample size of
subjects and the preliminary nature of this study, some tentative observations
can still be drawn. Firstly, in answering the research question one and two, when
the overall data without group division being examined, it is safe to argue that
the cognitive approach to teaching Chinese radicals/characters explicating some
metonymic and metaphorical principles of word formation can benefit and
motivate the learning of Chinese writing and culture.
Participants‘ overall perception indicates that CRILL was very favorably

received, particularly with respect to how radicals/characters were organized
and introduced, even though such learning approaches were new or partially
new to them. These results lend a positive support to the pedagogical application
of integrating cognitive linguistic theory into teaching Chinese as a foreign
language. Cognitive approaches that incorporate metonymic and metaphorical
knowledge with a touch of etymology (Boers et al., 2007) in language teaching
has been found to be beneficial and motivating in the learning of English
vocabulary (Boers, 2004; Boers and Lindstromberg, 2009), idioms (Hu & Ho,
2009), prepositions (Hu & Fong, 2010; Tyler and Evans, 2004), and phrasal verbs
(Abreu & Vieira, 2009; Yasuda, 2010). There have been even studies and research
in FL and SLA that call for the overall promotion of metaphorical competence in
foreign language learning (Littlemore & Low, 2006).
There is substantial empirical evidence in support of the efficacy of explicating

the semantic cues in the radicals for Chinese characters recognition and retention
(Shen and Ke, 2007, Wang and Koda, 2013). However, none of these studies
examined the role of metaphor and metonymy in word meaning, formation of
compound words, and polysemy. There are also very few extensive teaching
materials based on semantic cue nor cognitive approach in the field of TCFL.
In light of this, the design of CRILL is making an important step in the direction
of developing efficient approaches and teaching materials to teaching the
Chinese writing system. Similar to semantic cues, CRILL seeks to guide learners
with patterns and principles so that learning to write and read is not an arbitrary
and mundane task. Although semantic cues are useful in recognizing and
predicting the meaning of characters, they work best in transparent characters.
However, the approach applied in CRILL went further by incorporating
metaphorical and metonymic clues, so that hopefully less transparent characters,
compound words, and polysemy can be better explained, thus, recognized and
retained. Above all, cognitive approach highlights the rich cultural background
encoded in the characters and compound words/phrases. If teaching a language
is concerned with passing on cultural knowledge at the same time, CRILL is
certainly more satisfying in this respect and the results of the study arguably
support this assertion.

175
Language Background
In addressing whether the linguistic background or any previous knowledge in
the cognitive nature of Chinese radicals/characters could motivate and benefit
learners or not, the results suggest that Indo-European speakers from Europe and
the Americas tended to enjoy the cognitive method provided in CRILL more than
Japanese speakers. Even speakers from OACs considered this approach more
positively than those from Japan. This finding came as a surprise when the
number of participants in each group who had not had any knowledge of
Chinese radicals before their participation of the CRILL program is fairly
comparable.
According to the Japanese participants, they had to learn Kanji from a young age
but had never been taught explicitly about the cognitive principles involved in
the composition of characters. Despite a lack of prior knowledge, they did not
find this approach as engaging and motivating as their counterparts in the
Europe/Americas group and OACs. At this point it is difficult to determine
whether the difference in attitude is because of a language issue, that is, the
familiarity with Kanji which could give participants the impression that these
cognitive clues encoded in the radicals/characters are not that challenging or
interesting. This observation is supported by their own admission in class to the
researcher that they found Chinese (character) writing relatively easy, while
learners from Europe/Americas found the opposite. The latter considered
speaking easier than writing, whereas the former regarded speaking harder to
master.
There could be other factors that contributed to the difference. As CRILL is in

English, and the classroom language was also English, with learners from
Europe/Americas having higher English proficiency than those from Japan and
OACs, there was a possibility that learners from the former group enjoyed the
approach more because they understood more. The other factor could be learning
style, with Japanese students considered to be more closure-oriented (Hansen &
Stanfield, 1981: Joy & Kolb, 2007; Oxford, 1990), and thus, it was possible that
they enjoyed holistic teaching less.
Criticism of CRILL
Judging from participants‘ criticism and suggestions, they tended not to be
satisfied with general on-line language learning and the technical issues
associated with it. Research has shown in several cases, the efficacy of on-line
learning over traditional face-to-face classrooms, at least in higher education
settings (Xu & Jagger, 2013). However, failure does occur when inadequately
equipped e-learning systems are implemented (Hara & Kling, 2000; Zhang, et al
2004). The results from question D and E reflect some of the key challenges many
on-line language learning tools are facing nowadays: feedback and technical
issues. For language learners using an interactive, self-learning, and
asynchronized on-line learning tool, it is frustrating when they are unable to
check their own input. Although CRILL is equipped with some feedback
mechanism for character writing and vocabulary as well as grammar practice, the
technology involved is fairly basic and breaks down occasionally due to the

176
limitation of available technology and funding. These issues can certainly create
frustration.
Despite these issues, it is encouraging to learn that participants‘ overall

perception of CRILL in terms of radicals/characters and how they compose other
characters and vocabulary was positive. Participants also enjoyed the
incorporated cultural learning. In light of these indications, the design of CRILL
has achieved its primary goal which was to illustrate Chinese cultural beliefs and
values through its writing system as an introductory course to beginners. It,
however, may fall short as a full-fledged language learning program.
Limitations and Implications of the Study

As the sample size of each group is not always comparable, with OACs
comprising the smallest group (N=7), it was difficult at times to determine
whether the difference observed was due to individual preference or affected by
other variables such as language, learning style and culture. Despite the fact that
in some cases, there seem to be a difference between participants from the
Europe/Americas and those from Japan, it was quite difficult to measure any
effect from the OACs group as their number was too small. Therefore, any
observation regarding this region is tentative.
Furthermore, the study could not provide a definite answer to the question of the
efficacy of CRILL as pre- and post-test were not administered. It also did not
examine how the CRILL approach can facilitate greater learning in writing and
reading Chinese characters. The study, at most, examined the perceptions of the
participants based on self-reporting, and it is also not clear whether the learning
approach adopted in CRILL is more effective than other approaches as there
were no control groups. Better experiment designs are undoubtedly required for
any future research.
The pedagogical implications for this study are crucial for effective learning
approaches and on-line tools in the future. In fact, a new Chinese on-line learning
website has been under development that has sought to rectify the shortcomings
of CRILL while also continuing to develop its positive features. These
undertakings hope to demonstrate the importance of combining sound theories
with viable practices in language teaching and learning.
5 Conclusion
This study set out to explore the viability of a teaching approach based on the
linguistic insight gained from CL in recent years. The results so far can establish
its overall merit in the field of TCF by providing a holistic view of the Chinese
writing system and in what way it is deeply rooted in the social and cultural
worlds of the people in the Chinese speaking communities. The study
demonstrates that understanding metaphor and metonymy in lexicon extension
can not only enhance the learning of Chinese characters but also promote the
understanding of the social and cultural knowledge encoded in them. Such
conclusions certainly require caution as it was found that this approach may suit
learners differently. The learner differences could be partly individual or partly
cultural. As CRILL was originally designed with American and European

177
learners in mind, it is of great interest to find that the Japanese learners in this
study were not as motivated by the approach as their European and American
counterparts. This finding is of value for any future development of teaching
materials and pedagogy.
Acknowledgments
This design and study of CRILL was funded by the Ministry of Science and
Technology (MST), Taiwan (99-2631-S-032-001). Any opinions, findings,
conclusions, or recommendations expressed in this material are those of the
author and do not necessarily reflect the views of MST.
References
Abreu, A. S., & Vieira, S. B. (2009). Learning phrasal verbs through image schemas: A
new approach. Retrieved, from http://ssrn.com/abstract=1491689
Barcelona, A. (2000). On the plausibility of claiming a metonymic motivation for
conceptual metaphor. In A. Barcelona (Eds.), Metaphor and metonymy at the crossroads
(pp. 31-58). Berlin: Mouton de Gruyter.
Boers, F. (2004). Expanding learners’ vocabulary through metaphor awareness: What expansion,
what learners, what vocabulary?. In M. Achard & S. Niemeier (Eds.), Cognitive
linguistics, second language acquisition, and foreign language teaching (pp. 211-232).
Berlin, New York: Mouton de Gruyter.
Boers, F., Eyckmans, J., & Stengers, H. (2007). Presenting figurative idioms with a touch of
etymology: More than mere mnemonics? Language Teaching Research, 11, 43-62.
Boers, F., & Lindstromberg, S. (2009). Optimizing a lexical approach to instructed second
language acquisition. Basingstoke: Palgrave Macmillian.
Casasanto, D., & Boroditsky, L. (2007). Time in the mind: Using space to think about time,
Cognition, 106(2), 579-593. doi:10.1016/j.cognition.2007.03.004
Dictionary of Chinese Character Information [汉字信息字典] (1988). Shanghai, China:
Science Press [科学出版社].
Dirk, G. (Ed.). (2006). Cognitive linguistics: Basic readings. Berlin: Mouton de Gruyter.
Dirven, R. (1985). Metaphor as a basic means for extending the lexicon. In W. Paprotte &
R. Dirven (Eds.), The ubiquity of metaphor: Metaphor in language and thought (pp.
85-119). Amsterdam: John Benjamins Publishing Company.
Dunlap, S., Perfetti, C. A., & Liu, Y. (2011). Learning vocabulary in Chinese as a foreign
language: Effects of explicit instruction and semantic cue of reliability. Retrieved
from: http://www.pitt.edu/~perfetti/PDF/DunlapLearningVocabulary.pdf
Goossens, L. (1990). Metaphtonymy: The interaction of metaphor and metonymy in
expressions for linguistic action, Cognitive Linguistics, 1(3), 323-340.
Györi, G. (1996). Historical aspects of categorisation. In E. H. Casad (Eds.), Cognitive
linguistics in the redwoods (pp. 175-206). Berlin, New York: Mouton de Gruyter.
Györi, G. (1998). Semantic change, semantic theory and linguistic relativity. Duisburg:
L.A.U.D.
Hansen, J., & Stanfield, J. B. (1981). The relationship of field dependent independent
cognitive style to foreign language learning achievement. Language Learning, 31,
349-369.
Hara, N., & King, R. (2000). Students' distress with a Web-based distance education
course: an ethnographic study of participants' experiences. Information,
Communication and Society, 3(4), 557–579.
Hu, Y. H., & Fong, Y. Y. (2010). Obstacles to CM- guide L2 idiom interpretation. In S.
Knop, F. Boers & A. Rycker (Eds.), Fostering language teaching efficiency through
cognitive linguistics (pp. 293-316). New York: Mouton de Gruyter.

178
Hu, Y. H., & Ho, Y. C. (2009). Prepositions we live by: Implications of the polysemy
network in teaching English prepositions in and on. In B.
Lewandowska-Tomaszczyk & K. Dziwirek (Eds.), Studies in cognitive Corpus
linguistics (pp. 336-370). Frankfurt: Peter Lang Verlagsgruppe.
Johnson, M. (1987). The body in the mind. Chicago: The University of Chicago Press.
Joy, S., & Kolb, D. (2007). Are there cultural differences in learning style? International
Journal of Intercultural Relations, 33(1), 69-85.
Lakoff, G. (1987). Women, fire and dangerous things. Chicago: University of Chicago Press
Lakoff, G., & Johnson, M. (1980). Metaphors we live by. Chicago: University of Chicago
Press.
Lakoff, G., & Johnson, M. (1999). Philosophy in the flesh: The embodied mind and its
challenge to western thought. New York: Basic Books.
Littlemore, J., & Low, G. (2006). Metaphorical competence, second language learning,
and communicative language ability. Applied Linguistics, 27(2), 268-294. doi:
10.1093/applin/aml004
Ma, K. Y. (1997). The relation between the writing system and the use of metaphor in English
and Chinese (Unpublished master‘s thesis). University of North Dakota, Grand
Forks, ND.
Oxford, R. L. (1990). Missing link: Evidence from research on language learning styles
and strategies. In J. E. Alatis (Ed.), Linguistics, language teaching acquisition: The
interdependence of theory, practice and research (pp. 438-460). Georgetown University
Round Table on Language and Linguistics, 1990.
Radden, G. (2000). How metonymic is metaphor. In A. Barcelona (Eds.), Metaphor and
metonymy at the crossroads (pp. 59-78). Berlin: Mouton de Gruyter.
Ruiz de Mendoza, F. J. (2000). The role of mapping and domains in understanding
metonymy. In A. Barcelona (Eds.), Metaphor and metonymy at the crossroads (pp.
109-132). Berlin: Mouton de Gruyster.
Shen, H. H. (2000). Radical knowledge and character learning among learners of Chinese
as a foreign language. Linguistic Studies, June, 85-93.
Shen, H. H. (2004). Level of cognitive processing: Effects on character learning among
non-native learners of Chinese as a foreign language, Language and Education, 18,
167-182.
Shen, H. H., & Ke, C. (2007). Radical Awareness and Word Acquisition Among
Nonnative Speaker of Chinese. The Modern Language Journal, 91(1), 97-111.
Sweetser, E. (1990). From etymology to pragmatics: Metaphorical and cultural aspects of
semantic structure. Cambridge: C.U.P.
Talmy, L. 1988. Force dynamics in language and cognition. Cognitive Science, 12, 49-100.
Tyler, A., & Evans, V. (2004). Applying cognitive linguistics to pedagogical grammar:
The case of over. In M. Achard & S. Niemeier (Eds.). Cognitive
linguistics, second language acquisition, and foreign language teaching, (pp. 257-280).
Berlin: Mouton de Gruyter.
Ungerer, F., & Schmid, H-J. (1996). An introduction to cognitive linguistics. London:
Longman.
Wang, J., & Koda, K. (2013). Does partial radical information help in the learning of
Chinese characters? In E. Voss, S-J D. Tai & Z. Li (Eds.), Selected proceedings of the
2011 second language research forum: Converging theory and practice (pp. 162-172).
Somerville, MA: Cascadilla Proceedings Project.
Xu, D., & Jaggars, S. S. (2013). The impact of online learning on students‘ course :
Evidence from a large community and technical college system. Economics of
Education Review, 37, 46-57.
Xu, S. (121 AD). Shou wen jie zi [Text in Chinese].
Yasuda, S. (2010). Learning phrasal verbs through conceptual metaphors: A case of
japanese EFL learners. TESOL Quarterly, 44(2), 250-273.

179
Zhang, D., Zhao, J., Zhou, L., & Nunamaker, J. (2004), Can e-learning replace classroom
learning? Communications of the ACM, 45(5), 75-79
Zhou, L. J. (2012, May). The teaching of Chinese characters—A fun way of learning
characters to decode Shou wen jie zhi. Retrieved from:
http://mandarin.nccu.edu.tw/data/teacher/pdf [Text in Chinese].
Zhou, Y. L. (2013, May). Understanding the ontology of the radical/character 土
(Soil/Earth) in Shou wen jie zhi through conceptual metaphor theory and semantic
field theory. Paper presented at the 15th Symposium on Chinese Philology in Central
Taiwan 第十五屆中區文字學學術研討會, Chinese Cultural University, Taipei. [Text
in Chinese].

180

Vol. 11, No. 1, pp. 180-199, April 2015
Coaches‟ Perceptions of how Coaching Behavior

affects Athletes: An Analysis of their Position on
Basic Assumptions in the Coaching Role
F. Moen
Norwegian University of Science and Technology,
Trondheim, Norway
R. Giske
University of Stavanger, Norway
R. Høigaard
University of Agder, Norway
Abstract. This study explores coaches‟ beliefs about what they think their
athletes expect from them as coaches in sport. A sample of 36 different
statements representing different opinions about coach behaviours and
how coach behavior affects athletes‟ motivation, performance, focus, and
emotions, was presented to 23 Norwegian coaches working in high
schools specialized for elite sports. The participants were coaches in
various sport disciplines and were asked to consider and rank-order the
statements by using a Q sorting procedure. The authors discuss their
analysis from a Q methodical factor analysis. In general, the coaches
share some common viewpoints that are represented in two different
factors (consensus). Each factor represents congruence views about
expectations in the role as a coach in sport. The dominant view (factor A)
is that coaches believe that their athletes expect involvement leadership,
whereas servant leadership was dominant in factor B; a view that only a
few of the coaches shared.
Keywords: coaching styles, coaching behavior, athletes, sport.
1. Introduction
The question as to what coaching behaviour is constructive in order to develop
the athlete in sport has occupied researchers and practitioners for several decades,
and the influence of the coach on the athletes is well documented (Abraham,
Collins, & Martindale, 2006; Blom, Watson II, & Spadaro, 2010; Côté & Gilbert,
2009; Myers, Chase, Beauchamp & Jackson, 2010). When a coach emphasizes
training and instruction, and gives positive feedback that recognizes and rewards
good performance, athletes are more satisfied with their leadership behaviour
(Chelladurai, 2007). Similarly, a study performed by Moen, Høigaard, and Peters
(2014) found that athletes who were most satisfied with their performance

181
progress evaluated their coaches‟ leadership behavior in accordance with the

guidelines described by Chelladurai (2007) above. From a coaching perspective
athletes‟ performance, behaviour, motivation, and emotional states within sport
is affected of coaches‟ behaviour (Jowett & Cockerill, 2002; Lyle, 1999; Mageau &
Vallerand, 2003; Strachen, Cote & Deakin, 2011). For an athlete to reach their
potential the amount of time spent on deliberate practice is of vital importance
(Ericsson, 2009) as well as the ability to manage or carry out the extensive amount
of practice motivation is necessary (Deci & Ryan, 2002; Ericsson, Krampe, &
Tesch-Römer, 1993). Lyle (2002) suggests that performance coaching embrace an
attempt to control contributory variables and this perspective the coaches‟ role is
to adapt high quality exercise, based on the athlete‟s physically, mentally and
stage of social development. The ability to create a positive coaching process
demands knowledge about the sport (e.g. technical, tactical, physiological, and
psychological), the learning process, and the athletes‟ abilities and potential
(Jones, 2006). Furthermore, good communications skills and the ability to
establish a „productive‟ coach-athlete relationship are also of vital importance
(Jowett & Cockerill, 2003). Moen (2012) claims that the ability to ask open-ended
questions and listen to the athlete is essential in the coaching process in order to
optimally stimulate an athlete‟s intrinsic motivation. This can be considered as
the origin of an optimal athlete-coach relationship based on mutuality (Jowett &
Meek, 2000; Olympiou, Jowett, & Duda, 2008).
Lyle (2002) argues that it is not the individual coach, but the coaching process
that is unique, and that the coaching role is determined by the nature of the
coaching process, which takes place in a personal and social space. Furthermore,
the values, idiosyncrasies and personal qualities of the coach are reflected in the
interpersonal behaviour and the engagement in the coaching process. This is in
line with Horn‟s (2008) working model of coaching effectiveness, which
emphasizes that coaches‟ expectancies, values, beliefs, and goals influence a
coach‟s behaviour. Previous research on teachers‟ beliefs has shown that beliefs
may influence their perceptions and pedagogy in action (Nespor, 1987; Rokeach,
1976). Rokeach (1976) claims that all beliefs are potential predictors to action, and
that both action and beliefs have an influence on one another (Haney, Lumpe,
Czerniak, & Engan, 2002). Both teaching and coaching behaviour reflect values,
which may be considered as means to evaluate the experience. Personal value
judgements about what is appropriate, good, bad, and worthy coaching
behaviours affect the coaching process and reflect core elements in a coaching
philosophy (Lyle, 1999; 2002; Wilcox & Trudel, 1998). Thus, an examination of
beliefs about coaching behaviours delve into the heart of understanding coaches
actions and may contribute to better grasp the coaching practice (Cassidy, Jones,
& Potrac, 2009).
Taken together, a coach‟s main aim is to stimulate the athlete to develop his or
her talent and capabilities by affecting the variables that have an impact on

182
performance (Jowett & Poczwardowski, 2007). Coaches‟ beliefs and values

precursor practice and according to Cassidy et al., (2009), it is essential to know
because it may contribute to explain why coaches do as they do.
In the present study we focus on understanding coaches‟ perception of how their
behavior effect an athlete. More specifically we want to explore the coaches‟
opinion on how and perhaps which coaching behavior affects the athlete‟s
motivation, emotion, performance, and focus. We are particularly interested in
identifying coaches whom express similar subjective views in order to unveil
patterns among the participants. In the present work the Q methodology as a tool
for studying the coaches‟ beliefs was used. Even though this research strategy has
been applied studying teacher beliefs (Lim, 2009; Thorsen, 2009; Øverland,
Thorsen, & Størkesen, 2012) for at least two decades, only few study in sports
science have utilized this approach (Moen, 2012; Moen & Garland, 2012; Moen &
Kvalsund, 2014; Moen & Kvalsund, 2013; Moen & Verburg, 2012). Thus, a second
aim was to clarify if Q methodology is useful for studying coaches beliefs related
to their actions as coaches.
2. Method
A Q methodology was chosen because this methodology in general investigates
subjectivity related to a defined topic (Brown, 1980). Subjectivity in all forms,
including beliefs, views, experiences and opinions, are investigated in Q
methodology (Brown, 1996). The methodological approach is completed through
five tasks: 1) selecting participants, 2) defining a concourse, 3) developing a Q
sample, 4) completing a q sorting, and 5) completing data analysis (Brown, 1996;
Moen & Garland, 2012; Watts & Stenner, 2012).
2.1 Participants
The data in this study was collected from 23 Norwegian coaches (mean 46 yrs.,
range 26 – 64 yrs.). Their average education was 4 years at the University level
with an average of 19 years practicing as a coach. The coaches were recruited
from one high school specialized for various sport disciplines (e.g. cross country
skiing, biathlon, track and field, football, volleyball, and handball). This
particular high school was selected because of its long experience with
developing youth athletes into top international athletes. The coaches work with
athletes ranging from 16 to 19 years old with performance levels varying from
national top level to national top regional level.
2.2 The Concourse

Based on literature, theories, and research within the coaching field in sport
(Chelladurai, 2007; Côté & Gilbert, 2009; Jowett & Cockerill, 2002) we developed
a concourse that consisted of a list of about 80 statements that covered different

183
beliefs about the current research question (Stephenson, 1986). The statements
were written from an athlete‟s point of view: “My coach does not have to be open
for questions.” The concourse was then reduced into a meaningful Q sample in
order to create a balanced sample for stimulating the Q-sorters (coaches) to use
the subjective statements (sample) to rank-order them self-referentially and draw
a picture of their own self-conceived view on the topic (McKeown & Thomas,
1988).
2.3 The Q sample
In the present study, two main themes (what Stephenson, 1950, calls effects)
emerged in the concourse, coach behaviour and effect. Within the theme coach
behaviour three sub-themes (what Stephenson, 1950 calls levels) seemed to be
relevant: the coach‟s decision making style, the coach‟s motivational tendencies,
and the coach‟s instructional behaviour (see Table 1). Within the theme effect four
subthemes emerged: the athlete‟s motivations, focus, performance and emotions
(Table 1).
Table 1
The design of the statements based on coaching behaviour and effect
Levels
Coaching a. coach‟s b. coach‟s c. coach‟s
behaviour decision motivational instructional
making style tendencies behavior
Effect d. athlete‟s e. athlete‟s f. athlete‟s g. athlete‟s

motivation performance focus emotions
The next step is to combine each of the three sub themes of coach behavior with
each possible effect (Moen & Garland, 2012). Each possible combination becomes
a categorical cell and it results in 3 x 4 cells (coach behavior x effects), as shown in
Table 2.
Table 2
The combination of levels in the design
Combination of levels
Coaching a a a a b b b b c c c c
behaviour
Effect d e f g d e f g d e f g
Statement 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12,
No 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24,
25 26 27 28 29 30 31 32 33 34 35 36

184
The authours decided to use three statements that most clearly represented the
viewpoints from the concourse to represent each combination of cells. The final Q
sample resulted in 36 statements (3 x 12) that represent the different
combinations of cells showed in Table 1 and 2 (see Appendix). The statements in
each cell are interrelated and represent the viewpoint of that cell, but each cell
obtains both negative, neutral and positive statements. This is done to ensure
reflections related to the particular viewpoint representing each cell. As shown in
Table 2, the first statements in each cell were allocated numbers from 1 to 12, the
second statements were given numbers from 13-24, and the third statements
were given numbers from 25-36. In this way, it will be more challenging for the Q
sorter (the coach) to understand very clearly how the system is built up (Moen &
Garland, 2012).
2.4 The Q sort

The coaches were invited to voluntarily participate in the study and they were all
gathered in a classroom at this specific school for about 45 minutes. They were
asked to provide their names, age, participated sports, education level and
experience as a coach onto an individual scoreboard. The general data was
gathered to explore if coaches from similar sport background or education shared
similar views about their coaching practice in order to better understand the
emerging factors. The coaches were given a specific condition to reflect upon and
were then asked to take their time to read through all the statements presented to
each of them in the Q sample (36 cards with the different statements from the Q
sample). The coaches were asked to consider what they believed were expected
coaching behavior from themselves as coaches. They were asked to rank-order
the statements in a scoreboard ranging from a score of +5 for “most strongly
agree” to -5 for “most strongly disagree” under the so-called forced quasi-normal
distribution of the statements, as shown Figure 1 below (Brown, 1980, p. 197-198).
most very strongly disagree disagree neutral agree agree strongly very most
strongly strongly disagree agree strongly strongly
disagree disagree agree agree
-5 -4 -3 -2 -1 0 1 2 3 4 5
Figure 1. The scoreboard used in Q methodology

185
The coach is however free to place each statement from the Q sample anywhere
within the distribution, but the participant is forced to keep to the distribution
form in order to make all the necessary nuanced evaluations of the 36 different
statements (Kvalsund, 1998). The statements that are placed on both extreme
ends of the scoreboard, ±5 and ±4, are normally the statements that the
participants have a strong connection with. Statements placed in the middle of
the scoreboard are normally statements they have a more nautral connection
with (Moen & Garland, 2012).
2.5 Q factor analysis

After the all the scoreboards from the coaches who participated in the
investigation were collected, each Q sort (each coach‟s score on the different
statements in the Q sample that is placed on the scoreboard) is entered into a
program that is tailored for Q methodology called PQMethod (Schmolck, 2002).
PQMethod uses factor analysis to analyse congruent scores from different
coaches into factors (Allgood & Svennungsen, 2008; McKeown & Thomas, 1988;
Moen & Garland, 2012; Rhoads, 2007). The aim is not to generalize the findings,
but to find and qualitatively explore the viewpoints related to the different
factors that emerge from the factor analysis. Thus, it must not be mistaken by the
quantitative factor analysis (Brown, 1980). For any n Q sorts, the correlations
from the Centroid factor analysis produce a matrix of 23 x 23 cells (n x n). A high
correlation coefficient indicates that two Q sorts (two scoreboards from two
coaches) are sorted more or less the same way. The next step is to use a Varimax
factor analysis to analyse the correlation matrix in order to find possible factors
across the Q sorts. Different numbers of factors were tested for extraction, but the
initial factor analysis showed that two main factors emerged: one factor had an
eigenvalue (EV) of 9.6 counting for 42 % of the variance, whereas the other
extracted factor had an EV of 1.54, counting for 7 % of the variance. The most
influential factor(s) is the one with the highest EV, and EV is used to decide how
many factors that are going to be extracted in the analysis. If a factor has an EV
higher than 1 that factor is defined as a significant factor (Brown, 1980; Kvalsund,
1998).
After experimenting with various alternatives by Varimax rotation of factors, the
authors decided to consider a hand rotation of factors based upon an unrotated
two-factor solution from the Centroid factor analysis. The main argument for
using this strategy was that the initial analysis revealed a high correlation
between the factors from the Varimax rotation. Factors that are highly correlated
indicate that there is probably only one main factor with which virtually all
participants are associated. After studying the statements that represented the
different factors, a two-factor solution was chosen. From using the unrotated

186
factors as the final solution the overarching consensus of the factors reveals, since
the Varimax rotation is spreading the consensus across the rotated factors, which
causes them to be highly correlated.
3. Results
The two factors that were discovered in this study are the two categories of
beliefs related to the research question among the coaches who participated in
this study (Brown, 2002). Thus, the emerging factors are created and influenced
by the coaches who load on this particular factor. Q methodology uses an
estimate developed by Brown (1980) to decide how high a factor loading needs to
be to contribute to a factor or not (Pett, Lackey, & Sullivan, 2003, p. 208). A factor
loading on a minimum of .41 was estimated in this study to decide if a Q sort (a
coach‟s individual scoreboard) contributed to a factor (Brown, 1980; Kvalsund,
1998). The factor matrix in Table 3 shows that factor A has 16 pure cases (sorts
that load only on one factor) and 21 loadings when mixed cased are included.
Factor B has 2 pure cases and 7 cases when mixed cases are included.
Table 3
The Matrix of Rotated Factors and their Loadings
Q sort Factor A Factor B

1 0.64X 0.00
2 0.50X 0.06
3 0.69X -0.28
4 0.84X 0.04
5 0.36 0.44X
6 0.81X -0.14
7 0.64X -0.09
8 0.55X -0.35
9 0.79X -0.07
10 0.47X 0.26
11 0.69X -0.32
12 0.30 -0.46X
13 0.55X -0.11
14 0.72X -0.51X
15 0.72X -0.51X
16 0.74X -0.44X

187
17 0.64X -0.22
18 0.50X 0.23
19 0.65X -0.39
20 0.41X -0.50X
21 0.53X -0.61X
22 0.64X 0.02
23 0.42X -0.02
Pure cases 16 2
Mixed cases 5 5
% variance explained 38 10
Note: X= significant factor loading. Factor loadings with bold faces are pure
cases loading on a factor, and loadings with italic faces are mixed cases loading on
more than one factor.
As seen in Table 3 all sorts that significantly load on factor A are positive, while
only one of the sorts that load on factor B is positive. Six of the significant
loadings on factor B are negative. The analysis found a negative significant
correlation between factors A and B (-.65).
The statements on both extreme ends of the scoreboard, ±5 and ±4, are the
statements that involves the most reflected beliefs among the coaches. Therefore,
our analysis in this study focuses on the statements on both extreme ends of the
scoreboard (Brown, 1980). Based on characteristic statement, distinguishing
statement and consensus statement we labelled factor A: Involvement leadership
and factor, B: Servant leadership.
3.1 Factor A: Involvement leadership

The most extreme statements loading on factor A on the positive side (+5 and +4)
emphasize the importance of involvement of athletes to affect their motivation,
emotions (such as curiosity and interest), and performance (statement number 1,
2 and 4). The most extreme statements on the negative side (-5 and -4) also
highlight the importance of involvement to affect an athlete‟s emotions (such as
commitment), and focus (statement number 16 and 27) (see Table 5). Finally,
feedback and social support are also emphasized in order to affect performance
(statement number 18). A democratic coaching behavior is when athletes are
invited to participate in coaching decisions together with the coach (Chelladurai,
1989). The involvement leadership factor represents the views of 16 of the
participant coaches. A comparison with the respondents who not loaded on
factor A and the general variables on each scoreboard that documented the

188
coaches coaching experience and education level, shows no differences in

coaching experience or education level between the two groups of coaches. The
strong loading on factor A seems to reflect that coaches consider stimulating
athlete involvement as a core element in their coaching practice. Following
Chelladurai´s (1989, 1990) description of democratic behavior, the loading on
factor A seem to be associated with this dimension affording participation in
athletes‟ decisions in their sport (item 16). However, the views in factor A also
represent a fundamental pedagogical perspective emphasized that athlete
involvement are a major source influencing athlete motivation, curiosity and
performance.
Vygotsky (1978) suggest that all learning and development is based on social
activity and interaction with tools. Coaching effort directed towards stimulating
athlete involvement can be considered as measure that gives opportunities to
reciprocate reflection and enable the coach and the athlete to think and
understand each other. Mediation is a key concept in Vygotsky‟s theory, and the
role of the mediator (coach) is to clarify what kind of involvement is effective or
not in order to enhance performance of the individual (Kozulin, 2003). According
to Kozulin (2003) human meditation seems to be too numerous and context
dependent to allow a simple classification, but belongs to interactive activity. The
participants loading on factor A is a strong disagreement suggesting that neither
feedback nor social support are crucial for their performance in sport indicating
that this coaching behavior is a kind of mediating involvement that enhancing
performance (statement 18).
Table 4
Distinguished Statements Loading on Factor A; Involvement leadership
Number Statement Strength

1 My motivation for training increases when my coach involves me. +5
4 I become curious and interested if my coach involves me in matters +4
concerning my training.
2 If I‟m involved in the process concerning my training I perform better. +4
27 I become stressful if my coach involves me in important matters regarding my -4

training.
18 Neither feedback nor social support are crucial for my performances in sport. -4
16 I become uncommitted if my coach includes me in decisions regarding my -5
sport.
Note: Included mixed cases, 21 coaches loaded on factor A.

189
3.2 Factor B: Servant leadership - caring and non-task controlling

Viewpoints loading on factor B were held of two (pure cases) of the participant
coaches. Compared with the other respondents‟ general variables (sports,
experience and education level) they are both involved in individual sport
(athletic) with a long coaching and teaching experience, but with no difference in
educational background. On the positive side (see Table 6), they emphasize the
importance of not being met by instructive behavior from the coach (statement
number 24 and 33). The coaches who are loading factor B believe that clear
instructions negatively affect motivation and involvement negatively affects
focus (statement number 27). Therefore, this coaching behavior is undesirable.
The most extreme statement on the negative side further emphasize that there is
no need for clear instructions to develop the athlete‟s performance (statement
number 10). The lack of faith in instruction as a pedagogical measure appear to
be a main characteristic in factor B, which contradicts that a basic element in
coaching is guided improvement the long term development of athletes.
However, textbooks in motor learning and physical education have argued that
instructions can be ineffective and the effect of instruction may depend on the
skill and the performance level of the athlete (Schmidt & Wrisberg, 2008;
Siedentop & Tannhill, 2000). An analysis of the general variables to the
respondents loading on factor B shows that they are from athletics. In some of the
athletics exercises like distance running, technical skill instruction may be
consider excessive especially if the athlete has developed and automatized a
rather efficient running technic. Coaching these athletes is more about creating a
training program with recommended training volume (training frequency,
duration and intensity).
The two other psychological statements representing factor B negatively
emphasizes that either a close relationship with a coach or involvement are
needed to affect emotions such as curiosity and focus (statement number 28 and
31). However, coaches disagree with these statements indicating that the
respondents view a close relationship and involvement from the coach as suitable.
Jowett (2005) claims that coach athlete relationship can be described in a)
prizewinning and b) helpful/caring relationship dimensions. The two
dimensions are interrelated and the prizewinning category has effective and
ineffective as sub-dimension, while the helpful/caring relationship has
successful and unsuccessful subcategories. The belief that instruction is not
efficient, which represent one belief loading one factor B may be categorized in
the ineffective prizewinning dimension in Jowett‟s (2005) taxonomy. The belief
that closeness and athlete involvement are desirable can be considered as a basic
prerequisite developing a successful relationship, which is a category in the
helpful/caring dimension in the taxonomy.
According to Stone, Russell, and Patterson (2003, 2004) servant leaders gain
influence in a non-traditional manner that derives from servant hood itself. They

190
allow extraordinary freedom for followers to exercise their own abilities and
place much higher degree of trust than would be the case in any leadership style
required by the leader to be directive. The coaches loading on factor B seem
therefore to have elements of a servant leadership.
Table 5
Distinguished Statements Loading on Factor B
Number Statement Strength

24 I am losing my curiosity when my coach gives me clear instructions. +5
27 I become stressful if my coach involves me in important matters +4
regarding my training.
33 If I‟m told exactly what to do I lose my motivation. +4
31 I’m calm and steady regardless of a close relationship with my coach or not. -4
28 I’m curious regardless of involvement or not from my coach. -4
10 Clear instructions regarding what I am supposed to do develop my -5
performances.
Note. Included mixed cases, 20 cases loaded on factor B.
4. Discussion
The 23 coaches who participated in this investigation were instructed to sort 36
statements about different views on coaching behaviours, and rank the
statements on a scoreboard ranging from +5 to -5 regarding what they believe are
expected behavior in their roles as coaches in sport. Based on their experience
they were asked to reflect on the content of the statements and prioritized them
in accordance with their own personal view. The results show that 21 out of the
23 coaches, (when mixed sorts are included) loaded significant positive on Factor
A: Involvement leadership (Table 3). This factor counts for 38% of the variance.
Seven coaches (when mixed sorts are included) loaded on factor B: Servant
leadership (Table 3). However, only one coach loaded significant positive on
factor B and the rest of the loadings were negative.
After analysing the two different factors it is clear that the factors represent
individual viewpoints that clearly separate them from each other. The negative
correlation between the factors confirms this as well as the scores on each
statement representing the two factors (Appendix). In the discussions below
these two factors will therefore be treated based on their typical individual
viewpoints.

191
4.1 Involvement leadership

The theory of transformational leadership was developed by Bass (1985) and he
proposes that transformational leaders display certain characteristics such as
showing concern and care for each subordinate, and stimulating them
intellectually to think about old problems in an innovative way. In sport,
Charbonneau, Barling, and Kelloway (2001) showed that intrinsic motivation
mediate the relationship between transformational leadership and sport
performance, suggesting that transformational leadership may enhance intrinsic
interest in the task. The following strong statement loading on factor A ”I become
curious and interested if my coach involves me in matters concerning my
training” (4) and “Neither feedback nor social support are crucial for my
performances in sport” (-4) seems to be beliefs that correspond with elements of
transformational leadership. A belief in athlete involvement in the training
process seems to be a prerequisite to stimulate athletes intellectually, and provide
a foundation for innovations. A fundamental belief in involvement is also a
precondition for stimulating athlete autonomy and competence, which are
cornerstones in self-determination theory (Ryan & Deci, 2002). A basic
assumption in self-determination theory is that intrinsic motivation is affected by
the extent to which the fundamental human needs for competence, autonomy,
and relatedness are fulfilled or satisfied. According to Amorose and
Anderson-Butcher (2007) there are motivational benefits of autonomy-supportive
coaching behavior (Amorose & Horn, 2000). The respondents loading on factor A
strongly disagree that feedback and social support are crucial for performance in
sport indicating that these coaching behavior are important for performance
enhancement. This finding also seems to be in line with transformational
leadership, which emphasizes caring and concern for subordinates or athletes.
Interestingly, 21 out of the 23 coaches in this study loaded on factor A when
mixed sorts are included. There are five mixed sorts across the two factors and
they are all loading negatively on factor B. Thus, the views representing factor A
are strong among the coaches in this study and factor B seems to represent a
contrasting view compared with factor A; since their statements are sorted
completely different (see Appendix and the different scores for each statement on
factor A and B).
Viewpoints representing factor A in this study do not solely confirm earlier
research on coaching behaviours (Chelladurai, et al., 1988; Horne & Carron, 1985;
Moen & Sandstad, 2013; Riemer & Chelladurai, 1995). This research shows that
coaching behaviour associated with training and instruction, positive feedback,
and social support are highly correlated with athletes‟ satisfaction and their
intrinsic motivation. Viewpoints that are associated with positive feedback and
social support are represented in factor A, but viewpoints associated with
training and instructions are not. This is rather surprising, while instructive
behaviour from a coach, especially with junior athletes, might be necessary to

192
meet the quality that is needed to enhance the athlete‟s performance level
(Ericsson, 2009). Importantly, exercise that is needed to improve an athlete‟s
performance levels is not found to be a playful enjoyment (Ericsson, et al., 1993).
Developing an athlete‟s level of performance is an effortful endeavour that takes
engagement, curiosity and inspiration. Interestingly, the coaches that are loading
on factor A in this study believe that coaching behaviour promoting involvement
has an affect on these emotions.
The belief in the involvement of athletes during the training process can be
considered as a basic value and a prerequisite to empower the athlete to become
more independent and take ownership of the learning process. According to
Jowett (2007) is an interdependent relationship between coach and athlete
described in terms of closeness, commitment and complementarity. A belief in
athlete involvement in the training process, as well as an opinion of the necessity
of social support and feedback in the learning process seems to be an important
precondition to promote a positive and healthy coach-athlete relationship.
However, athlete involvement is also a necessary precondition establishing a
more democratic leadership.
One can ask if the emphasis on athlete involvement and social feedback among
these coaches on the one side, together with the absence of determined behavior
such as criticism and instruction on the other, are too friendly of nature for
coaches who are working to improve a junior athlete‟s levels of performance. A
recent study found that junior athletes expect a paradoxical mixture of humility
(involvement, positive feedback, a personal relationship and social support) and
determinate behavior (criticism and instructions) from their coaches (Moen &
Sandstad, 2013).
4.2 Servant leadership - caring and non-task controlling

The most psychologically significant statements representing factor B seem to
represent the view that an athlete does not need instructions from a coach to
affect motivation, performance and emotions. Two of the most psychological
negative statements representing factor B (statements number 28 and 31) are
understood as concerns among the coaches loading on this factor. The consensus
representing factor B seems that an athlete is expected to act independently of the
coach and that the coach does not expect to take too much responsibility
regarding the athlete. Thus, the way coaches loaded factor B suggests that they
believe their role is expected to be in the background and not being involved too
much in the work with the athlete and the athlete is expected to take
responsibility him- or herself. Werthner & Trudel (2006) suggest that elite
coaches learning can be understood in terms of mediated, unmediated and
internal processes. Where in a mediated learning situation a coach directs the
athlete, opposed to no coaches present in unmediated situations. Thus, the

193
athlete must take the initiative and the responsibility for choosing what to learn
in unmediated learning situations. These concepts may also be applicable
concerning athlete learning and coaches loading on factor A seems to believe
strongly in learning these young athletes is a mediating process, while coaches
loading on factor B believe this is an unmediated process.
4.3 Conclusion, implications and future research

The aims of this study were to explore coaches‟ opinion about how and
eventually which coaching behavior affects the athletes. Furthermore, discover
patterns among the participants and identify coaches who express similar
subjective views, and clarify if Q methodology is suitable for studying coaching
beliefs. The present findings show that the coaches share some common
viewpoints represented in two different factors (consensus). Each factor
represents congruent views related to the role of coaching expectations in sport.
The dominant view (factor A) is that coaches believe that their athletes expect
involvement leadership, whereas servant leadership was dominant in factor B, a
view that only a few of the coaches shared.
The coaching role in sport is relatively free with great opportunities for the
individual to create and design it based on personal convictions (Heinemann,
1983). Thorsen (2009) argues that Q methodology stimulates participants‟
awareness of their own position related to the research question. Based on
findings from studies in cognitive therapy, which shows that attitudes, beliefs
and expectations can shape peoples reality and behavior, we assume that the
finding in this study is important for coaching in sport (David, Lynn & Ellis,
2010). We often assume that coaches‟ values are observable in their behavior or
whish that it should be. However empirical examination shows that the
connection is not as straightforward as the coaching literature would have us to
believe (Lyle, 1999). This is because little account is taken of the contextual
pressure, which also influences coaching behavior (Cassidy, Jones, & Potrac,
2009). These beliefs seem to be easy to say and difficult to keep, and the link
between coaches‟ beliefs‟ and their actions has rarely been empirically
investigated (Lyle, 1999). Further research should illuminate the development of
these beliefs and clarify how coach education influences each factor. The findings
in the present study show that a Q methodology approach can be a helpful in the
exploration of coaching beliefs
4.4 Limitations
Q studies are not designed to generalize results to larger populations or to
determine causal relationship between variables or estimate prevalence
(Øverland, Thorsen, & Størksen, 2012). Generally, Q methodology explores
subjective views and this was the intent in studying coaches‟ perceptions of

194
significant beliefs. Qualitative research has occasionally been criticized for

modest quality (Mays & Pope, 2000), but clear guidelines for conducting a Q
study should accommodate this critic. Brown, (1980) argue that Q methodology
is a less biased than questionnaires, where the content can be predetermined, and
interviews where it is up to the researcher to categorize the results.
Acknowledgments
The authors would like to thank the coaches for their cooperation, enthusiasm,
and most of all, their participation in this study.
References
Abraham, A., Collins, D., & Martindale, R. (2006). The coaching schematic: Validation
through expert coach consensus. Journal of Sport Sciences 24, 549-564.
Allgood, E., & Svennungsen, H. O. (2008). Toward an articulation of trauma using
creative arts and Q methodology. Journal of Human Subjectivity, 6, 5-24.
Amorose, A. J., & Anderson-Butcher, D. (2007). Autonomy-supportive coaching and
self-determined motivation in high school and college athletes: a test of self-
determination theory. Psychology of Sport and Exercise, 8, 654-670.
Amorose, A. J., & Horn, T. S. (2000). Intrinsic motivation: Relationships with collegiate
athletes gender, scholarship status and perceptions of their coaches‟ behavior.
Journal of Sport & Exercise Psychology, 22, 63-84.
Bass, B. M. (1985). Leadership and performance beyond expectations. New York: Free Press.
Blom, L. C., Watson II, J. C., & Spadaro, N. (2010). The impact of a coaching intervention
on the coach-athlete dyad and athlete sport experience. Athletic Insight- The Online
Journal of Sport Psychology, 12. Retrieved from http://www.athleticinsight.com/
Vol12Iss3/Feature.htm.
Brown, S. R. (1980). Political subjectivity, applications of Q methodology in political science.
Yale University.
Brown, S. R. (1996). Q methodology and qualitative research. Qualitative Health Research, 6,
561-567.
Brown, S. R. (2002). Subjective behaviour analysis. Operant Subjectivity 25, 145-163.
Cassidy, T., Jones, R., & Potrac, P. (2009). Understanding sports coaching: The social, cultural
and pedagogical foundations of coaching practice (2nd ed.). London: Routledge.
Charbonneau, D., Barling, J., & Kelloway, E. K. (2001). Transformational leadership and
sports performance: The mediating role of intrinsic motivation. Journal of Applied
Social Psychology, 1, 1521- 1534.
Chelladurai, P. (1989). Manual for the leadership scale for sports. Unpublished manuscript.
The University of Western Ontorio.
Chelladurai, P. (1990). Leadership in sports: A review. International Journal of Sport
Psychology, 21, 328-354.
Chelladurai, P. (2007). Leadership in sports. In G. Tenenbaum., & R. C. Eklund (Eds.), The

195
handbook of sport psychology (3rd edition, pp. 113-135). New York: Wiley.
Chelladurai, P., Imamura, H., Yamaguchi, Y., Oimnuma, Y., & Miyauchi, T. (1988). Sport
leadership in a cross-national setting: the case of Japanese and Canadian university
athletes. Journal of Sport & Exercise Psychology, 10, 374-389.
Côté, J., & Gilbert, W. (2009). An integrative definition of coaching effectiveness and
expertise. International Journal of Sport Science and Coaching, 4, 307-232.
David, D., Lynn, S., & Ellis. A. (2010). Rational and irrational beliefs. Implications for research,
theory, and practice. NY: Oxford University Press.
Deci, E. L., & Ryan, R. M. (2002). Handbook of self-determination research. Rochester, New
York: University of Rochester Press.
Ericsson, K. A. (2009). Development of professional expertise: Toward measurement of expert
performance and design of optimal learning environments. Cambridge, UK: Cambridge
University Press.
Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role of deliberate practice in
the acquisition of expert performance. Psychological Review, 100, 363–406.
Haney, J. J., Lumpe, A. T., Czernaik, C. M., & Egan, V. (2002). From beliefs to actions: The
beliefs and actions of teachers implementing change. Journal of Science Teacher
Education, 13, 171-187.
Heinemann, K. (1980). Introduction to the sociology of sport. Schorndorf: Hofmann Verlag.
Horn, T. S. (2008). Coaching effectiveness in the sports domain. In T. S. Horn (Ed.)
Advances in sport psychology (pp. 309-354). Champaign, IL: Human Kinetics.
Horne, T., & Carron A. V. (1985) Compatibility to coach-athlete relationships. Journal of
Sport Psychology 7, 137-149.
Jones, R. L. (2006). The sports coach as educator: re-conceptualising sports coaching. Routledge:
Taylor & Francis Group, London.
Jowett, S. (2005). The coach-athlete partnership. The Psychologist, 18, 412-415.
Jowett, S. (2007). Interdependence analysis and 3+1 Cs in the coach-athlete relationship.
In S. Jowett, & D. Lavallee (Eds.), Social psychology in sport (pp.15-28). Champaign,
IL: Human Kinetics.
Jowett, S., & Cockerill, I. M. (2002) Incompatibility in the coach–athlete relationship. In I.
M. Cockerill (Ed.), Solutions in sport psychology. (pp. 16–31) London: Thompson
Learning.
Jowett, S., & Cockerill, I. M. (2003). Olympic medalists‟ perspective of the athlete-coach
relationship. Psychology of Sport and Exercise, 4, 313-331.
Jowett, S., & Meek, G. A (2000). The coach–athlete relationship in married couples: An
exploratory content analysis. The Sport Psychologist, 14, 157–175.
Jowett, S., & Poczwardowski, A. (2007). Understanding the Coach-Athlete Relationship.
In S. Jowett, & D. Lavallee (Eds.), Social Psychology in Sport (pp.3-14). Champaing IL,
Human Kinetics.
Kozulin, A. (2003). Psychological tools and mediated learning. In A. Kozulin, B. Gindis, V.
Ageyev & S. Miller (Eds.), Vygotsky's educational theory in cultural context (pp. 15- 38).
Cambridge: Cambridge University Press.

196
Kvalsund, R. (1998). A theory of the person. Department of Education Faculty of Social

Science and Technology Management. The Norwegian University Of Science And
Technology Trondheim.
Lim, C. (2009). Understanding Singaporean preschool teachers‟ beliefs about literacy
development: four different perspectives. Teaching and Teacher Education, 26,
215-224.
Lyle, J. (1999). Coaching philosophy and coaching behaviour. In J. L. Cross, & J. Lyle
(Eds.), The coaching process: Principles and practice for sport (pp. 25–46). Oxford:
Butterworth-Heineman.
Lyle, J. (2002). Sports coaching concepts. London: Routledge.
Mageau, G. A., & Vallerand, R. J. (2003). The coach-athlete relationship: A motivational
model. Journal of Sports Sciences, 21, 883-904.
Mays, N., & Pope, C. (2000). Qualtative research in health care: Assessing quality in
qualitative research. BMJ, 320, 50-52.
McKeown, B., & Thomas, D. (1988). Q Methodology. London: Sage.
Moen, F. (2012). Subjective beliefs among coaches about how relational factors affect
intrinsic motivation, responsibility and development in Sport. Athletic Insight
Journal, 3, 219- 235.
Moen, F., & K, Garland (2012). Subjective beliefs among Sport Coaches about
Communication during practice. Scandinavian Sport Studies Forum (SSSF), 3,
121-142.
Moen, F., & R. Kvalsund (2013). Subjective beliefs among sport coaches about
communication during coach-athlete conversations. Athletic Insight Journal, 5,
229-250.
Moen, F., & R. Kvalsund (2014). Subjective beliefs among sport coaches about
communication during practice and coach-athlete conversations . Operant
Subjectivity, 37, 1–19
Moen, F., & Verburg, E. (2012). Subjective beliefs among athletes about how relational
factors affect intrinsic motivation, responsibility and development in sport. The
International Journal of Coaching and Science, 1,81-100.
Moen, F., & Sandstad, H. (2013). Subjective expectations among athletes about coaches in
sport. Journal of Human Subjectivity, 11, 135-155.
Moen, F., Høygaard, R., & Peters, D. M. (2014). Athletes’ performance progress in relation to
perceived coach-leadership behavior. The International Journal of Coaching Science, 8,
67-79.
Myers, N. D., Chase, M. A., Beauchamp, M. R., & Jackson, B. (2010). The coaching
competency scale II – High School Teams. Educational and Psychological Measurement,
70, 477-494.
Nespor, J. (1987). The role of beliefs in the practice of teaching. Journal of Curriculum
Studies, 19, 317-28.
Olympiou, A., Jowett, S., & Duda, J. L. (2008). The psychological interface between the
coach-created motivational climate and the coach-athlete relationship in team

197
sports. The Sport Psychologist, 22, 423-438.

Pett, M. A., Lackey, N. R. & Sullivan, J. J. (2003). Making Sense of Factor Analysis: The Use of
Factor Analysis for Instrument Development in Health Care Research. Thousand Oaks,
CA: Sage.
Rhoads, J. C. (2007). Q methodology. In N. J. Salkind & K. Rasmussen (Eds.), Encyclopedia
of measurement and statistics (pp. 799-802). Thousand Oaks, CA, London, & New
Delhi: Sage.
Riemer, H. A., & Chelladurai, P. (1995). Leadership and satisfaction in athletics. Journal of
Sport and Exercise Psychology, 17, 276-293.
Rokeach, M. (1973). The nature of human values. New York: The Free Press.
Rokeach, M. (1976). Beliefs, attitudes, and values: A theory of organization and change. San
Francisco: Jossey-Bass Publishers.
Schmidt, R. A. & Wrisberg, C. A. (2008). Motor Learning and Performance, (4th ed.).
Champaign, IL: Human Kinetics.
Schmolck, P. (2002). PQMethod download mirror retrieved from
http://www.qmethod.org/Tutorials/pqmethod.htm
Siedentop, D., & Tannehill, D. (2000). Developing teaching skills in physical education (4th ed.).
Palo Alto, CA: Mayfield.
Stephenson, W. (1950). A statistical approach to typology: The study of trait universes.
Journal of Clinical Psychology. 6, 26-38.
Stephenson, W. (1986). Protoconcurcus: The concourse theory of communication. Operant
Subjectivity, 9, 37-58, 73-96.
Stone, A. G., Russell, F. R., & Patterson, K. (2003). Transformational versus servant
leadership: A difference in leadership focus. Proceedings of the Servant Leadership
Research Roundtable. Retrieved July 15, 2004, from
http://www.regent.edu/acad/cls/2003ServantLeadershipRoundtable/
Stone, A. G., Russell, R. F., & Patterson, K. (2004). Transformational versus servant
leadership: A difference in leader focus. Leadership & Organization Development
Journal, 25, 349-361.
Strachan, L., Côté, J., & Deakin, J. (2011). A new view: Exploring positive youth
development in elite sport contexts. Qualitative Research in Sport, Exercise, and Health,
3, 9-32.
Thorsen, A. A. (2009). Teachers' priorities and beliefs: A venture into beliefs,
methodologies, and insights. Department of Education, University of Stavanger,
UiS, Doctoral dissertation.
Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes.
Cambridge, MA: Harvard University Press.
Watts, S., & Stenner, P. (2012). Doing Q methodological research. CA: Sage.
Werthner, P. & Trudel, P. (2006). A new theoretical perspective for understanding how
coaches learn to coach. The Sport Psychologists, 20, 196-210.
Wilcox, S. & Trudel, P. (1998). Constructing the coaching principles and beliefs of youth
ice hockey coach. AVANTE, 4, 39-66.

198
Øverland, K., Thorsen, A. A., & Størksen, I. (2012). The beliefs of daycare staff and
teachers regarding children of divorce: A Q methodological study. Teaching and
Teacher education, 28, 312-323.
Appendix A
Factors
Statements A B
1. My motivation for training increases when my coach involves me. 5 -3
2. If I‟m involved in the process concerning my training I perform better. 4 -3
3. My coach does not have to be open for questions. -1 2
4. I become curious and interested if my coach involves me in matters 4 -2
5. My motivation increases when my coach is concerned about my 3 -2
personal well-being.
6. If I‟m supposed to achieve good performances my coach needs to focus -2 3
on my personal welfare.
7. My situation becomes less stressful when my coach contributes in 1 0
personal affairs.
8. A personal and close relationship with my coach makes me enthusiastic 2 0
9. My motivation increases when I‟m told exactly what to do. -1 -1
10. Clear instructions regarding what I am supposed to do develop my 3 -5
performances
11. I keep my focus if the coach intervenes in training and explain what is 0 -3
right and wrong.
12. I become curious if my coach gives me clear instructions about what I 1 0
need to do.
13. My motivation increases when my coach takes decisions that concern -3 0
me.
14. My performances are not good when my coach denies complying with 1 -1
my opinions.
15. My coach needs to consult me if I‟m supposed to have an effective focus. 2 2
16. I become uncommitted if my coach includes me in decisions regarding -5 3
my sport.
17. My motivation increases when I receive positive feedback. 3 -2
18. Neither feedback nor social support is crucial for my performances in -4 1
sport.
19. A close and personal relationship with my coach makes me stressful. -3 2
20. My curiosity is best stimulated when the relationship with my coach is -1 1
not too close and personal.
21. I lose my engagement when I‟m observed by my coach and receive no -1 -2
feedback.

199
22. I perform at my best when I have to clarify my own task for training. 0 1
23. I become insecure if a coach does not tell me exactly what to do. 2 -1
24. I am losing my curiosity when my coach gives me clear instructions. -3 5
25. My motivation decreases when my coach needs my approval in 0 3
important matters concerning my training.
26. I‟m not able to perform if my coach often asks me for approvals in -2 0
important matters.
27. I become stressful if my coach involves me in important matters -4 4
regarding my training.
28. I‟m curious regardless of involvement or not from my coach. 0 -4
29. My motivation increases when my coach does not have focus on 0 1
personal issues.
30. In order to develop my performances I also need critical feedback from 2 -1
my coach.
31. I‟m calm and steady regardless of a close relationship with my coach or 1 -4
not.
32. Whether my coach is concerned about personal issues or not do not -2 0
affect my curiosity.
33. If I‟m told exactly what to do I lose my motivation. -2 4
34. I perform at my best when my coach just observes what I do during -1 1
training.
35. I lose my focus when it is too much instructions. 1 -1
36. It is easier to be curious when the coach is more in the background. 0 2
* Translated from Norwegian to English by the authors.

200

Vol. 11, No. 1, pp. 200-211, April 2015
Regional Educational Development Research

and School Improvement: A Systematic
Literature Review of Research
Associate Professor Lena Boström

Departement of Education, Mid Sweden University
Abstract. With the background of regional educational problems in

some counties in Sweden, in the form of lower educational level and
school achievements an integrative and systematic literature review
(SLR) has been implemented in order to examine regional educational
development in combination with school development worldwide
during the last 50 years. The literature review shows an internationally
sprawling view of the content and object of the study as well as a few
international studies on the relationship between regional development
and school improvement in mainstream school. Different regions,
countries, and continents show various problems within this focus. The
study shows different aspects; networks and knowledge-based clusters
as guidelines; political decentralization and its repercussions on student
learning; technical solutions; the importance of vocational schools in
rural areas; tax reduction to compensate rural areas; the importance of a
majority/national language and rural versus urban values. Regional
development policies aimed at stimulating regional industrial
competitiveness and innovativeness, educational attainment, and
creative occupations emphasize that creative professionals are strongly
related to regional income. The most important implication common
worldwide, are concerns pertaining to urbanization, depopulation of
rural areas, the need to highlight regional development factors, and a
belief in education, especially involving mainstream schools.
Keywords: literature review, mainstream schools, regional educational

development, school improvement.
Introduction
Two counties in Mid-Sweden, Jämtland and Västernorrland, have identified
regional educational problems. These counties have lower educational levels
and school achievements compared to other parts of Sweden (Skolverket, 2014)
and have a relatively large proportion of an out-migration of well-educated
individuals from the region (Statistiska centralbyrån, 2014). The national school
results have, in some parts of the region, steadily declined for nine years; other
regional areas also need to be improved (Skolverket, 2014). An integrative and

201
systematic literature review (SLR) concerning Swedish (and Nordic) research

has been carried out in order to get an overall picture of the best possible
knowledge in this area. This review showed that there has been little debate, and
the sources were mostly descriptive studies. In addition, one-third of the sources
were official reports. The discussions, debates, and investments pertaining to
universities and education programs‘ role in the Swedish rural areas began in
the mid-1980s. Even then, the role of universities for rural and Sweden‘s
prosperity was discussed in light of increased global competition. There are not
any research studies or debates to discuss the grade levels through the whole
school system. Studies about school improvement and the impacts on students
and regional development are also missing. Young people‘s and women‘s voices
are absent (Boström, 2015). The concern for the lack of research on ‗regional
youth‘ is raised by (Svensson, 2006), with the words, “When rural areas are
studied, the focus is outside the adolescent themes and the adolescents studied, the focus
is outside the rural perspective‖ (p. 24). A key question for a practitioner teaching
in school or higher education today could be as follows: How can we make sense
of current regional educational research and debates in order to implement the
results into teaching and courses? (Svensson, 2006). In other words, there is a
research gap regarding the role of school (mainstream) and regional
development.
The purpose of this overview is therefore to gather international knowledge

about the object of knowledge. The aim of the study was to deepen the
knowledge of previous research on regional educational development and
school improvement worldwide as well as to capture and describe the
phenomenon of variation in different contexts.
Previous Research
The past decade may have witnessed considerable expansion and development
within the field of regional educational research (From & Olofsson, 2014). Good
education systems, ranging from preschools to universities, are vital for
development. This mutual relationship is described and emphasizes that
initiatives in the classroom or department are influenced by the surrounding
context of the school, the district, and the nation (Hernandez & Goodson, 2010;
Veugelers & Ziljsra, 2010). It is important not only from a regional perspective,
but also in a national and international context.
The importance of higher education for a region‘s growth has increasingly

attracted international attention in many countries, including Russia, Britain,
Canada, Turkey, China, and Australia, and on all continents. If countries want to
be globally competitive, regional innovation systems need to be strengthened. In
order to achieve this, cooperation between higher educational institutions,
public authorities, and the business sector becomes vital. Regional engagement,
academic excellence, and research are complementary activities. In order to
allow innovative efforts in peripheral regions to be as productive as they are in
core areas, they need to be complemented by huge investments and human
capital (Crescenzi, 2005). The role of human capital and the creative class, as well
as technology factors, sharpen regional development. For regions to compete,
evolve, and survive, education is crucial, and there are clear links between

202
education and regional development (From & Olofsson, 2014). A further aspect
is the concept of regional management, which seems to play an important role in
developing regional headquarters into dynamic competence centers (Ambos &
Schlegermilch, 2010).
Even in Sweden, the regional educational development in cooperation with

academic attention is problematized and researched (e.g., Persson et al., 2003).
The majority of the research in Sweden during the last 40 years has focused on
higher education, regional projects, and collaboration between industries and
universities. Only a few studies have touched on the younger generation‘s
situation (Boström, 2015). An attempt to connect to regional development in the
school context is the research project ―The best regional education systems in the
world,‖ which is the partnership of one university, huge companies, and three
selected schools in the region of Mid-Sweden. The intent behind the project is to
make it possible for a region like Mid-Sweden to develop in a positive way for
citizens companies, and the public sector, and the underlying assumption is that
the regional educational system needs to contribute to regional changes
regarding economic growth as well as the public sector (Mittuniversitetet, 2012).
An important background factor is that these countries have lower educational
levels and school achievements compared with other parts of Sweden, and they
have a relatively large proportion of out-migration of well-educated individuals
from the region. Therefore, the aim of this study is to identify regional
educational-development research and regional school improvements across a
variety of contexts through a systematic review of associated literature
published between 1983 and 2014. Such a review contextualizes and is an
international background for the research project. The concepts in the study are
based on research into both regional educational development and regional
school development
By virtue of the foregoing, the purpose of this study was, from an international
perspective, to identify and classify the published research on regional
educational development and school improvements. The objectives were to (a)
identify patterns and trends in the research, (b) describe and compare the
published findings, and (c) point to a future research agenda.
Theoretical Framework
The study departs from regional educational development and school
improvement theory. Regional educational development is a useful analytical
tool as it can mirror the relation between regional development and education.
There are correlations between level of education and regional development. To
succeed in regional development, a highly skilled workforce is required where
education is a cornerstone (e.g. Florida, Mellander & Stolarick, 2010; Tomaney &
Wray, 2011). Universities were perceived of utmost importance with regard to
regional development (Westlund, 2004), but regional development is also a
social change and transformation (Berglund & Johansson, 2007). To go from
stability and recognition to the new and unfamiliar is described in the following
words: ―It is the combination of the new and the traditional providing
innovative opportunities for regional development and economic growth, which
requires interaction and communication‖ (From & Olofsson, 2013, p. 35).

203
The core of the definition of school improvement is described as “… a systematic,

on-going, and supported efforts to make principals and teachers in schools more capable
of achieving the national and local objectives in the school” (Björkman, 2008, p. 33),
with references to P. Dalin, 1994, and W. van Velzen et al., 1985, and implies a
particular form of changes in a certain direction to make something better.
Goodland, 1994, believes that school improvement needs a focus as a whole in
order to be successful. Culture and structure both have to be consciously
processed in order to achieve effective and sustainable change (Hargreaves &
Fink, 2008). School improvement theory must, according Höög and Johansson,
2014, bear cornerstones and different focuses concerning structure, culture,
processes, and leadership. These elements collaborate, interact, and have
repercussions on each other in order to change and adapt the structure and long
term consistent processes which in turn affect the school‘s leadership by
providing dynamics and conflict.
In summary, the theoretical frameworks are set around school improvement

comprising structure in the context of regional development, including culture,
processes, and leadership, which requires communication and interaction with
both new and traditional activities.
Aim and Methods

The aim of this study was to deepen the knowledge of previous research on
regional educational development and school improvements worldwide. This
study is based on an integrative and systematic literature review (SLR), inspired
by Polit and Beck‘s (2008) model for a qualitative, thematic, content analysis.
Integrative literature reviews offer a means for researchers to find and assess
what is known about a particular topic, with the aim of finding a solution to a
particular problem or suggesting directions for future research. It is described as
being the broadest category of research reviews, combining the findings of a
range of different research designs including those of qualitative and
quantitative studies (Evans & Waring, 2012).
To analyze the contents of the various texts, a descriptive content analysis was
performed. The contents of the results of the studies were examined
methodically and progressively while interpreting texts to find prevalent
phenomena. Distinctive categories were identified and then narrowed down to
sub-categories. Traditional content analyses can be divided into three steps:
selection of focus texts, encoding of the texts, and interpretation of the results
(Auhiva, 2008). The situational context was taken into account, so the
―maximum variation sampling‖ was achieved (Franzosi, 2008). Maintaining
scientific integrity involves great attention to validity during the phase of the
integrative review, and not defining the operational definitions too narrowly or
too broadly. The reviewer must balance the definitions and methods review
constantly during the research process. The integrative literature review has
many benefits to the scholarly reviewer, such as identifying gaps in current
research and the need for future research, bridging related areas of work, and
bringing focus to central issues in an area (Cooper, 1998). A thematic integrative
study design was chosen to obtain a holistic understanding of the subject. This

204
approach, in accordance with the method, allows researchers to combine data

from theoretical literature as well as empirical literature (Whittemore & Knafl,
2005 ), which also makes the analysis more complex and difficult. The purpose
could therefore be answered by a wider range of sources.
Strategy and Inclusion Criteria

Employing progressively detailed inclusion criteria, the systematic review of the
literature involved multiple phases: an initial identification of potential studies
via an advanced search of PRIMO; subsequent screening of abstracts and titles;
hand searching of seven books, in-reference lists, and bibliographies of all
retrieved papers, 15 book chapters, and five reports, ten books, five
dissertations, ten debate articles on the theme; and an in-depth review of 190 full
articles along with a detailed thematic analysis of 33 full articles, five chapters,
one dissertation, two reports and 7 books. An automated advanced search of this
database was conducted using increasingly refined search criteria to identify
those potentially relevant studies for use in the review (see Figure 1).
Figure 1. Summary of Method of Review
Keywords used were: regional educational development*, school improvement*,

school efficiency*, regional school development*, and school effectiveness*, with
different combinations. The literature search was limited to 1980–2014 and to
publications written in English. To contextualize the findings of the review, an
overview of the current debates and conclusions within regional educational
development and school improvement internationally were provided, including
a consideration of the relationships between regions, education, school
improvements, and success factors. The authors identified applicable articles
and verified their inclusion in the search results in order to ensure the sensitivity
of the search strategy. References from included studies were also assessed.

205
Structured around two levels of analysis, a summary and discussion of the

outcomes of the review are presented. This involved a general overview
concerning the distribution; research foci; and associated theoretical framework,
methods, and design of the articles reviewed; followed by a thematic analysis of
regional educational development research in relation to school improvement
from an international perspective. The sources were reviewed and evaluated
concerning content, scientific quality and relevance. Data processing, quality,
sorting, categorization, and examination of the items were made using
assessment documentation. Finally, the content was thematized.
Results and Analysis

This section outlines the outcomes of the review. It is structured around two
levels of analysis: distribution of themes, years, methodological approaches,
assessed relevance, and the sources and languages of the articles reviewed, and
was followed by a thematic analysis and discussion of the data in relation to the
nature of an impact on regional educational development and school
improvement. Only 48 articles remained for the analysis. The reason for this may
be that regional developments in mainstream school do not have a specific focus
worldwide.
Distribution
Figure 1, above, shows the 48 sources that were included in the review and
Table 1, below, specifies the themes. There were six themes: Political aspects,
Social Capital & Networks, Higher education, General Education, Values,
Languages & Cultures, and one named ―Others.‖ The sources were categorized
as one entity.
Table 1. Distribution of themes among the sources.
Themes N = 48 Percent
Politics 14 29,2%
Networks & Social 8 16,7%
Capital
Higher Education 7 14.6%
General Education 6 12.5%
Values 5 10,4%
Languages & 4 8.3%
Cultural
Others 4 8.3%
Two dominant themes are political aspects and social capital and networks
(almost half of the content). The theme ―Others‖ are technical solutions,
women‘s roles and dichotomization, major universities, and small colleges.
The majority of the sources were published from 2001 and forward, which were
a majority of the articles (see Table 2). In other words, interest in the subject has
grown rapidly in the last fifteen years.

206
Table 2. Distribution of 48 sources reviewed by year.

Years Sources Percent
N= 48
1980–1985 1 2.1%
1986–1990 0
1991–1995 2 4.2%
1996–2000 4 8.3%
2001–2005 10 20,8%
2006–2010 20 41,7%
2011– 11 22,9%
The geographic distribution shows that almost half of all the sources are
published in European countries, and approximately one-fourth come from the
US/Canada (see Table 3). This means that the research has been conducted in
Western countries.
Table 3. Continents among the sources.
Continent Sources Percent

N= 48
Europe 25 52,1%
North 11 22,8%
America
Asia 5 10,4%
Africa 3 6.3%
South America 2 4,2%
Australia 1 2,1%
International 1 2,1%
Thus, this area of research seems to be a typical western phenomenon.

Concerning methodology approach, there is an almost total domination of
qualitative approaches, 54.2% (Table 4) followed by quantitative studies (27.1%).
The studies which do not clearly have a methodological approach usually have a
descriptive, historical approach.
Table 4: Distribution of methodological approach.

Methodological Sources Percent
approach N= 48
Qualitative 26 54.2 %
Approaches
Mixed Method 4 8.3
Design
Quantitative 13 27.1 %
Approaches
Not 5 14.4
Reported/Unclear

207
The qualitative approaches are highly varied from case studies to focus group
interviews and observations. A small percentage uses quantitative approaches or
mixed methods. When it comes to data collecting strategies, most of the sources
have one strategy, but some have 2 to 4 strategies. The strategies are many and
various, from video-taping to self-assessment reports to content analyses. We
have not been able to classify them, as they are too sprawling. There are also
problems with being able to classify analysis levels.
Thematic Analysis
In this SLR, it is possible to observe a lack of empirical research involving the
combination of regional educational development and school improvement,
especially for mainstream schools (cf. Boström 2015). Research is published, but
it focuses mostly on one aspect (school improvement) or the other (regional
educational development) or each, but not so much the combination. Political
aspects seem to be the most dominant theme with it described in almost one-
fourth of the contents. The content of policy moves at different levels, political
control, labor market reforms, decentralization, tax reduction to compensate
rural areas, and ideological debates about the central cities versus regions.
However, there are no concrete descriptions or visions of how the different
proposals could have repercussions on school improvement. The quote below
will illustrate the general approach:
Education today is characterized by two opposite tendencies: —a centripetal
tendency … a centrifugal tendency … these tendencies map out the main vectors
of the development of innovative activity in regional systems of education.
(Larina, 2006, p. 31)
Another focus is Social Capital & Networks, preferably in combination with the
development of a ―creative class‖ and a diversity that can develop business
ideas and promote entrepreneurial learning. The importance of interaction
between people trained in creative professions so that they can create new
products/services and generate jobs, new technology, and tolerance for the
regions as described by Florida et al. (2010): ―Canadian regional development is
shaped by the 3Ts of technology, talent, and tolerance. Talent in the form of
human capital and the creative class is strongly associated with regional
income‖ (p. 31).
The power of building networks within the regions is highlighted as an

opportunity to recruit students of different origins and to raise and support
multi-cultural awareness and social support. Thus, capacity, knowledge, new
ideas, and norms will be created (Cheong, Wing, & Alan, 2009). Through
building networks within the region, there are opportunities to recruit students
of different origins, and to raise and support multi-cultural awareness and social
support. Thus, capacity, knowledge, new ideas, and norms that are created
(Cheong, Wing & Yen, 2010) and socio-economic cohesion may be facilitated
(Pachura, 2010). Cluster strategies can also act as a bridge between the political
rhetoric and the observed reality (Doloreux & Shearmur, 2006).
Higher education is also mentioned as a cornerstone in the study object, but
more in connection with the importance of new research. General education is
also mentioned as important, and more accurately, the quality of schools (Ava-

208
los, 1996) and opportunities available through curricula (Rahimia et al., 2010).
Values are about the mental images that prevail and who prevail. Dichotomiza-
tion of urban and rural standards is evident above all in Swedish Research
(Svensson 2006; 2008; 2014). Cultural and linguistic aspects in certain regions are
language-related changes that can enhance or hinder regional development
(Daoud, 2010), and attitudes, as well as behaviors that may influence or be influ-
enced by culture (Karahanna, Evaristo, & Srite, 2005), such as settings for the
native language in school (Huguet & Llurda, 2001).
Conclusions
In summary, what appear to be common worldwide is the concerns pertaining
to urbanization, depopulation of rural areas; the need to highlight regional
development factors; as well as a belief in education. One result of this study is
that there are surprisingly few international studies on the relationship between
regional development and school improvement. Approximately 70% of all the
sources are published in Western countries, Europe and the USA/Canada. This
does not mean that the problems are greater there than in the rest of the world.
Perhaps instead, there are more research resources in these countries compared
to the rest of the world. The literature review shows that the study object was
observed more and more during the millennium. This may indicate that it will
be noticed increasingly more henceforth. More than half of all resources have a
qualitative approach, which means that the texts are descriptive. Needed are the
long-term, consistent empirical studies—preferably interventions studies—with
a mixed methods design.
Different regions, countries, and continents show various problems within this
focus. For example, regional educational development does not mean the same
thing in West Africa compared to northern Canada. Countries and continents
have different economic, structural, and cultural conditions and orientations.
Education in this context typically refers to higher education, while factors that
involve mainstream schools are not illustrated to any greater extent.
The literature review shows an internationally sprawling view of the content
and object of this study. It involves very different aspects, such as:
 Political issues, such as policy (Mc Dade & Spring,2005), ideological (Cf
Laurina, 2006; Quiang, 2011) or economic governance, the effects of
decentralization (Toi, 2010) and its repercussions on schools and student
learning and ideological solutions;
 Social capitals and network structures (Pachura, 2010), such as knowledge-
based c1usters as a guideline for regional development policies aimed at
stimulating regional industrial competitiveness and innovativeness,
with a focus on the importance to create eeducational achievement and
creative occupations (Florida et al,, 2010) and emphasize that creative
professionals are strongly related to regional income.
 Educational focus, mainly the connections between regions and
universities.
 Cultural and linguistic aspects and values within society and school
systems. Particularly interesting for the project V-brus is dichotomization
between urban and rural values (Svensson 2006; 2010; 2013)

209
Regarding the Swedish research in relation to international research, there are

three similarities, but otherwise they have pretty big differences. The emphasis
in education connected to regional development is on higher education and not
so much concerned with mainstream school. There are concerns about
urbanization and beliefs in education both worldwide and in Sweden. The
Swedish review shows examples of regional initiatives. The differences are that
there is much more emphasis on different cultural and political aspects from an
international perspective. The international literature is the national contingent,
the Swedish is more regionally-focused, and the international review shows the
importance of network clusters and the ability to inspire innovation through a
―new‖ creative class. If this new creative class should be created, developed, and
have influence in the regions, a political strong will is required for school
improvement in combination with regional development and research on a
broad front. But from another perspective one might also ask about the logics
and rurality and political rhetoric about the Swedish North (cf Nilsson &
Lundgren, 2015).
References
Ahuvia, A. (2008). Traditional, interpretative and reception based content analyses:
Improving the ability of content analysis to address issues of pragmatic and
theoretical concern. In R. Franzosi (ed.), Content Analysis: Vol. 1. (pp. 183–202).
London: SAGE.
Ambos, B., & Schlegelmilch, S. (YEAR). The new role of regional management. Vienna:
Palgrave Macmillian.
Avalos, B. (1996). Education for global/regional competitiveness: Chilean policies and
reform in secondary education. Compare, 26(2), 217–32.
Berglund, K., & Johansson, A. (2007). Entrepreneurship, discourses and conscientization
in processes of regional development. Entrepreneurship & Regional Development:
An International Journal, 19(6), 499–525. doi:10.1080/08985620701671833
Björkman, C. (2008). Internal capacities for school improvement: principals’ views in Swedish
secondary schools (Doctoral dissertation). Umeå University, Umeå.
Boström, L. (2015). Regional educational development research in Sweden: A literature review
of research over half a century (Unpublished paper, in progress). Mid Sweden
University
Cheong, C., Wing, N., & Yen, C. (2011). Development of a regional education hub: the
case of Hong Kong. International Journal of Educational Management, 25(5), 474 –
493. doi:10.1108/09513541111146378
Cooper, H. (1998). Synthesizing research: A guide for literature reviews. Thousand Oaks, CA:
SAGE.
Crescneszi, R. (2005). Innovation and regional growth in enlarged Europe: The role of
local innovative capabilities, peripherality, and education. Growth and Change,
36(4), 471–507.
Dalin, P. (1994). Skolutveckling. teori. Bok 2. [School development theory. Book 2].
Stockholm: Liber Utbildning AB.
Daoud, M. (2006) The Language Situation in Tunisia. Current Issues in Language Planning,
2(1), 1–52. doi:10.1080/14664200108668018
Doloreux, D., & Shearnur, R. (2006). Regional development in sparsely populated areas:
The case of Quebec‘s missing maritime cluster. Canadian Journal of Regional
Science, 29(2), 195–220.

210
Evans, C., & Waring, M. (2012). Applications of styles in educational instruction and
assessment. In L. F. Zhang, R. J. Sternberg, & S. Rayner (Eds.), The handbook of
intellectual styles (pp. 297–330). New York: Springer.
Florida, R., Mellander, C., & Stolarick, K. (2010). Talent, technology and tolerance in
Canadian regional development. The Canadian Geographer, 54(3), 277–304.
Franzosi, R. (2008). Content analysis: Objective, systematic, and quantitative description
of content. In R. Franzosi (Ed.), Content analysis, Vol. 1 (pp. xxi–xli). London:
SAGE.
From, J., & Olofsson, A. (2013). Kunskapsekonomi och regional utveckling. [Knowledge
economy and regional development]. In Y. Fiedrichs, M. Gawell, M. & J.
Wincent (Eds.), Samhällsentreprenörskap för lokal utveckling (pp. 17–42). Sundsvall:
Mittuniversitetet.
Goodland, J.(1994). The national network for educational renewal. Phi Delta Kappan,
75(8), 35–42.
Hargreaves, A., & Fink, D. (2008). Distributed leadership: democracy or delivery?
Journal of Educational Administration, 46(2), 229–240.
doi:10.1108/09578230810863280
Hernandez, F., & Goodson, I. F. (2010). Social geographies of educational changes:
Drawing a map for curious and dissatisfied travellers. In F. Hernandez & I.F.
Goodson (Eds.), Social and Geographies of Educational Changes (pp. xi–xxi).
Dordrecht: Kluwer Academic Publisher.
Höög, J., & Johansson, O. (2014). Framgångsrika skolor: mer om struktur, kultur, ledarskap.
[Successful schools: more about the structure, culture, leadership]. Lund:
Studentlitteratur.
Huguet, A., & Llurda, E. (2001). Language attitudes of school children in two
Catalan/Spanish bilingual communities. International Journal of Bilingual
Education and Bilingualism, 4(4), 267–282. doi:10.1080/13670050108667732
Karahanna, E., Evaristo R., & Srite, M. (2005). Levels of culture and individual behavior:
An integrative perspective. MIS Quarterly . Vol.30(3), pp.679-704
Larina, V. P. (2006). The innovative activity of schools in a regional system of education.
Russian Education and Society, 48(3), 31–42.
Lemar, S. (2006). Developing collaboration between schools and labour market in
vocational education training—A fruitful or futile ambition? conference
presentation.
Luo, W., & Chen, S. (2010). Fiscal decentralization and public education provision in
China. Canadian Social Science. 6(4), 28–41.
Marmolejo, F., &. Pukka, J. (2006). Supporting the contribution of higher education to regional
development: Lesson learned from an OECD review of 14 regions through 12 countries.
Paper presented at the UNESCO Forum on Higher Education, [Location].
McDade, B., & Spring, A. (2005). The ‗new generation of African entrepreneurs‘:
Networking to change the climate for business and private sector-led
development. Entrepreneurship & Regional Development, 17, 17–42.
McFarlane, C. (2010). The comparative city: Knowledge, learning, urbanism. International
Journal of Urban and Regional Research, 34(4), 725–42
doi:10.1111/j.1468-2427.2010.00917.x
Mittuniversitetet (2012). Projektplan Världens bästa regionala utbildningssystem.
Nilsson , B. & Lundgren, A.S. (2015). Logics of rurality: Political rhetoric about the
Swedish North. Journal of Rural Studies, 37, pp85-95.
Pachura, P. (2010). Regional cohesion: Effectiveness of network structures. Berlin: Springer.
Persson, L. O., Sätre Åhlander, A. M., & Westlund, H. (2003). Rural communities facing
challenges and conflicts. In L. O. Persson, A. M. Sätre Åhlander, & H. Westlund
(Eds.), Local responses to global changes (pp. 1–9). Stockholm: National Institute for
Working Life.

211
Polit, D., & Beck, C. (2008). Nursing research: Generating and assessing. Philadelphia:
Lippincott, Williams & Wilkins.
Rahimia, A., Ahmad, S., Borujenib, M., Esfahanic, A., & Liaghatdard, H. (2010).
Curriculum mapping: a strategy for effective participation of faculty members in
curriculum development. Procedia - Social and Behavioral Sciences 9, 2069–2073.
Santema, M. (1997). Regional Development and the Tasks of vocational Education and
Training Professionals. Journal of European Industrial, Training. 21(6), 229–237.
Shi, L. & Wei-qing, C. (2010). Fiscal decentralization and public education. Canadian
Social Science, 6(4), 28–41.
Skolverket. (2014). SALSA/SIRIS. Retrieved from http://www.skolverket.se/statistik-
och-utvardering/statistik-i-databaser. 2014-02-02
Statistiska Centralbyrån (2014). Utflyttande högskoleutbildade från Jämtlands och
Västernorrlands län. [Out-migration of college-educated in Jämtlands and
Västernorrlands counties compared to the whole of Sweden]. Retrieved from
http://www.scb.se/sv_/Hitta-statistik/, 2014-02-02
Svensson, L. (2006). Vinna och försvinna? Drivkrafter bakom ungdomars utflyttning från
mindre orter (Doctoral dissertation). Linköpings universitet, [Location].
Svensson, L. (2013). We don‘t want you to join us if you don‘t leave us! Economic and
Social Aspects of Peripheral Regions Special Issue, 4(99), 46–52.
Svensson, L. (2014). Nu styr vi upp stan. In M. Vallström (Ed.), När verkligheten inte
stämmer med kartan. Lokala förutsättningar för hållbar utveckling. Lund: Nordic
Academic Press.
Toi, A. (2010). An empirical study of the effects of decentralization (From, 2014) in
Indonesian junior secondary education. Educational Research for Policy and
Practice, 52(3), 107–125.
Tomaney, J., & Wray, F. (2011). The university and the region: An Australian
perspective. International Journal of Urban and Regional Research, 35(5), 913–31.
doi:10.1111/j.1468-2427.2011.01020.x
van Velzen, W., Miles, M. B., Ekholm, M., Hameyer, U., & Robin, D. (1985). Making
school improvement work. A conceptual guide to practice. Leuwen Acco: OECD-
publication.
Veugelers, W., &. Ziljstra, H. (2010). Networks of schools and constructing citizenship in
secondary education. . In F. Hernandez & I.F. Goodson (Eds.), Social Geographies
of Educational Changes (pp. 65–78). Dordrecht: Kluwer Academic Publishers.
Westlund, H. (2004). Regionala effekter av högre utbildning, högskolor och universitet: en
kunskapsöversikt. [Regional effects of higher education, colleges and universities:
a systematic review]. Östersund: Institutet för tillväxtpolitiska studier;
A2004:002
Whittemore, R., & Knafl, K. (2005). The integrative review: updated methodology.
Journal of Advanced Nursing, 52(5), 546–553.

212

Vol. 11, No. 1, pp. 212-226, April 2015
The Value-Added Assessment of Higher

Education learning: The case of Nagoya
University of Commerce and Business in Japan
Hiroshi Ito Surname and Nobuo Kawazoe

Nagoya University of Commerce and Business
Nisshin, Japan
Abstract. Assessment of higher education learning has been considered

increasingly important. One of the current trends in this field is the
value-added assessmenthow much students learn during a certain
period of time at university. In the United States, for example, Arum
and Roksa (2011) conducted a large-scale assessment on second-year
university students’ learning with the Collegiate Learning Assessment
(CLA) to examine how much university students improved generic
skills during the first two years of higher education. Findings suggested
that they did not improve much. The researchers concluded that the
poor result was attributed to the fact that American university students
on average study only 12 hours a week. In Japan, the situation may be
even worse as Japanese university students on average study 3.5 hours,
much less than their cohorts in the United States. However, studies on
the value-added assessment of Japanese university students’ learning
are scarce. With the Progress Report on Generic Skills (PROG), an
assessment tool similar to CLA, as well as interviews with students who
took PROG, this study quantitatively examines how much students
improved generic skills at a Japanese university during the first two
years of higher education. The findings show that as was the case of the
US peers, Japanese university students in this study did not improve
their generic skills very much in the first two years of higher education.
This study also qualitatively explores possible reasons for such results.
The findings also show that the researched students on average studied
only 40 minutes a week. This study suggests offering more courses with
active learning approaches to intrinsically motivate students in order for
them to spare more time for learning.
Keywords: higher education; learning assessment; PROG

213
Introduction
Assessment of higher education learning has been considered increasingly
important (Clouder et al., 2012; Kushimoto, 2010; Sambell et al., 2012). Yet,
measuring how much students learn has been a challenge for stakeholders
involved in higher education assessment (Hardison and Vilamovska, 2009).
While the grade point average (GPA) has traditionally been used for measuring
students’ academic performance at university, it is not considered a reliable
indicator for learning because grading varies according to institutes, instructors,
and other factors (Shavelson 2009). For instance, “As” in one institute and in
another institute (or from one instructor and another instructor) may not have
the same academic value. The implication of GPA from a certain year to another
is also difficult to interpret, as they do not measure the same components. Even
if a student’s GPA was “A” in the first year and “B” in the second year, it is
infeasible to determine that his or her learning deteriorated because the level of
academic content may differ in the first year and the second year. In this context,
various kinds of assessment tools have been created to measure university
students’ learning. This paper will review these assessment tools and then
quantitatively and qualitatively analyze how much students improved generic skills
at a Japanese university during the first two years of higher education.
Literature Review
Learning assessment tools

In the 1980s, tests such as the College Outcome Measures Program (COMP), the
Academic Profile, and the Collegiate Assessment of Academic Proficiency were
developed in order to measure the outcomes of general education programs.
Especially, COMP drew attention from stakeholders in higher education
assessment because it evaluated value-added learning: how much students
learned during a certain period of time. COMP has two different forms: the
Objective Test (multiple-choice questions) and the Composite Examination
(multiple-choice questions, essays, and speeches). The Objective Test comprises
of multiple-choice questions and the Composite Examination comprises of
multiple-choice questions, essays, and speeches. Despite its popularity, COMP
has been criticized for its inability to measure communication skills as well as
critical thinking skills.
In the late 1980s, the College Basic Academic Subjects Examination (College
BASE) was introduced. Apart from the subject content areas (i.e., English,
mathematics, science, social studies), the College BASE assesses generic skills:
interpretive reasoning, strategic reasoning, and adaptive reasoning. The College
BASE has three forms: 1) the long form with content areas, 2) the short form with
English and mathematics, and 3) an institutional-matrix form. As is the case with

214
COMP, however, a study revealed that the test assesses only a fraction of generic
skills (Pike, 2011).
It is in this context that CLA has emerged as one of the most popular assessment
tools of generic skills in higher education in the United States (Klein et al., 2007).
Other parts of the world now recognize CLA because the Organization for
Economic Cooperation and Development (OECD) has been developing
Assessment of Higher Education Learning Outcome (AHELO) based on CLA
(Douglas et al., 2012). CLA is an open-ended, value-added, performance
assessment tool that measures generic skills such as critical thinking, analytical
reasoning, problem solving, and written communication through writing tasks,
make-an-argument tasks, critique-an-argument tasks, and realistic performance
tasks (Council for Aid to Education, 2013).
In the United States, Arum and Roksa (2011) examined the CLA scores of 2,322
students at 24 universities over two years between the beginning of their first
year in 2005 and the end of their second year in 2007. The study indicates that
undergraduate students on average improved generic skills by 7%. While there
are no universal standards for learning in higher education, they argue,
students’ gains in academic performance were low. They concluded that the
poor result was attributed to the fact that the US college students on average
study only 12 hours a week.
Studying hours of university students in the US and Japan

According to the National Survey of Student Engagement (2011), even 20 hours
of studying time per week is not sufficient to fully prepare students for class.
Apparently, both instructors and students have “If you don't bother me, I won’t
bother you” attitudes. That is, Kuh (2003) explains, instructors do not make
students work hard so that they do not have to grade many papers or exams.
Peters (2011) also argues that university instructors are responsible for students’
low study habits. While students make as little effort as possible for their course
work, university instructors tend to minimize course preparation time. The
situation, however, may be even worse in Japanese higher education as Japanese
university students study much less than their peers in the United States. As
shown in Table 1, nearly 70% of Japanese university students study less than five
hours a week.

215
Japanese university students US university students

0 hours 9.7% 0.3%
1-5 hours 57.1% 15.3%
6-10 hours 18.4% 26%
11-15 hours 7.3% 22.3%
16-20 hours 3.2% 16.8%
21-25 hours 1.9% 9.8%
26- hours 2.4% 9.5%
Table 1. Studying hours per week
University Management and Policy Research Center/National Survey of Student
Engagement (Tsuji, 2013)
Tsuji (2013) estimates that the average number of hours spent studying by
Japanese university students is only 3.5 hours. He explains that while Japanese
university students are no less intelligent than students in other nations, their
analytical reasoning and problem-solving skills are not well developed due to
their lack of studying time. MacVeigh (2002) echoes Tsuji, stating that many
Japanese university students are unable to think and write critically and
logically and describes Japanese higher education as “a nationwide educational
failure” (p.4).
Tsuji (2013) argues that there exists what he calls “(p. 77) spiral” between
Japanese industry and higher education: companies’ human resources personnel
believe that university students do not study and they dedicate themselves to
their part time jobs and/or circles/clubs, but companies’ human resources
personnel want students to study to acquire generic skills that lead to
employability. University students claim that human resources personnel do not
consider GPAs for job applications. Students thus spend more time on part time
jobs and circles/clubs because they believe that it is more important for their
future employability. Instructors are afraid that if they made students study
hard, students would evaluate them poorly in the course evaluations. It is
therefore better for instructors to give “whatever” lectures without sufficient
preparation and spend more time on their research. Students then complain that
instructors do not teach them well and spend even more time on part time jobs
and circles/clubs instead of studying.

216
Human Resources Personnel:

we want university students to
study and acquire generic skills
for employability
! "
Students:
Students:
Human Resources Personnel
Instructors don't teach us well.
don't consider GPA for job
The classes are boring.
application. What is the point of
studying hard? # $
Instructors:
If we make students study hard,
they will be harsh on us in the
course evaluation. We give so-so
Figure 1. Negative Spiral (Tsuji, 2013)
In 2011, Tsuji (2013), as part of his NPO project, conducted surveys with 2,000
senior students in 28 departments at nine prestigious Japanese universities such
as Waseda, Keio, Hitotsubashi, and Sophia. The student participants reported
that only four out of approximate forty courses that they took at university
helped them learn to think. A student, for example, reported that “Instructors
didn’t ask us any questions. At the end of the semester, we were only given a
one-page report for evaluation.” Another student claimed that “Professors just
read textbooks in front of us.” Tsuji’s study indicates that Japanese university
students are not in an environment where they are encouraged to study and
develop their generic and employability skills. However, how can we know how
much students learn at university in Japan? One answer is to employ learning
assessment tools such as CLA and PROG.
PROG
This study employs PROG in an attempt to measure students learning. PROG
examines two sets of generic skills: literacy and competency. This usage of the
terms literacy and competency can be confusing as the elements of literacy and
competency overlap. For instance, the Organization for Economic Cooperation
and Development (OECD)’s definition of literacy—using tools interactively (e.g.
language, technology)—is one of its Definition and Selection of Competencies
(DeSeCo)’s key competencies. As Matsushita (2010) puts it, these tools include
non-cognitive elements such as social and emotional elements that are part of
competency (See Figure 2).

217
DeSeCo's Competencies PROG's Literacy and Competency
Literacy Competencies Literacy Competency
Figure 2. DeSeCo’s Competencies vs PROG’s Literacy and Competency

Based on (Matsushita, 2012)
In Japan, however, literacy is separated from competency (Matsushita, 2010).

Despite weaknesses in the terminology, this paper follows PROG’s usage of the
terms literacy and competency but uses them in italics to differentiate them from
OECD’s usage.
In the PROG test, 45 minutes are allocated for the Literacy section and 40 minutes
for the competency section. Literacy is composed of data collection, data analysis,
problem solving, and conceptual thinking skills. Critical thinking skills, which
are considered important generic skills, are partially integrated into the data
analysis skills. PROG’s literacy assessment also involves a few short essays to
measure written communication and other skills. These elements are similar to
what CLA examines. Competency is composed of skills in general
communication, collaboration, networking, leadership, negotiation, and stress
management as well as problem solving that is also included in literacy.
According to Kawaijuku and Riasec (2015), problem-solving skills in literacy are
tested on whether students can solve problems logically while problem-solving
skills in competency are tested to what extent students solve problems as young
professionals do (as described later). There are some notable differences between
CLA and PROG. CLA is composed of open-ended essays while PROG is based
on a combination of short essays and multiple-choice questions. Also, CLA is
designed to produce results at the institutional level such as school average
scores while PROG is designed to produce results at the individual level, which
is scores for each individual who takes the exam. PROG also provides feedback
sheets after the test with suggestions of how to improve generic skills.
The PROG score ranges from 1 to 7 for both literacy and competency. Score 4 is
the level desired to be reached by the end of the first year of university. Students
with this score are expected to be able to adequately understand and rephrase
information from documents and graphs. Score 7 is the level desired to be
reached by the time of graduation. Students with this score are expected to be

218
able to organize data and demonstrate information derived from the data in
academic writing and graphs. Students at this level are able to establish
arguments logically (Riasec, 2012).
The following problem is an example of a question that intends to test

conceptual thinking skills in PROG’s literacy section (Riasec, 2012).
You are going to do a group presentation about globalization for a

university seminar. Each group has 10 minutes, including questions and
comments. Each group can decide the aspects of globalization on which to
focus. You have 20 days until the presentation. Please make a plan,
showing the process of how you would proceed with the preparation,
taking the following elements into consideration:
a. Preparing a presentation
b. Collecting information and selecting ideas
c. Deciding a group theme
d. Analyzing information
e. Deciding the content
f. Practicing and modification
g. Reviewing the presentation
h. Deciding the roles
Students are expected to draw a flowchart that illustrates what they would do
and when they would do it (Riasec, 2012). PROG has a few short essay questions
that are similar to CLA’s make-an-argument prompt. For instance, students are
asked to read questions and answer in writing. The following is an example of
such as question.
A university student, who travelled to South Korea the other day, said that
while young Koreans did not understand Japanese, those who were close
to 80 years old whom s/he met were fluent in Japanese. Why do you think
elderly Koreans are able to speak Japanese fluently? Briefly write down the
reason(s).
While a student could write a creative story, for example, that the Koreans had
lived in Japan in their youth and learned Japanese, they are expected to write a
short essay based on his/her knowledge of Japanese colonial education in Korea
between 1910 and 1945.
How does PROG address competency? The following question is an example

from the competency section (Riasec, 2012).
You are a project leader of developing a new product at a company. One

of your subordinates came up with an interesting idea about a new

219
product. However, it is difficult to develop the idea into a product. In

order to do so, it is necessary to deal with X, a major company with a
marketing network, and Y, a venture enterprise with innovative
technology. However, your company has not dealt with them in the past.
What would you do? Choose one of the following options.
A. I would tell my subordinate that it is difficult to develop his idea into

a product though his idea is interesting
B. I would encourage my subordinate to think of a realistic method of
developing his idea into the product
C. I would tell my subordinate that we would keep the idea and wait for an
opportunity to arise
D. I would think of how to communicate with X and Y through
acquaintances
E. I would contact Y immediately to see if we could collaborate with each
other to develop the idea into a product
Arguably, there is no right or wrong answer for this question. How does PROG
score competency then? Kawaijuku and Riasec (2013) explain that they
administered the test to the young business leaders that are currently active in
society and collected sample data. They then analyzed the patterns of this
group’s answers to each question. That is, PROG attempts to measure students’
competency by comparing their answers with the young professional leaders’
answers and how similar students’ answers are to those of the young leaders.
Similar answers to young leaders’ answers score higher in competency while
dissimilar answers score lower.
According to Kawaijuku and Riasec (2014), as of April 2014, more than 100,000
university students took PROG. The average scores of PROG are 3.89 in literacy
and 3.22 in competency. Approximately 63% of the test takers were first year
students, 13.3% second year students, 19.7% third year students, and 3% fourth
year students.
Year Literacy Competency

First year students 3.82 3.20
Second year students 3.97 3.16
Third year students 3.97 3.23
Fourth year students 4.28 3.58
Total 3.89 3.22
Table 2. PROG Scores
As Table 1 shows, students’ scores in literacy improve through the entire course
of university; however, the results show that there exists no improvement from
the second to third year and then a remarkable improvement from the third to
the fourth year. In competency, scores deteriorated slightly from the second to
third year but improved from the third to fourth year. Although the number of

220
test takers is smaller than in other years, fourth year students substantially
improved in both literacy and competency. It may be important to note, however,
that Japanese fourth year students hardly go to university due to job hunting.
Their improvement in generic skills, therefore, may not be attributed to
university education, as described in the next section.
NUCB Education: Developing generic skills toward learning goals

The current research examines the learning of the first two years of education at
the Nagoya University of Commerce and Business (NUCB). This university has
participated in a national project entitled Improving Higher Education for
Industrial Needs. In the context of improving university students’ employability
to meet industrial needs, NUCB started focusing on generic skills and thus
employing PROG to assess students’ generic skills.
Given that this study examines how much students improved generic skills in
the first two years of NUCB education, it may be important to explain the first
year experience (FYE) program called the Vision Planning Seminar (VPS) and
the following year seminars at this university. The purpose of VPS is to help the
first year students acquire generic skills and envision the professional careers
based on the assumption that if they can envision their futures in the early stage
of university life, they should be able to set goals and work toward acquiring
skills necessary to achieve those goals. As shown in the Table 1, students are
explicitly expected to acquire generic skills such as critical thinking, analytical
reasoning, problem-solving and writing.
Critical Thinking Analytical Reasoning

ë ì
Developing Generic Skills
í î
Communication Problem Solving
í ê î
Writing Discussion Presentation
Figure 3. Purpose of NUCB Education: Seminars and Generic Skills

The table constructed based on the skills measured by CLA (Council for Aid to Education, 2013)
The generic skills acquired in VPS feed into the second year to fourth year
seminars and for writing the bachelor’s thesis. NUCB has set eight learning goals
(LGs) to be achieved before graduation. LGs are assessed through the following
skills:
1. Establishing a concrete, important, and feasible research theme

2. Acquiring academic knowledge and applying it to solve problems
3. Arguing convincing conclusions through proper processes
4. Expressing arguments through organized structure in writing
5. Creating and performing presentations

221
6. Communicating messages clearly and effectively

7. Utilizing information technology
8. Thinking and acting globally
Students are expected to demonstrate the first four of these eight skills in their
bachelor’s thesis and the last four in their second to fourth year seminars. These
skills are generic skills as well. For instance, the skills to set a research topic or
apply knowledge entail critical thinking, analytical reasoning, problem solving
and writing. After all, generic skills overlap with basic research skills in many
respects.
Methodology
The current research examines the PROG scores of 45 NUCB students who took
PROG tests twice, first in April or May 2013 and second in December 2014, and
analyzes the difference between these scores. That is, this study focuses on how
much NUCB students learn in the first two years of university. One of the
limitations of this study is its low sample size, which may hinder the
generalization of the result. Yet, this type of value-added assessment is still rare
in Japan and serves as an exploratory study. Some may also point out that the
research period of two years is insufficient for this type of longitudinal value-
added study. As Arum and Roksa (2011) affirm, however, “most of the gains in
generic skills occur in the first two years of college…seniors do not spend much
more time studying than freshmen” (p. 36-37). Although Kawaijuku and Riasec’s
(2014) study shows that Japanese university students substantially improved
generic skills in the fourth year, the majority of Japanese university students do
not attend regular courses in their fourth year and thus their improvement in
generics skills are more likely attributed to their own study for the Synthetic
Personality Inventory (SPI), an aptitude test for the selection of personnel, or
related exams to seek employment, not university course work. Thus, the first
two years of university learning is a reasonable indicator for the overall learning
at university.
Signed Wilcoxon’s rank sum test was conducted to examine whether there was a
significant difference in the medians of the PROG scores between April/May
2013 and December 2014 (p<0.05) In selecting a statistical hypothesis test, we
conducted the Lilliefors test based on the Kolmogorov-Smirnov test, which can
determine whether sample data is normally distributed and found that p-value
was less than 0.05 for all sample data, which shows that the sample data do not
seem to be normally distributed. We thus selected Wilcoxon’s rank sum over
matched sample t-test that examines the means of sample data difference.
The analysis of PROG scores is supplemented by interviews with students who

took PROG. The interview questions explore the following issues: how many
hours a week students study how many hours a week students work and how

222
they have found courses at NUCB. The result of the current research might also
be supplemented in the future by another longitudinal research of the same
students at the end of their fourth year.
Results
N Means SD Min. Max.
L1 45 3.09 1.427 1 6
C1 45 2.91 1.379 1 6
L2 45 3.16 2.225 1 7
C2 45 3.20 1.375 1 6
Table 3. Signed Wilcoxon’s rank sum test: Descriptive statistics
*L1: Literacy score in the first year; L2: Literacy score in the second year
*C1: Competency score in the first year; C2: Competency score in the second year
While students’ PROG scores improved by 0.07 (2.27%) in literacy and improved
by 0.29 (9.97%) in competency, the difference of the medians between L1 and L2
was not statistically significant at any critical value (p-value=0.394). The
difference of the medians of the medians between C1 and C2 was not
statistically significant at p<0.05 but significant at p<0.10 (p-value=0.057). Given
that Arum and Roksa (2011) consider 7% improvement in CLA for two years
“not much” in the US context, the PROG results of Japanese students can also be
interpreted as “not much.”
N Mean Rank Sum of Ranks

L2-L1 Negative 17a 16.59 282.00
Ranks 17b 18.41 313.00
Positive 11c
Ranks 45
Ties
Total
C2-C1 Negative 13d 15.23 198.00
Ranks 20e 18.15 363.00
Positive 12f
Ranks 45
Ties
Total
Table 4. Signed Wilcoxon’s rank sum test: Rank
a. L2<L1, b. L2>L1, c. L2=L1, d. C2<C1, e. C2>C1, f. C2=C1
In literacy, 17 students scored worse in the second test than in the first test while
17 students improved their scores. In competency, 20 students improved their
score while 12 worsened it. Overall, NUCB students little improved generic
skills for the first two years of higher education, though the results are not
statistically significant and further studies at a larger scale are required to
generalize the results.

223
With regard to interview results, while on average students spent 13.5 hours
working, they spent only 40 minutes studying per week. The studying hours of
NUCB students mark below the average studying hours of Japanese university
students, which are 3.5 hours, as indicated in the literature review section. The
maximum studying hours of NUCB students per week were three hours while
one third of the students reported that they did not study at all. Three-quarters
of students reported that they have encountered courses that they found
interesting such as marketing, management and statistics. However, the number
of courses that students found interesting is limited to a few out of many others.
Some students claimed: “There is not much difference in content between high
schools and universities,” “The university courses are boring,” “I always sleep in
class,” “High school teachers teach better than university professors,” “One class
has too many students,” and so forth. One student also lamented that she had
never received any feedback from professors regarding her assignments.
Discussion
This study indicates that in terms of PROG scores, NUCB students little
improved generic skills during the first two years of higher education. Literacy
did not improve at any significance level. While it is possible that NUCB
students have made a statistically significant improvement in competency at
p<0.10, though not at p<0.05, they might have acquired competency-related skills
through interactions with others off campus (e.g., part-time jobs).
Apart from descriptive interpretations of PROG results, there may also exist two
possible reasons/interpretations of the result: 1) PROG may not measure generic
skills that NUCB intends to develop (or NUCB may not develop generic skills
that PROG measures) and 2) (some) students make different levels of effort
during the first and the second PROG tests.
Regarding the issue of incompatibility of measuring and developing different

generic skills between PROG and NUCB, PROG, for example, does not measure
ethical standards or global perspectives (Akihiro Tanabe, executive operating
officer of Riasec, personal communication, February 18, 2015), both of which are
nowadays considered generic skills and are elements of NUCB mission
components. At the same time, while NUCB has set LGs to improve generic
skills and measure them through students’ self-evaluation and rubrics
developed by NUCB’s Assurance of Learning Committee, the university has
been unable to provide evidence that it has developed generic skills
comprehensively and objectively, as PROG results partially indicate. With
regard to a plausible chance that some students make different levels of effort at
the first and the second PROG tests, given that PROG results do not reflect on
their GPAs or anything else, students may not motivated to try their hardest

224
when taking the PROG test. At any event, however, the sampled NUCB students
study little as the study shows.
NUCB makes institutional efforts to increase students’ studying time by

establishing the self-study extra (SSE) point system. This system means that if
students hours studying in the library, they earn academic benefits. If they
spend 100 hours studying, for example, they can improve their grades by one
level in five subjects, for example, from B to A. NUCB also sets strict standards
for term-exams (one-fourth to one-third of test takers fail and must take re-
exams to pass the course). Yet, NUCB students do not study much, as the
surveys suggest.
Offering more courses that interest students is essential to intrinsically motivate

them to learn. Employing active learning approaches may be recommended. As
some scholars argue, active learning approaches help students learn to acquire
generic skills (Barkley, 2010; Harper & Quaye, 2009; Ito, 2014c). A study
examining an NUCB course employing active learning approaches also suggests
that it has contributed to improving generic skills such as: agility, adaptability,
and initiative; problem-solving skills; and curiosity and imagination (Ito, 2015).
This paper does not suggest that NUCB should completely redesign its
curriculum to focus on improving generic skills that PROG measures;
nevertheless, the university needs to better demonstrate how generic skills,
which it intends to develop as expressed in its mission statement and LGs, are
nurtured. Making and using rubrics to measure these skills is one option.
NUCB’s AOL has indeed developed rubrics to measure skills to achieve learning
goals, though the committee struggles with implementation because rubrics can
require substantial amount of time and effort to make and use. Administering a
test that measures NUCB generic skills (e.g., global perspective) is another
option. Riasec, the company that administers PROG, has flexibly dealt with
measuring generic skills that PROG cannot measure. In order to measure
students’ global perspectives, for example, the company has engaged 735
professionals (aged between 25 and 49), who worked for global companies and
managed foreign subordinates, to take the competency portion of the exam. These
data can be used to make comparisons with students to know how similar these
students are with global professionals: in other words, how global their
perspectives are (Kawaijuku & Riasec, 2014).
Although this paper discusses the case of a particular Japanese university, the
issue of developing and measuring generic skills are applicable to other contexts.
Any university has to identify generic skills needed by their students and its
institutional role in promoting generic skills. It then needs to develop or
implement an effective set of tools to measure students’ generic skills.

225
Conclusion
This study quantitatively examined how much students improved generic skills
at a Japanese university during the first two years of higher education and
qualitatively explored possible reasons for such results. The findings show that
Japanese university students did not improve their generic skills by much
during the first two years of higher education, arguably because students study
little during this time. This study suggests that offering more courses with active
learning approaches to intrinsically motivate students to spare more time for
learning can contribute to improving generic skills.
Acknowledgement
We are indebted to Mr. Hirotaka Nishio at NUCB for his support in this
research. We are also grateful to Mr. Ezra Anton Greene at University of British
Columbia for editing this paper.
References
Arum, R. and Roksa, J. (2011). Academically adrift: Limited learning on college campuses.
Chicago and London.
Barkley, E. F. (2010). Student engagement technique: A handbook for college faculty. New
York: Jossey-Bass
Clouder, L., Broughan, C., Jewell, S. & Steventon, G. (2012). Improving Student
Engagement and Development through Assessment. London and New York:
Routledge.
Douglas, J., Thomson, G. & Zhao, C. (2012). The Learning Outcomes Race: the Value of
Self-Reported Gains in Large Research Universities. Higher Education 64(3): 317-
335.
Hardison, C. M. and Vilamovska, A. (2009). The Collegiate Learning Assessment: Setting
standards for performance at a college or university. Santa Monica: Rand Education.
Ito, H. (2014a). Assessing an assessment tool of higher education: The case of PROG in
Japan. International Journal of Evaluation and Research in Education 3(1): 1-10.
Ito, H. (2014b). What’s wrong with learning for the Exam? An assessment-based
approach for student engagement. Journal of Education and Learning 3(2): 145-152.
Ito, H. (2014c). Shaping the First-Year Experience: Assessment of the Vision Planning
Seminar at the Nagoya University of Commerce and Business. International
Journal of Higher Education 3(3): 1-9.
Kawaijuku and Riasec. (2013). PROG: Progress Report on Generic Skills. Tokyo: Kawaijuku
and Riasec.
Kuh, G. D. (2012). What we are learning about student engagement. Change 35: 28.
Kushimoto, T. (2012). Outcome Assessment and its role in self-reviews of undergraduate
education: in the context of Japanese Higher Education Reforms Since 1990s.
Higher Education 59: 589-598.
McIntyre, J., Todd, N., Huijser, H., and Tehan, G. (2012). Building pathways to academic
success. A Practical Report. The International Journal of the First Year in Higher
Education 3(1): 109-118.
McVeigh, B. J. (2002). Japanese higher education as myth. New York and London: An East
Gate Book.
National Survey of Student Engagement. (2011). How much time college students spend
studying varies by major and corresponds to faculty expectations, survey finds.

226
Available at:
http://nsse.iub.edu/NSSE_2011_Results/pdf/NSSE_2011_Press_Release.pdf.
Peters, R. A. (2011). Enhancing academic achievement by identifying and minimizing the
impediments to active learning. Public Administration Quarterly 35(4): 466-493.
Pike, G. R. (2011). Assessing the Generic Outcomes of College: Selections from
assessment measures. San Francisco: Jossey-Bass.
Riasec. (2012). Measuring Generic Skills. Tokyo: Riasec.
Sambell, K., McDowell, L. and Montgomery, C. (2012). Assessment for Learning in Higher
Education. London and New York: Routledge.
Shavelson, R. (2009). Measuring College Learning Responsibility: Accountability in a New Era.
Palo Alto: Stanford University Press.
Tsuji, T. (2013). Nagze nihon no daigakusei wa sekai de ichiban benkyo shinai no ka [Why
Japanese university students study the least in the world]. Tokyo: Toyo keizai
shinbunsya.

Vol 11 No 1 - April 2015

Enviado por

Dados do documento

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

Vol 11 No 1 - April 2015

Enviado por

Direitos autorais:

Formatos disponíveis

IJLTER.

Using Particle Swarm Optimization Approach for Student Engagement Measurement............................................ 12

A Study of Formative Assessment Strategies in Teachers‘ School-Based In-Service Training ................................... 53

Designing and using interactive e-books in Vietnam ..................................................................................................... 75

The Concept of In Situ Lecturing ...................................................................................................................................... 128

International Journal of Learning, Teaching and Educational Research

Using Natural Language Processing Technology

Ming Liu, Weiwei Xu, Qiuxia Ran and Yawen Li

Abstract. Writing an essay is a very important skill for students to

Keywords: Writing Feedback, Text Analysis, Natural Language

© 2015 The authors and IJLTER.ORG. All rights reserved.

Since the early 1980s, researchers have investigated feedback on students’

With the advanced development of information technology and natural

© 2015 The authors and IJLTER.ORG. All rights reserved.

Table 1: Criterion Category

Criterion Category Examples

© 2015 The authors and IJLTER.ORG. All rights reserved.

3.1 Proposed Feedback Taxonomy

Table 2: Feedback Frequency and Pearson Correlations between Raters

© 2015 The authors and IJLTER.ORG. All rights reserved.

3.2 Textual Feature Extraction

Descriptive indices: It includes the number of paragraphs, number of sentences,

Cohesion: Cohesion is a key aspect of understanding language discourse

Sentence Complexity: The grammatical structure of a text is also an important

Lexical sophistication: Lexical sophistication refers to the writer’s use of advanced

© 2015 The authors and IJLTER.ORG. All rights reserved.

Introduction Portion: the ratios of number of words in introduction to the total

Conclusion Portion: the ratios of number of words in conclusion to the total

Percentage of grammatical errors: the ratios of the number of sentence with

© 2015 The authors and IJLTER.ORG. All rights reserved.

3.3 Pearson Correlation

Table 3: Correlations between Textural Features Scores and Raters’

© 2015 The authors and IJLTER.ORG. All rights reserved.

3.3 Test Set Model

© 2015 The authors and IJLTER.ORG. All rights reserved.

score on a scale of 1 to 3 to each feedback category of each student essay. The

© 2015 The authors and IJLTER.ORG. All rights reserved.

2 Uses simple sentence with minor errors (between 5-7).

Anderson, J. (2005). Mechanically Inclined:Building Grammar, Usage, and Style into

© 2015 The authors and IJLTER.ORG. All rights reserved.

© 2015 The authors and IJLTER.ORG. All rights reserved.

International Journal of Learning, Teaching and Educational Research

Using Particle Swarm Optimization Approach

Ming Liu, Yuqi Wang, Hua Liu, Shujun Wu and Chang Li

Abstract. Measuring Student Engagement is a difficult task. Previous

Keywords: Student Engagement Measurement, Advanced Educational

© 2015 The authors and IJLTER.ORG. All rights reserved.

The measurement of behavioral engagement is more obvious because behavioral

© 2015 The authors and IJLTER.ORG. All rights reserved.

Walkthrough Checklist, observers such as administrators, instructional

© 2015 The authors and IJLTER.ORG. All rights reserved.

Engagement Visualization and Measurement

Engagement is critical to the success of learning activities such as writing, and

© 2015 The authors and IJLTER.ORG. All rights reserved.

Particle Swarm Optimization

Vidt  Vidt 1  c1r1  pbestidt  X idt   c2r2  gbestidt  X idt 

When the iteration number reaches a pre-determined maximum iteration

PSO enhanced Engagement Measurement Algorithm

MSE is a common evaluation measurement for numeric value prediction, which

© 2015 The authors and IJLTER.ORG. All rights reserved.

© 2015 The authors and IJLTER.ORG. All rights reserved.

Figure 2: the user interface in the online writing system

© 2015 The authors and IJLTER.ORG. All rights reserved.

Table 1: Correlation of engagement time

教育部 (1999). 國小英語師資培育檢核相關報導。 Retrieved from