Proceedings 11th Annual Conference

11th Annual Conference of the Subject Centre for
Information and Computer Sciences

24th – 26th August 2010
Durham University

Edited by: Higher Education Academy ICS Subject Centre

Copyright © HE Academy
£35.00
ISBN 978-0-9565220-1-6
Published by:
HE Academy, Subject Centre for ICS
Faculty of Computing & Engineering
University of Ulster
Shore Road, Newtownabbey
Co Antrim, BT37 0QB
Welcome
It is again my pleasure to welcome all to this our 11th Annual Conference and on behalf of the
Organising Committee, both in the Subject Centre and at the Durham University, I thank all who
have contributed to the event. Each year we are dependent on key staff within the host institution
to coordinate arrangements and, once again, we have had tremendous support at Durham
University. As usual we have also been dependent on many colleagues who have refereed papers
and contributed in other ways.
The programme itself is indicative of sustained activity from our two communities with offers of
workshop provision, poster and formal presentations. This year, our keynotes are Professor Mark
Guzdial and Dr Adam Farquar.
Mark is a Professor in the School of Interactive Computing in the College of Computing at Georgia
Institute of Technology. His research focuses on learning sciences and technology, specifically,
computing education research. He has published several books on the use of media as a context for
learning computing. He was the original developer of the "Swiki" which was the first wiki designed
for educational use. He received the Ph.D. degree in Education and Computer Science from the
University of Michigan in 1993. He serves on the both ACM's Education Board and the Special
Interest Group in CS Education (SIGCSE) Board, and is on the editorial boards of the "Journal of the
Learning Sciences," "ACM Transactions on Computing Education," and "Communications of the
ACM." His keynote, titled “Meeting everyone’s need for computing” presents methods for teaching
computing that have improved success rates for non‐computing majors in America.
Adam is Head of Digital Library Technology at the British Library, where he co‐founded the Library's
Digital Preservation Team (www.bl.uk/dp) and initiated the Library's dataset programme. He is Co‐
ordinator and Scientific Director of the EU co‐funded Planets Digital Preservation project
(www.planets‐project.eu) and was a lead architect on the BL's Digital Library System. He is President
of DataCite, an international association dedicated to making it easier to identify, cite, and reuse
scientific data; founder and Chairman of the Open Planets Foundation; and serves on the Board of
the Digital Preservation Coalition. Prior to joining the Library, he was the principal knowledge
management architect for Schlumberger (1998‐2003) and research scientist at the Stanford
Knowledge Systems Laboratory (1993‐1998). He completed his PhD in Computer Sciences at the
University of Texas at Austin (1993). Over the past twenty years, his work has focused on improving
the ways in which people can represent, find, share, use, exploit, and preserve digitally encoded
knowledge. His keynote, titled “Caring for our digital future” outlines some of the key risks to digital
information and recent progress that we have made to address them and ensure the longevity of the
UK’s digital cultural and scientific heritage.
We are delighted to have such eminent speakers and with this and the other streams of the program
I trust you will find the conference of use and take some of the experiences learned back to your
own institution. Ultimately the outcome of our event must have an improvement on the learning
experience of our students.
Gerry McAllister
Director
Higher Education Academy Subject Centre for Information and Computer Sciences
1
COMMITTEE
Conference Organizing Committe
Karen Fraser, Hazel White, Barbara Hunter, Lesley Anne Fitzgerald, Giuseppe Trombino
and Stephen Hagan, University of Ulster.
Local Organising Committee
Janet Lavery and Liz Burd, Durham University.
2
CONTENTS
WELCOME 1
Gerry McAllister, University of Ulster
COMMITTEE 2
Conference Co-Chairs, Programme Committee, Referees and Conference Committee
PAPERS
The Problems Of Teaching Programming: Do They Change With Time? 6

Janet Carter, University of Kent and Tony Jenkins University of Leeds
Using The Principles Of Variation To Create Code Writing Problem Sets 11

Errol Thompson, University of Birmingham
A Mad Approach To Java Concepts 17

Simon Sharpe, Mark Dixon and Gill Harrison, Leeds Metropolitan University
Supporting The Learning Of Object Oriented Design With A Lightweight Web-

Based Collaborative Modelling Environment 22
Quan Dang and Tingkai Wang, London Metropolitan University
Towards an Interactive Multimedia-based Environment for Teaching Academic

Writing 27
Alma Whitfield, Obinna Anya, Stewart Blakeway and Nabil Sultan, Liverpool Hope
University
On Second Life As An E-Learning Environment For Teaching Academic Writing 32

Obinna Anya, Alma Whitfield, Janette Bonar-Law and Stewart Blakeway, Liverpool
Hope University
Education in Library Management Systems (ELMS) 38

Alan Poulter, Strathclyde University
Inquiry-Based Learning And Information Literacy With International Masters

Students 42
Briony J Oates, Elisabeth Yaneske and Gary Griffiths, Teesside University
Enhancing Student Engagement Through Pair Programming 47

Sonya Coleman and Eric Nichols, University of Ulster
Teaching Media And Information Literacy To Postgraduate Researchers 52

Andrew Whitworth, Steve McIndoe and Clare Whitworth, University of Manchester
Anytime-Anywhere Student Support Through Virtual Mentors 58

J.C. Augusto, P. McCullagh, A. McRoberts and V. McNair, University of Ulster
Teaching Problem Solving And Ai With Pac-Man 63
Jim Smith and Steve Cayzer, University of the West of England
Student Engagement And Modes Of Study: What Can Vles Tell Us? 68
Nick Efford, Roger Boyle and Royce Neagle, University of Leeds
Enhancing Student Engagement And Support With Digital Video 73

Paul Newbury and Phil Watten, University of Sussex
Tags As A Feedback Mechanism In Programming Courses: Analysis To Support

Just-In-Time Teaching 78
Stephen Cummins, Liz Burd and Andrew Hatch, Durham University
Turnitoff – Defeating Plagiarism Detection Systems 84

Lee Gillam, John Marinuzzi and Paris Ioannou, University of Surrey
Transition, Engagement And Retention Of First Year Computing Students 90

H.M. Sayers, M.A. Nicell and A. Hinds, University of Ulster
Re-thinking Student Induction 96

Mark Lee and Gale Dawson, University of Birmingham
Widening Access, Cpd And Work-Based Learning Using Asynchronous Dialogic

Learning: Aberystwyth, A Case Study 101
Judith Broady-Preston, Aberystwyth University
Engaging Schools With Computer Programming 106

Laura Crane and Phillip Benachour, Lancaster University
Listening Step By Step 112

Janet Lavery and Liz burd, Durham University
Sustaining Sustainability: Developing Material For Higher Education 117

Neil Andrew Gordon, University of Hull
Developing Sustainable International Partnerships: A Case Study Of The Indian

Sub Continent 123
Dharmendra Shadija and Neil Richardson, Sheffield Hallam University and Richard
Hill, University of Derby
Cutting Wedge Or Cutting Edge? - Embedding E-Learning For Engagement 128

Paul Carden, London South Bank University
Introducing A.L.L. To Computing 134

Lisa Payne, Coventry University
Using EVS And ResponseWare To Enhance Student Learning And Learning
Experience 141
Emma Cliffe, James H. Davenport, Marina De Vos, Nitin R. Parmar and Alan Hayes,
University of Bath
Engaging The Attention Of Technical Students In Learning Business Skills 147

Marion Gerson, Sheffield Hallam University
Deep Learning Via Student Engagement With Semiautomated Lecture Transcripts 152
Marcus Lynch and Philip Phelps, University of the West of England
Technology Enhanced Learning: Systems To Increase Engagement &

Participation During Lectures 157
Andy Stephenson and Liz Burd, Durham University
Investigating ‘Voice Email’ Technology Efficacy In Information Management

Assessment 162
George Macgregor, Alex Spiers and Chris Taylor, Liverpool John Moores University
The Impact Of Peer Assessment On Academic Success 167

Marie Devlin, Chris Phillips and Lindsay Marshall, Newcastle University
Staff Led Individualised Assessment – A Case Study 173

Thomas Lancaster and Robert Clarke, Birmingham City University
The Influence Of Presentation Upon Examination Marks 178

J. Hughes and S. Akbar, University of Dundee
An Evaluation Of The Durham Tec Mobile Educational Software System In The

Context Of Pask-Laurillard Conversational Frameworks 183
Michael Beatty and Phyo Kyaw, Durham University
The Use Of Simulation In Digital Forensics Teaching 187

Jonathan Crellin, Mo Adda and Emma Duke-Williams, University of Portsmouth
Sharing Practice For Computing Educators 193

Janet Finlay, Leeds Metropolitan University and Sally Fincher University of Kent
Facilitating Reuse Of Learning Resources: A Tool To Support Self-Deposit And

Automatic Metadata Generation 198
Dawn Wood and Janet Finlay, Leeds Metropolitan University
Enhancing International Student Engagement Through Making And Sharing Films 203
Andrew Cox and Nashrawan Taha, University of Sheffield
You Can’t Solve A Jigsaw With Only One Piece – Why Cross-Topic Lta Is Key To
Student Engagement 208
Siobhan Devlin, University of Sunderland
An Investigation Of The Successful Application And Development Of A Quick,
Easy To Use, Cross-Disciplinary Information Evaluation Matrix 212
Mike Leigh, Lucy Mathers and Kaye Towlson, De Montfort University
Fostering Social Competence, Community Awareness And Employability Skills

Through Volunteering And Emotional Intelligence (EI) Skills 222
Lynne Humphries, University of Sunderland
Playing With Clouds: Making Web Application Assessments More Realistic 229
Bruce Scharlau, University of Aberdeen
Investigating The Use Of Web 2.0 Tools And Open Educational Resources For ICS 234
Tina Wilson and Giselle Ferreira, The Open University
Virtual Worlds For Computer Science Education 239

Alan Miller, Colin Allison, John McCaffery, Thomas Sturgeon, Ross Nicoll, Kris
Getchell, Indika Perera and Iain Oliver, University of St Andrews

THE PROBLEMS OF TEACHING PROGRAMMING: DO THEY
CHANGE WITH TIME?
Janet Carter Tony Jenkins

Computing Laboratory School of Computing
University of Kent University of Leeds
Canterbury, Kent Leeds, W Yorks
J.E.Carter@kent.ac.uk tony@comp.leeds.ac.uk
www.cs.kent.ac.uk/~jec/ www.comp.leeds.ac.uk/tony/
ABSTRACT
It has now been 10 years since the first annual HEA-ICS 1-day Teaching of Programming conference was
held in Leeds. Each year people have reported on their initiatives: to help struggling students; to motive the
top of the class; to make learning more interesting and relevant to the entire cohort. What has worked? What
has failed? Are the current issues exactly the same as they were in 2000?
In this paper we attempt to answer some of these questions. We have undertaken a literature review of
papers presenting both problems and initiatives and have summarised and synthesised their conclusions. As
well as searching for perennial issues and good ideas that are still in use we have identified some that have
been discontinued; forgotten, impractical, or ineffective.
This paper addresses two of the three key issues for the HEA-ICS conference this year: quality enhancement
and assessment; and student engagement. There have been many significant impacts upon the
programming aspects of Computer Science over the last 10 years. For example: many Universities (not just
in the UK) now teach Java as a first programming language and many of those have tried using BlueJ as an
initial learning environment. We will bring together opinions that have been expressed and tips for coping in
order to consolidate the information that is available.
Academics with increasing teaching loads and diminishing preparation time do not want to spend what little
time they do have searching through archives of papers reporting the highs and lows of other people’s
experience.
The aims of this work are to synthesise our findings and provide a digest of issues and initiatives along with a
short recommended reading list. “We’ve done the hard work so you don’t have to.”
Keywords
Teaching methods, motivation, programming.
1. INTRODUCTION
The HEA-ICS 1-day teaching of programming conferences began in Leeds in 2001. The idea was to bring
together like-minded people who were struggling to come to terms with the fact that, despite our best efforts,
our attempts at teaching programming were still leading to final year students graduating without being able to
write code.
“Few would disagree that learning to program is a fundamental part of degree-level education in computing.
Fewer would disagree that teaching programming effectively in today’s mass Higher Education is a problem.
The graduating student who professes a complete inability to write the simplest program is commonplace”
(Jenkins 2001).
One obvious corollary of this is that if students don’t engage with the material they are not going to expend the
effort required to learn and are going to find other means of obtaining the marks they require to pass our
courses. The spectre of plagiarism immediately raised its head.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy
otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission.
© 2010 Higher Education Academy
Subject Centre for Information and Computer Sciences
6
2. MOTIVATION
Motivation has been a common thread throughout; starting with the observation that the majority of university
level courses offer a similar experience to all students. Davis et al (2001) argue that in the teaching of
introductory programming this practice has become increasingly difficult to justify. Beginners find the pace too
quick and are intimidated by the more vocal students who appear to know it all. This leads to students at both
ends of the ability spectrum expressing “unambiguous dissatisfaction” (Davis 2002). It is clear that the main
role of the teacher is less information dissemination and more motivation coaching (Jenkins 2001).
There have been several initiatives aimed at increasing student motivation and engagement. The use of toys
such as clickers (Davis 2002) and robots (Jadud 2005, Lawhead et al 2003) have successfully been tried on
several occasions, as has story telling and narrative (Curtis 2004). Some less effective strategies, such as
acting (Scott 2009), musical composition (Hamer 2004), and Judo grading (Jamieson 2003), have
experienced partial success but may be categorised as gimmicks rather than something that seamlessly
blends with the topic being taught.
The one thing that all the effective strategies have in common is that they have related to the personality and
charisma of the individual rather than being a transferrable package that will work wholesale for everyone.
The most obvious (but not actually confirmed by principled research) conclusion is that a keen and
sympathetic teacher who obviously cares about their students is the key ingredient.
3. IN THE CLASSROOM
With Computer Science (CS) Education being a relatively new research area there has been a rash of papers
describing different techniques and learning styles and how they apply, or otherwise, in the CS classroom.
Whilst it is a given that if students don’t engage with the material they aren’t going to learn, it has been useful
to apply labels and correct terminology to what already happens – providing a shared vocabulary.
Ben-Ari (2001) introduced the language of constructivism to CS. Constructivism is a theory of learning, which
claims that students actively construct knowledge rather than passively store information received from the
teacher. It is a theory that had been influential in science and mathematics education for some time, but
much less so for CS education. This oft cited paper paved the way for a number of studies into the
applicability of different educational theories.
Kolb (1984) described a theory of learning containing four distinct learning styles, which in turn are based on a
four-stage learning cycle. Kolb’s model offers both a way to understand individual people's different learning
styles, and also an explanation of a cycle (or more correctly a spiral) of experiential learning that applies to us
all. Concrete experiences provide a basis for reflection which in turn leads to assimilation and then an action
that in turn creates new experiences. Basing their work upon Kolb and a Myers-Briggs (1994) personality
typing test, Benest et al (2005) investigated CS students’ preferred ways of learning. Their rationale being
that although we cannot change a student’s personality, we can, if necessary, adjust our teaching strategies
to better accommodate them. The research showed that CS students differ quite markedly from the general
population by indicating a strong preference for organisation, structure and schedules.
Fuller et al (2007) investigated the efficacy of simply applying taxonomies (descriptions and classifications of
the stages involved in learning) of learning developed in other disciplines directly to CS. They reviewed the
literature and created a taxonomy which can be used for CS, especially introductory programming. Also in
2007, Pears et al undertook an international survey of literature relating to the teaching of programming; their
aim was to answer the question “what body of research literature can inform instructors when designing a new
introductory programming course?” The working group comprised eight academics from around the world,
including two from the UK.
3.1 Start From Where They are at

One important principle of many learning theories is that of starting from where the students are at; i.e.
keeping the gap between where you start and what students already know as small as possible; students can
learn much more if they are not left floundering at the outset. With this in mind McChesney and Alexander
(2007) undertook a large scale survey of 500 new UK CS undergraduates. The survey confirmed previously
recognized factors affecting student choice, such as previous computing interests, personal choice, and
career aspirations, as well as providing insights into factors such as the specific nature of the computing
interests and activities of new students, and their career motivations.
Barnes et al (2008) used the McChesney and Alexander (2007) survey as a basis for developing an induction
programme for new undergraduates; their particular area of interest was the improvement of social
interactions. In 2009 Lavery also reported on an induction programme aimed at helping students get to know
7
their peers and members of the lecturing staff, which has also evolved into a programme that helps students
to better understand the degree programme they have chosen.
3.2 Differentiation in the Classroom

Jenkins and Davy (2000) used an aptitude test to categorize students as Rocket Scientists, Averages or
Strugglers. Rocket Scientists were already highly proficient programmers before starting the course, whilst
Strugglers were those who would need support to help them achieve. Birch et al (2005) also discovered that
streaming was effective. Carter and Boyle (2002) reported on many different teaching initiatives that
instructors find helpful; including the setting of a programming essay to help students appreciate the concepts
without stumbling over the syntax.
In recent years, as retention has become an ever important issue, work has also been done to maintain the
motivation of the students at the top of the class to help reduce drop out rates. Successful techniques have
included inter university competitions (Carter et al 2007). The activity and judging process were designed to
retain student motivation and to value the integration of professional and technical skills, and the approach is
seen as a useful addition to existing teaching methods.
Another obvious mechanism for maintaining student motivation has been pair programming, although when
Keenan and Coleman (2003) asked their students’ opinions on the issue they, perhaps unsurprisingly,
discovered that none of their students wanted to work with a partner that was perceived as being weaker.
Barrett and Rainer (2004) did, however, determine that pair programming does reduce incidences of
plagiarism.
3.3 Plagiarism
When we can’t motivate all our students to try to learn the material we present, how do they attempt to gain
marks? Jenkins and Helmore (2006) discovered just how easy it now is to buy tailor-made assessment
solutions that won’t trigger plagiarism detection tools. Lancaster and Clarke have continued this theme by
investigating contract cheating sites (2007) and highlighting simple methods students can use to cheat simply
because academic institutions sometimes make it very easy to do so (2008).
3.4 Language Wars?

Back in 2001 Java was an upcoming language. A short while later it would not be surprising to find that every
programming teacher in the room was teaching it as a first language; many using BlueJ to do so. Now there
seems to be a quiet and almost stealthy backlash; not exactly a language war rather an acceptance that Java
has become too overblown and complex and that something simpler might be better as an initial language.
Some have turned to Alice (Eccleston 2007), but increasing numbers are turning to Python, which is actually
used in industry, because it is possible for students to write “reasonably impressive programs with a very
small amount of Python code” (Jenkins and Efford 2008).
4. SUMMARY
In this paper we have identified four major issues which, although they have changed focus over the years,
have nevertheless remained important.
1. Plagiarism – detection tools have changed and improved with time, but are no more effective now since
students can easily buy code from external sources.
2. Motivation – there has been a shift of emphasis over the years. We now recognise that it is important to
motivate the entire cohort not simply to teach to the middle and provide remedial help for strugglers.
3. Expectation – we now try to address the mismatch between students’ initial expectations and ours.
4. Fun – we still try to make things fun in the classroom, the emphasis was robots but has now shifted to
games.
Do the problems change with time …?
5. REFERENCES
Slides from HEA ICS presentations may be downloaded from the HEA ICS events archive,
http://www.ics.heacademy.ac.uk/events/archive.php
 Barnes S, Gardener J and Quigley S, (2008), Undergraduate Induction: we did it our way, In
proceedings of 9th annual HEA ICS conference, HEA ICS, Liverpool
8
 Barrett R, Malcolm J and Lyon C, (2003), Are We Ready For Large Scale Use of Plagiarism Detection
Tools?, In proceedings of 4th annual LTSN conference, LTSN, Galway
 Barrett R and Rainer A, (2004), 'With a Little Help from my Friends...', In proceedings of 4th annual
LTSN conference, LTSN, Galway
 Ben-Ari M, (2001), Constructivism in Computer Science Education, In Journal of Computers in

Mathematics and Science Teaching, 20(1), pp 45-73
 Benest I, Carter J and Chandler J, (2005), The Personality and Attitude Traits of Computer Science
Students, In INNOVATIONS 2005: World Innovations in Engineering Education, chapter 26, pp 291-
304, Begell House Publishing, Redding CT USA
 Birch M, McCormick F and Haddo J, (2005), Improving Student Progression by a combination of

Streaming, Close Attendance and Target Setting, In proceedings of 6th annual HEA ICS conference,
HEA ICS, York
 Carter J and Boyle R, (2002), Teaching Delivery Issues: Lessons from Computer Science, In Journal
of Information Technology Education, 1(2), pp 65-90
 Carter J, Efford N, Jamieson S, Jenkins T and White S, (2007), The TOPS project - teaching our over-
performing students, In proceedings of 8th annual HEA ICS conference, HEA ICS, Southampton
 Curtis S, (2004), Signposts and Dialogues, In proceedings of 4th Annual LTSN 1-day conference on
the teaching of programming, Birmingham
 Davis HC, (2002), Approaches to dealing with Diversity in Introductory Programming, In proceedings
of 2nd Annual LTSN 1-day conference on the teaching of programming, LTSN, Wolverhampton
 Davis HC, Carr L, Cooke E and White S, (2001), Managing Diversity: Experiences Teaching
Programming Principles, In proceedings of 2nd annual LTSN conference, LTSN, London
 Dick M, Sheard J, Bareiss C, Carter J, Joyce D, Harding T and Laxer C, (2003), Addressing student
cheating: definitions and solutions, ACM SIGCSE Bulletin, 35(2), pp 172-184
 Eccleston G, (2007), Using Alice to develop Introductory Programming Skills, In proceedings of 7th
annual HEA ICS 1-day conference on the teaching of programming, HEA ICS, Canterbury
 Fuller U, Johnson C, Ahoniemi T, Cukierman D, Hernan-Losada I, Jackova J, LahtinenE, Lewis T,

McGee Thompson, Riesdel c and Thompson E, (2007), Developing a computer science-specific
learning taxonomy, In ACM SIGCSE Bulletin, 39(4), pp 152-170
 Hamer J, (2004), An Approach to Teaching Design Patterns using Musical Composition, In

proceedings of 9th annual ITiCSE conference, ACM SIGCSE, Leeds
 Jadud M, (2005), Toys + Motivation = Cool Stuff in Computer Science, In proceedings of 5th annual
HEA ICS 1-day conference on the teaching of programming, HEA ICS, Oxford
 Jamieson S, (2003), Introductory Programming Meets The Martial Arts, In proceedings of 3rd Annual
LTSN 1-day conference on the teaching of programming, LTSN, Huddersfield
 Jenkins T, (2001), Teaching Programming – A Journey from Teacher to Motivator, In proceedings of

2nd annual LTSN conference, LTSN, London
 Jenkins T, (2003), Pragmatic Possibilities of Pre-Programming with Python, In proceedings of 3rd

Annual LTSN 1-day conference on the teaching of programming, LTSN, Huddersfield
 Jenkins T and Davy J, (2000) Dealing With Diversity in Introductory Programming paper, In
proceedings of 1st annual LTSN conference, LTSN, Edinburgh
9
 Jenkins T and Efford N, (2008), Cool Python Stuff, In proceedings of 8th annual HEA ICS 1-day
conference on the teaching of programming, HEA ICS, Glasgow
 Jenkins T and Helmore S, (2006), Coursework for Cash: The Threat from On-Line Plagiarism, In
proceedings of 7th annual HEA ICS conference, HEA ICS, Dublin
 Keenan F and Coleman G, (2003), Extreme Programming: Results of an Educational Experiment, In

proceedings of 4th annual LTSN conference, LTSN, Galway
 Kolb D, (1984), Experiential Learning. Experience as the Source of Learning and Development,
Prentice-Hall, New Jersey
 Lancaster T and Clarke R, (2007), Assessing Contract Cheating Through Auction Sites - A
Computing Perspective, In proceedings of 8th annual HEA ICS conference, HEA ICS, Southampton
 Lancaster T and Clarke R, (2008), How to Succeed in Cheating Without Really Trying - Five Top Tips
for Successful Cheating, In proceedings of 9th annual HEA ICS conference, HEA ICS, Liverpool
 Lavery J, (2009), Evolution of an Induction Programme, In proceedings of 10th annual HEA ICS
conference, HEA ICS, Canterbury
 Lawhead P, Bland C, Barnes D, Duncan M, Goldweber M, Hollingsworth R and Schep M, (2003), A

Road Map for Teaching Introductory Programming Using LEGO Mindstorms Robots, In ACM SIGCSE
Bulletin, 35(2), pp 191-201
 McChesney I and Alexander S, (2007), First-year computing students -personal and contextual
factors in course choice, In proceedings of 8th annual HEA ICS conference, HEA ICS, Southampton
 Myers IB, (1994), Introduction to Type, 5th edition, Oxford Psychologists Press Ltd, Oxford
 Pears A, Seidman S, Malmi L, Mannila L, Adams E, Bennedsen J, Devlin M and Paterson J, (2007), A
Survey of Literature on the Teaching of Introductory Programming, In ACM SIGCSE Bulletin, 39(4),
pp 204-223
 Scott E, (2009), Actions speak louder than words, In proceedings of 9th annual HEA ICS 1-day
conference on the teaching of programming, HEA ICS, Bath
10
USING THE PRINCIPLES OF VARIATION TO CREATE CODE WRITING
PROBLEM SETS
Errol Thompson
School of Computer Science
University of Birmingham
Edgbaston Park Road, Birmingham,
B15 2TT
United Kingdom
kiwiet@acm.org
http://www.thompsonz.net/work/teac
hing.html
ABSTRACT
Code writing problems vary in complexity and difficulty. The style of student solutions also varies. In this
paper, we look at coding problems used in an introductory programming paper and using the principles
derived from the SOLO taxonomy, endeavour to rank the complexity of the questions and their solutions.
We argue that initial problems should be directly translatable and have direct translation solutions. Question
complexity should then allow for the variant answer types before moving to more complex question types.
Through using this strategy students are also introduced to problem solving techniques.
Keywords
PROGRAMMING EXERCISES, SOLO TAXONOMY, VARIATION THEORY.
1. INTRODUCTION
"If students are to learn desired outcomes in a reasonably effective manner, then the teacher's fundamental
task is to get students to engage in learning activities that are likely to result in their achieving those outcomes
. . . It is helpful to remember that what the student does is actually more important in determining what is
learned than what the teacher does." (Shuell, 1986).
Despite the time that students work on programming tasks, they still request in course more exercises so that
they can better understand the programming constructs and develop their understanding of the software
development process. From the lecturer’s perspective, how can problem sets be developed that help the
learner engage with the subject and to develop the required programming skills?
This paper presents one approach to developing programming problem sets. Experience with using
progressive problem sets suggests that they do assist with the learning of programming skills. There is a
wealth of educational research and an increasing body of research into the learning of programming that
should also inform a lecturer designing a programming course.
2. BACKGROUND
2.1 Categorising Questions and Answers
In doing the analysis of code reading exercises for the BRACELet research (Clear et al., 2008; Whalley, et al.,
2006), criteria were developed based on the SOLO taxonomy (Biggs & Collis, 1982). The code reading
categories focus on the degree to which the learner sees the code as a whole rather than focusing on the
parts. In Lister, et al. (2009), the first attempt at defining categories for code writing exercises were developed.
These were based on an example for language translation from Hattie and Purdie (1998).
11
When the SOLO taxonomy is applied to short segments of code, the learner needs to have more than a
working solution; they need to show an understanding of the types of constructs that best implement the
solution, to utilise program structures that communicate the intent of their code to others, and produce code
that is easy to maintain.
During the BRACELet work on analysing code writing tasks, it was identified that the nature of the question
had an impact on the type of solutions that were possible. Some questions did not allow for much more than a
direct translation into the programming language while others allowed for greater interpretation. Other
questions could not be directly translated but needed to be interpreted and decomposed before a solution
could be developed (Lister et al. 2009). The question types used in this paper relate to the way that the
question describes the specifications of the problem and the work that the programmer may need to do to
arrive at a specification that can be translated into program code.
The following table summarises the code reading categories as reported in Whalley, et al. (2006), and the
code writing solution categories as reported in Lister et al. (2009). The proposed code writing question types
are described in the central column. Although the table below represents the categories as though there are
distinct boundaries, it should be recognised that what is represented is more of a continuum with the
boundaries between categories often being blurred and difficult to clearly identify. Although some questions
and answers are easy to categorise others appear to sit closer to the boundary portraying characteristics of
two categories.
Table 2: SOLO Categories for Reading, Code Writing, and Programming Projects
Phase SOLO Code Reading Code Writing Question Code Writing Solution
category Types
Qualitative Extended Requires knowledge of the Used constructs and

Abstract problem domain from concepts beyond those
outside the specification in required in the exercise to
[EA] order to decompose the provide an improved
specification to arrive at a solution
suitable solution
Relational Provides a summary of what Requires interpretation and Provides a valid well
the code does in terms of decomposition in order to structured program that
[R] the code’s purpose. arrive at a suitable solution. removes all redundancy and
Provides a summary of the Although the specification has a clear logical structure.
code that recognises provides all of the details for The specifications have
applicability of the code a solution, it provides few been integrated to form a
segment to a wider context clues that would hint at the logical whole
structure of the solution
Quantitative Multistructural A line by line description is Requires some Represents a translation

provided of all the code. interpretation in order to that is close to a direct
[M] Summarisation of individual arrive at a suitable solution. translation. The code may
statements may be included Some parts of the have been reordered to
specification may be directly make a valid solution
translatable into the solution
Unistructural Provides a description for Requires a direct translation Represents a direct

one portion of the code (i.e. of the specification into a translation of the
[U] describes the if statement) possible solution specifications. The code will
be in the sequence of the
specifications
Prestructural Substantially lacks Substantially lacks

knowledge of programming knowledge of programming
[P] constructs or is unrelated to constructs or is unrelated to
the question the question
For larger programming projects, even at the multistructural level, the learner may be showing some signs of
recognising duplication and the need to integrate code (Thompson 2007).
12
3. DEVELOPING A PROBLEM SET
Samples from the code writing problem set are used to illustrate the categories. They focus predominantly on
the use of functions.
3.1.1 Unistructural questions and answers

These exercises are designed to enable direct translation from the specification to the required solution. The
exercises provide the required information and there is generally only be one solution that is able to be
directly written from the provided information. The following examples represent very simple examples of this
type of exercise.
3.1.1.1 Problem: Calculating area of a circle

Write a method for calculating the area of a circle using the formula 3.14 * r2, where r is the radius of the
circle. Assume the radius is of type double.
Solution:
public double areaOfCircle(double radius) {
return 3.14 * radius * radius;
3.1.1.2 Discussion
For the learner to implement the solution to these unistructural questions, the primary knowledge requirement
is of how expressions are constructed in the language. Code writing questions of this type provide only minor
variations in solutions. For example, the learner may have used the Math.pow function rather than the
multiplication operator.
Any question that provides the learner with an algorithm that can be directly translated to a solution without
interpretation or decomposition is a unistructural question. The learner would have limited choices in how to
implement the algorithm. There is no scope for interpretation and decomposition. Such algorithms would not
depend on understanding existing code or data structures. Such exercises allow the learner to explore the
usage of language constructs free of the pressures of question interpretation.
More complex unistructural questions may allow for removal of duplication or reordering of the code to provide
cleaner solutions. For example: the calculation of a discount based on price ranges.
3.1.2 Multistructural questions

Multistructural exercises provide less information about the possible solutions. The unistructural questions
endeavoured to leave the student little doubt about the solution. These questions leave some room for
interpretation.
3.1.2.1 Problem: Calculating area of a ring

Write a method for calculating the area of a ring that is constructed of two concentric circles. Assume the radii
of both circles are of type double. (The learner may have already completed the direct translation exercise to
calculate the area of a circle.)
Solution:
public double areaOfRing(double outerRadius, double innerRadius) {
return areaOfCircle(outerRadius) – areaOfCircle(innerRadius);
13
3.1.2.2 Discussion
In calculating area of the ring, the learner has to determine that the solution is found by subtracting the area of
the inner circle from the area of the enclosing circle. This hasn’t been explicitly given in the problem
specification although hinted at by “two concentric circles.” Since the learner may have completed the
previous exercise, the learner should already have the required knowledge of how areas are calculated. The
solution illustrated is closer to a relational solution since it recognises the use of previous code that was used
to calculate the area of a circle. The learner may have produced a solution where the formula for the
calculating of the area of a circle is included within the method. This problem introduces the idea that
solutions can be developed by having a function call another function.
3.1.3 Relational questions

A relational question provides the learner with a description of the problem to be solved. The description
should be complete in the sense that it describes fully the requirement from the perspective of the problem
domain but provides only minor clues as to how the problem might be programmed.
3.1.4 Problem: Engine cylinder representation

An engine designer is seeking to represent the cylinder of an engine in their program. Design a class that will
enable the engineer to create a diagram of the cylinder including the length of the engine stroke, and to
calculate the surface area of the cylinder, the total volume of the cylinder and the volume displaced during the
movement of the piston within the cylinder. Assume that the head of the piston and cylinder are flat.
3.1.5 Discussion
This question describes the problem domain rather than the specific requirements of the class. The learner is
not told what values are required to initialise the object although some clues are provided. There are also
hints at the methods required but they are not clearly defined. The learner needs to decompose the question
to arrive at a class design before they can begin to write code.
3.1.6 Extended abstract questions

Exercises of this type provide the learner with the requirements but leaves out domain knowledge that the
learner must either know or locate in order to complete the exercise.
3.1.6.1 Problem: Tic Tac Toe

Write a program to play tic tac toe. The program should ensure that the computer can win or if that is not
possible the computer does not lose.
3.1.6.2 Discussion
This question assumes that the learner knows the game of tic tac toe and what the rules of play are. It also
assumes that the learner can determine a strategy for ensuring “that the computer can win or if that is not
possible the computer does not lose.” Neither assumption may be valid for the learner. As a consequence
they have to explore resources beyond the scope of the course to both understand the question and to
explore the game play options. Once they understand the game and strategies, they are in a position to begin
decomposing the problem and considering appropriate program structures.
4. DISCUSSION
The area problems are an example of where problem sequences can help the learner focus on the required
learning. By starting with a unistructural question, the learner’s focus is on selecting the appropriate language
construct required and variations in the use of that construct. A full set of exercises would cover all types of
expressions including the manipulation of strings. These exercises lay a foundation for talking about types in
terms of the related behaviours or operations that are applicable to each data type. Since these questions
minimise the issues that the learner needs to focus on, they provide a starting point for introducing language
constructs.
Unistructural questions can have variations in solutions. Such exercises give the learner the opportunity to
explore and evaluate alternative solutions. The same question could be asked with a slight revision to focus
the learner on specific solutions or specific constructs.
14
The use of multistructural questions begins to open further the possible variations in the formation of
expressions or the use of constructs. With multistructural questions, the learner is beginning to be challenged
about how to construct the solution (i.e. program design). Multistructural questions leave areas for
interpretation by the learner but are not fully open to interpretation and decomposition. Using questions that
not only increase the level of complexity but build on each other give opportunities for the learner to be
introduced to and to explore program design issues.
Relational questions focus on describing the problem rather than providing clues for the solution. The learner
has to interpret and decompose the problem in order to develop a solution. Since the learner is provided with
little direction, they have to decide how to structure the application and the programming constructs that need
to be used.
Extended abstract questions require the learner to do additional exploration in order to understand the
requirements. Such problems are problematic in a learning situation unless the learners have some familiarity
with the problem domain. In a computer science program, problems of this type might be drawn from other
courses that teach data structures or algorithms.
With respect to teaching, having levels of exercise and variations in solutions, provides opportunities to teach
problem solving and decomposition of problems, alongside the evaluation of solutions. Using a sequence of
exercises such as calculating the area extended to calculating volume enable the teaching of program design
issues.
The problem set if designed appropriately, opens up variations around the constructs being taught. There are
variations in the way that programs can be constructed. Variations between questions also open the space of
learning around program design and the structuring of programs.
5. CONCLUSION
Applying the SOLO taxonomy to the development of programming problem sets gives structure to the way
that the programming problem set is developed. Starting with unistructural questions that have direct
translation answers helps focus the learner’s attention on the programming construct. Extending to
unistructural questions that have variations in solutions both in terms of the SOLO levels and the
programming constructs, enables the learner to compare and evaluate solutions and to explore issues of
program design. Multistructural questions remove some of the design support leading the learner work
through program design issues. If they have progressed through the unistructural questions then they have
seen and evaluated design issues. At this point, the students should be have some understanding of how
programming programs can be decomposed. Relational questions put the learner in the position of having to
interpret the problem space, decompose the problem, and design the program. Although the question should
clearly describe the problem, the learner still has to understand the problem and how the problem can be
solved. Extended abstract questions take the learner into the realm of doing research to understand the
problem domain before they can decide on a solution.
From a teaching perspective, the problem set hierarchy allows for the gradual introduction of the programming
process alongside the introduction of coding constructs. By utilising some of the problem set examples in the
teaching process enables constructive alignment between the teaching process and the learning exercises.
6. ACKNOWLEDGEMENTS
The work reported in this paper draws on the research conducted as part of the BRACELet project especially
that related to the analysis of code writing solutions. Interactions with BRACELet participants laid the seed for
the development of the code writing question types.
7. REFERENCES
Biggs, J. B., & Collis, K. F. (1982). Evaluating the quality of learning: The SOLO taxonomy (Structure of the Observed
Learning Outcome). New York: Academic Press.
Clear, T., Whalley, J., Lister, R., Carbone, A., Hü, M., Sheard, J., et al. (2008). Reliably classifying novice programmer
exam responses using the SOLO taxonomy. Paper presented at the 21st Annual conference of the National Advisory
Committee on Computing Qualifications (NACCQ 2008). Retrieved
from http://naccq08.unitec.ac.nz/proceedings/papers/23.pdf
15
Hattie, J., & Purdie, N. (1998). The Solo model: Addressing fundamental measurement issues. In B. Dart & G. M. Boulton-
Lewis (Eds.), Teaching and learning in higher education (pp. 145-176). Camberwell, Vic: Australian Council of Educational
Research.
Lister, R., Clear, T., Simon, Bouvier, D., Carter, P., Eckerdal, A., et al. (2009). Naturally occurring data as research
instrument: Analyzing examination responses to study the novice programmer. Inroads - The SIGCSE Bulletin, 156-173.
Marton, F., Runesson, U., & Tsui, A. B. M. (2003). The space of learning. In F. Marton & A. B. M. Tsui (Eds.), Classroom
discourse and the space of learning (pp. 3-40). Mahwah, NJ; London: Lawrence Erlbaum Associates, Publishers.
Shuell, T. J. (1986). Cognitive conceptions of learning. Review of Educational Research, 56(4), 411-436.
Thompson, E. (2007). Holistic assessment criteria - Applying SOLO to programming projects. Paper presented at the
Proceedings of the Ninth Australasian Computing Education Conference (ACE2007). Retrieved
from http://crpit.com/confpapers/CRPITV66Thompson.pdf
Whalley, J., Lister, R., Thompson, E., Clear, T., Robbins, P., Kumar, A., et al. (2006, 16 - 19 January). An Australasian
study of reading and comprehension skills in novice programmers, using the Bloom and SOLO taxonomies. Paper
presented at the Eighth Australasian Computing Education Conference (ACE2006), Hobart, Tasmania, Australia.
16
A MAD APPROACH TO JAVA CONCEPTS
Simon Sharpe Mark Dixon Gill Harrison
Leeds Metropolitan University Leeds Metropolitan University Leeds Metropolitan University
Headingley Campus Headingley Campus Headingley Campus
Leeds LS6 3QS Leeds LS6 3QS Leeds LS6 3QS
s.sharpe@leedsmet.ac.uk m.dixon@leedsmet.ac.uk g.harrison@leedsmet.ac.uk
ABSTRACT
The goal of engaging and enthusing students in higher-level computing concepts can remain elusive for
Higher Education institutions. This paper discusses the development of the Mobile Application Deployment
(MAD) Module that uses an approach to the teaching of advanced Java programming concepts that seeks to
achieve this goal by encouraging students to develop mobile phone applications using Java Micro Edition.
Keywords
engaging Java mobile phone game JavaME
1. INTRODUCTION
Higher Education Institutions generally share the goal of trying to engage and enthuse students in their
studies, which can sometimes be difficult. A widening participation agenda can result in classes containing
students with a wide range of ability and experience, which can make this goal even harder to achieve.
Emerging from a restructuring of the curriculum at Leeds Metropolitan University came the requirement for a
new second year module for all students on computing related courses. Formerly, students had chosen
between specialist modules on such topics as databases, software development, user interface design,
networks and web development, but the decision to harmonise their experience at second level meant that a
new module was needed. This consolidation of courses meant that students with different interests and areas
of expertise within the Computing discipline would be studying the same module together. This diversity
meant that it was imperative that the module be designed to be as appealing as possible to students of all
interests.
A decision was taken that the new module, now common to all Computing students, would retain aspects of
the specialist areas in that it would:
 contain programming, possibly building on the students’ previous programming experience
 require applications to be built, that would need the exercising of user interface design skills
 offer opportunities to store and manipulate data within such applications
 allow for the possibility of adding network connectivity in order to web-enable applications
The aim was thus to use a multidisciplinary approach, bringing together aspects from the computing discipline
in a way that was relevant to today’s demand for developers. Experience has shown that many students
become easily discouraged when they are learning programming, and therefore this new module should also
enable students, whose first choice of specialism would not be programming, to be able to understand
concepts that they would normally find challenging.
To accommodate all the above, and with the overriding need for an attractive, engaging module, the Mobile
Application Deployment module was created, and delivered for the first time in 2009/10.
2. DEVELOPMENT OF MAD MODULE

2.1 Influences on MAD’s design
There were a number of drivers that influenced the design of the new module.
 As discussed above, the need to engage and motivate students by providing a learning experience that
they would find enjoyable and relevant
17
 The explosive growth of different kinds of technology, especially mobile technology (Boretos, 2007), and
students’ easy familiarity with mobile devices such as phones and music players
 The growing demand in employment for developers of applications for mobile devices, and the desirability
for graduates to have transferable skills in the area of mobile devices (Chickowski, 2010)
2.2 Design Decisions

Taking into consideration the widespread familiarity of current students with all kinds of applications (for social
networking, games, etc.) and the increasing availability of these on mobile devices, it was decided that the
development of mobile phone applications was an area that students would find enjoyable, motivating and
relevant. For many students the mobile phone has replaced the desktop as their main computing device.
Therefore, developing mobile applications would provide a way to connect computing to their perception of
real-world technology. There is evidence of the success of this approach from a number of educators with
aims similar to ours. For example, the experience of integrating mobile devices into the Computer Science
curriculum at the University of Guelph in Canada has led Mahmoud (2008) to say that “Teaching computer
science and programming in the context of mobile applications provides a motivating framework for students
and inspires them to excel due to the practical experience they gain allowing them to develop applications for
their own mobile devices”. A similar approach is taken by Kulik (2007) in a post-graduate course in
Melbourne. Scharff et al (2009) have taught mobile application development for social changes in Senegal,
where “students develop and deploy applications for people of their local communities”.
Games development for motivational purposes has become increasingly popular, and in the context of mobile
applications Kurkovsky (2009), for example, at Central Connecticut State University considers games as
highly suitable mobile applications for students to develop. He comments that “Compared to traditional game
development, programming mobile games is less complex, which enables students with limited programming
experience to create playable mobile games within the scope of a single course”. The module team thus
decided that developing a simple game would provide an appropriate context for the module. This focus
provides sufficient scope for design and specification, a graded progression of programming difficulty and
sufficient possibilities for extension into the use of databases and networking.
2.3 Choice of programming language and development environment

A number of factors influenced the choice of programming language and development environment:
 students’ familiarity with language (they had studied basic Java programming)
 availability of free software for students’ own machines
 suitability for teaching programming concepts
 range of mobile devices that would support the students’ applications
 range of supporting tools within development environment
 staff familiarity and skills (for practical reasons, as time was short)
The two candidates that appeared most likely were Flash Lite and Java ME. Flash Lite was a strong
contender, but would require significant lead time to introduce students to the new language. Java ME
(formerly J2ME) is a cut down version of Java created to support the development of applications for
resource-constrained devices such as mobile phones. The final decision, in the light of these factors, was to
use Java ME.
2.4 Java ME and Netbeans

Java ME provides one of the most ubiquitous application platforms for mobile devices. It offers a collection of
technologies that can be configured for a range of devices such as mobile phones, PDAs and set-top boxes.
The majority of mobile phones support this technology, including most major brands such as Nokia, Samsung
and Sony Ericsson (Java User Group, 2010), but not the Apple iPhone. As a restricted version, Java ME
provides a simpler set of APIs for connectivity, gaming and in particular GUIs. However it still allows for
teaching the higher level Java programming concepts such as listeners, abstract classes, interfaces,
overloading, double buffering, threading, etc.
There are two base configurations:
 The “Connected Limited Device Configuration (CLDC)” for small mobile devices
 The “Connected Device Configuration (CDC)” for more capable devices (like set-top boxes).
18
“On top of the different configurations Java ME platform also specifies a number of profiles defining a set of
higher-level APIs that further define the application. A widely adopted example is to combine the CLDC with
the Mobile Information Device Profile (MIDP) to provide a complete Java application environment for mobile
phones” (Oracle, 2010).
Mobile applications are developed via MIDlets which use MIDP. A MIDlet extends the abstract MIDlet class
which controls the lifecycle of the application. MIDlets are controlled by the Application Management Software
on the mobile device
In addition, Java ME would allow the use of sophisticated development environments such as Netbeans or
Eclipse. Both of these IDEs provide useful emulator tools (see Figure 1) to support the development of
applications on a desktop that could subsequently be deployed to mobile phones. Netbeans was the chosen
environment, as it also provided a Game Designer tool, which facilitated much of the graphical work required
for games. The platform was readily available for students who wished to download it to their own machines.
Figure 1
3. TEACHING AND LEARNING ASPECTS OF THE MODULE

The module was delivered over a semester (twelve weeks) using weekly lectures of one hour and practical
supervised laboratory sessions of one and a half hours. Students were also expected to spend time working
independently. Each lecture introduced a set of concepts that were then reinforced during the practical
session. Formative practical exercises were presented as for example in Figure 2.
Figure 2
The course structure is summarised in the table below. Groups of topics were introduced as shown over a
ten week period.
Introduction to Java ME, CLDC, MIDP and Midlets
High Level User Interface (Forms and UI components, commands, listeners).
Low level UI – 2D Graphics, Keypress, MIDP Canvas
Timers Tasks and Threads
Opportunity to meet module “threshold requirements”
Game API
Game API + Extension work
19
The assessment was divided into two main sections. The first section involved the development of interactive
forms that could be used for game configuration or initiation. Successful completion of the first section meant
that students had reached the Threshold requirements to meet the learning outcomes for the module,
meaning they could pass the module at this stage. The second section required students to design and build
a game-style application suitable for deployment on a mobile device. They could choose to create the scenes
and sprites using the Game Designer tool that Netbeans provides. They were free to incorporate more
advanced features in order to obtain a higher grade.
4. EVALUATION AND ASSESSMENT

4.1 Student Engagement and Achievement
Attendance and engagement have been much better than in previous modules that have introduced similar
concepts. All of the 95 students who were able to engage with the module passed at the first attempt. About
20% chose to use the Threshold target. However, the majority continued to actively participate with the higher
level concepts producing some outstanding applications, which demonstrated considerable effort and ability.
The module team felt that the experience has left all students better prepared for their final year studies.
4.2 Student feedback

4.2.1 Standard Module Evaluation Questionnaire
The Faculty provide a standard module evaluation questionnaire which students are encouraged to complete.
The questions relate to their assessment of the quality of teaching, learning materials and assessment,
grading from 1 (excellent) to 5 (poor). Nearly all the students who completed the module graded the
questions at either 1 or 2. In addition there are two open ended question at the end where students could
comment on what they felt was good about the module and what could be improved. Below is a sample of the
comments received.
Programming is not a subject I particularly enjoy but this module was taught in a way which allowed me to
both enjoy and understand what I was doing. I have gained more understanding from this form of
programming than I have from any other. Great module and well taught.
Great module - loved it.
Fantastic module, if this was an option in year 3 I would have taken it
It was a really good class. I have learnt a lot from it and really enjoyed it.
module was really enjoyable and developed my understanding of java very good overall! thanks
Very enjoyable module!
Really enjoyed the module. Not a big fan of programming but got me sucked in and couldnt stop.
4.2.2 Additional Questionnaire

In addition, a separate questionnaire was given to a self-selecting sample of 20 students to assess how they
felt their understanding of a range of Java concepts had improved on a scale of 0 to 5. The aggregated gains
are shown in the table below. The numbers represent a relative range from 0 to 100 where a score of 0 would
indicate no improvement for any of the students and 100 would indicate that the concept was fully understood.
Improved Understanding Gain
Classes, Objects and Constructors 37
Inheritance 48
Interfaces 49
Commands and Listeners 55
Exception Handling 31
Forms and Form UI items 50
Timers and Timertasks 51
Multithreading 38
2D Graphics & Images 55
20
All scores showed gain in understanding (as one hopes they would). As expected, new topics such as
Graphics, Commands / Listeners naturally showed most improved understanding, but it was pleasing to note
that fundamental OO concepts such as classes and objects had been reinforced for many students.
There was also an open ended question asking if studying the module had influenced their choice of future
studies in any way. Many of the comments suggested that the views had not changed, and that many
previously held opinions and views on Java and programming had been reconfirmed. Example responses
included:
“I'm no good at programming”, “Programming not my niche - won't be studying further”, “Too difficult
and not a subject that interests me much”
However, for some, the module had changed their view of programming and perhaps choice of future studies.
Example responses included:
“Module has motivated me to do more programming”, “Module enjoyable liked to have progressed
more”, “Improved my java skills - might consider being a programmer”, “I would like to learn more
about this subject”, “I've decided to pursue software development”, “Would like to pursue this in
future”
5. CONCLUSIONS AND FURTHER WORK

There is a general acceptance of the success of using gaming applications as a teaching tool within
Computing. This was combined with developing applications for a mobile device which students found highly
motivating and relevant to their lives. The Java ME platform and Netbeans IDE provides a platform and
environment to make this achievable for students with a wide range of abilities and interests in different
computing disciplines.
The module was run for the first time in the academic year 2009-10. Engagement, achievement and feedback
were very pleasing.
It is planned to expand and develop the curriculum and offer the module to higher level courses. Students will
also be encouraged to use this as a starting point for final year projects.
6. REFERENCES
Boretos, G. (2007) The future of the mobile phone business. In Technological Forecasting and Social
Change, Volume 74, Issue 3 March 2007 pp 331-340.
Chickowski, E. (2010) Messaging and Collaboration: Mobile Software Development Tech Skills to Get the
Job. http://www.channelinsider.com/c/a/Messaging-and-Collaboration/Mobile-Software-Development-Tech-
Skills-to-Get-the-Job-498559/ (accessed 18th May 2010)
Java User Group (2010) List of the MIDP Java phones having downloaded Tastephone. http://www.club-
java.com/TastePhone/J2ME/MIDP_mobile.jsp (accessed 21st May 2010)
Kulik, L. (2007) Mobile Computing Systems Programming: A Graduate Distributed Computing Course. In
IEEE Distributed Systems Online, Volume 8 No. 5, art. no. 0704-o5004.
Kurkovsky, S. (2009) Engaging Students through Mobile Game Development. In: SIGCSE 2009,
Chattanooga, Tennessee, USA, March 3-7 2009.
Mahmoud,Q.H. (2008) Integrating Mobile Devices into the Computer Science Curriculum, 38th ASEE/IEEE
Frontiers in Education Conference, New York Oct 22-25 2008 http://www.fie-
conference.org/fie2008/papers/1895.pdf (accessed 4th May 2010)
Oracle (2010) Sun Developer Network – Java ME Technology

http://java.sun.com/javame/technology/index.jsp (accessed 4th May 2010)
Scharff, C., Wasilewska, A., Wong, J., Bousso, M., Ndiaye, I., and Sarr, C. (2009) A Model for Teaching
Mobile Application Development for Social Changes: Implementation and Lessons Learned in Senegal. In:
Proceedings of the International Multiconference on Computer Science and Information Technology,
Mragowo, Poland, October 12-14 2009 pp. 383-389.
21
SUPPORTING THE LEARNING OF OBJECT ORIENTED DESIGN
WITH A LIGHTWEIGHT WEB-BASED COLLABORATIVE MODELLING
ENVIRONMENT
Quan Dang Tingkai Wang

Faculty of Computing Faculty of Computing
London Metropolitan University London Metropolitan University
q.dang@londonmet.ac.uk t.wang@londonmet.ac.uk
ABSTRACT
Learning object-oriented (OO) design is much more than just learning to draw UML diagrams. It is about
learning the steps and deliverables of an OO modelling process, whereby gradual transitions from software
functional requirements to implementable class design can be achieved. This paper introduces objFusion, - a
web-based environment which was developed to be a lightweight, yet capable of supporting students learning
OO design with essential OO design heuristics and of maintaining the semantic links between models of
different steps in an OO design process.
Keywords
Computer science education, Object oriented design, Object oriented programming, CASE tool, E-learning.
1. INTRODUCTION
It has been observed from our experience of teaching introductory OO design and programming that students
completed an OO course did not always get a grip on the essence of the role of and what is the design activity
in an OO software development process, even though they might be able to use OO modelling notations (e.g.
UML [2]) and to write code in an OO programming language. In particular, students find it difficult to
semantically connect OO models created at different stages during an OO process in order to smoothly arrive
at an implementable class design. It is often that they are able to use sub-sets of OO techniques/notations here
and there to create various OO artefacts, but when it comes to the implementation their software classes would
usually come from nowhere, not being derived from the class design resulted from earlier stages of their design
process. In order to rectify this, our pedagogic approach is to help students to establish missing connections
between the OO models by placing an emphasis on maintaining semantic links between related models across
the stages of an OO process. In our teaching, we use of a set of explicitly written OO process- and product-
related heuristics, and model templates, the conformance to which is enforced with student peer review to
validate and to verify modelling artefacts (e.g. use cases, class diagrams, CRC cards [1] and sequence
diagrams).
In order to support our teaching approach we have looked for a suitable software tool with the following
requirements. Firstly, the tool should cover essential steps of an OO process and have a clear set of
deliverables for each of the steps together with some heuristics on how these steps can be carried out.
Secondly, the tool should allow students to create use case diagrams, class diagrams and sequence diagrams.
We would also like to have support for CRC cards because we have found that building CRC cards is a very
effective way for students to understand how software functionality, as modelled in use cases, can be assigned
to software classes. Thirdly, we’d like to have a tool which would enforce model traceability across the
modelling process, i.e. a model built in a subsequent step must be derived from related models built from the
precedent steps. For example, the objects and classes appear in a sequence diagram must be previously
defined in the class diagram. Fourthly, the tool must be simple, not bloated with abundant features, easy to use
and does not require much learning for an OO novice to use it. Finally, we also like that the tool can support
students working in group to collaboratively build their group’s model.
There are a plethora of CASE (Computer Aided Software Engineering) tools which support OO design (see [9]
for a list of these tools). Most of the full-blown OO CASE tools, e.g. IBM Rational Modeler [5] and Sparx
Enterprise Architect [6], provide comprehensive support. They cover the steps of a modelling cycle, together
with deliverables of each of the steps. It is common that these CASE tools also enforce model traceability to
some extent. Some tools support collaborative modelling e.g. Creately in [3]. However, these heavyweight tools
are too complex for the OO novice, and require stiff learning curves, which would usually be too much to
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that
copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page.
To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission.
© 2010 Higher Education Academy 22
be included in the context of an introductory course. There also exit numerous lightweight UML tools, including
web-based tools for drawing UML models, e.g. UMLet [7] and Violet [8]. With regard to our requirements these
tools are too limited, being just diagramming tools. These tools are not model-aware, which know nothing about
the modelling heuristics and model semantics. Besides, none of the lightweight tools provides support for
collaborative modelling. Further, in terms of the feasibility to use these tools for class teaching and learning,
heavyweight tools require local installation with a lot of disk space. For example, the IBM Rational Modeler
installer alone is of 200MB. As a consequence students can only use the software in certain labs where it is
installed. They cannot use it while working with a general build PC such as one in a library, and they wouldn’t
be willing to install the software on their own PC.
As a result, our evaluation has led to the development of our own tool, which provides the functionality we want
to have, at the same time is easy to use and lightweight.
2. A BIRD’S-EYE VIEW OF OBJFUSION

The aim of the objFusion development is to have a lightweight tool which is OO model-aware to support the OO
novice to learn OO design. An emphasis is given to provision of support for modelling rules and model
semantics rather than drawing diagrams of the models. Specifically, objFusion supports essential steps in an
OO process in order to build essential OO models, viz. use cases, class diagrams, CRC cards and sequence
diagrams. objFusion supports maintaining the model traceability across the modelling process and team
collaboration, which help students verify their team’s model and learn from each other. Being a web application
objFusion requires no installation on the client computer to run, provided a modern web browser, such as
Internet Explorer or Firefox, is present on these computers. We’d like to think of objFusion as a specialist virtual
learning environment to support teaching and learning of OO design, rather than a software tool in its
‘traditional’ meaning.
2.1 Main Features

In terms of functionality, the objFusion environment provides support for the following (a more detailed
description of the features will be given in Section 3).
1. An OO modelling process from software functional requirements to implementable class design. objFusion
incorporates an OO process which enforces students to follow a prescribed sequence of modelling stages,
and to maintain the model traceability of the deliverables across the stages.
2. Team collaboration i.e. to provide a collaborative environment where students of a project team can
collaboratively build OO models, exchange and log their ideas and reflective notes.
3. Provision of data repository of modelling artefacts and discussion threads created by students during the
course of an OO process. The logged items in the repository are to be used for the purpose of students’
own reflection and formative/summative assessment by tutors.
In terms of the usability, objFusion has been designed for ease of use and with an adequate level of security to
support web-based multi-user collaborative work.
1. The fact that objFusion is a web application allows tutors and students to be able to use it from virtually
anywhere they happen to have access to a computer with an Internet access.
2. objFusion is user friendly and requires no training to use it. This is a crucial factor which encourages tutors
and students to use the system in and outside their classes.
3. objFusion is secured with password controlled access. Tutors and students are organised in classes and
project teams. Each user can only access data and working within the scope of their classes (for the tutor)
and their projects (for the students).
2.2 Organisation of Functions and Multi-tabbed User-interface

The objFusion functions are grouped around user roles and presented in a browser-based multi-tabbed user
interface. There are three different user roles defined in the objFusion environment. These are administrator,
tutor and student. Each of the roles determines access to a specific group of functions in the environment. A
role is assigned to a new user at registration to the system. A person can at the same time register as a
student as well as a tutor in objFusion with separate usernames, one as a tutor and the other as a student. The
functions are organised around the roles as follows.
1. Student can perform
1.1. All of the functions related to the OO modelling for the project they are assigned to as a member of the
project’s team.
1.2. Online chat with the members of the project’s team and view the chat log.
23
2. Tutor can perform
2.1. All the functions of the student.
2.2. Managing the projects of their classes (courses) i.e. add/delete projects for their classes.
2.3. Managing project teams, i.e. add/delete students to/from project teams.
3. Administrator can perform
3.1. All the functions of the tutor (and of the student)
3.2. Managing the system users, i.e. register and unregister users
3.3. Managing the classes (courses) with the information of the class’ tutorship.
The objFusion’s functions, data input

and generated output are all
presented to the end user in the form
of web pages, which are displayed in
a web browser window. Related
functions are organised in tabs. A tab
contains a set of functions related to a
role or a modelling activity as showed
in Figure 1. For example, when a user
with the student role successfully
logged in objFusion will present the
Student tab with the functions which
are available to the student role. It will Figure 1: Support for OO modeling activities in the objFusion environment
also display the project the student
has been assigned to. The student may not access the functions of the other roles of the system.
3. SUPPORT FOR LEARNING AND TEACHING OF OO DESIGN

This section provides a more detailed description of the objFusion functions and how they support various
activities in an OO design process, and related learning and teaching tasks.
3.1 Support for OO modelling

objFusion supports essential activities of OO design, namely use case modelling, class modelling, and use
case realisation via use case analysis. The use case analysis is to elicit system operations, class
responsibilities, and object interactions, which are then further modeled in sequence diagrams. Messages from
the sequence diagrams will then be fed back to the software class design in terms of class attributes and
methods. Each tab in Figure 1 above is designed to support a step of OO design as detailed below.
1. Use case modelling
1.1. Use case diagrams, with comments on individual version of use case diagram by project team
members
1.2. Use case actors and relationships to those use cases an actor performs
1.3. Use case descriptions in terms of sequences of steps of interactions between the actor and the
system.
2. Class modelling
2.1. Class diagrams with classes and their relationships such as inheritance, association (aggregation and
composition) 1-to-many, 1-to-1 and many-to-many, dependency (uses), etc.
2.2. Individual class, its attributes and operations, whose data types and declarations scopes are also
modelled.
3. Use case analysis
3.1. Use case step analysis to elicit (application domain–bound) operations that the systems must carry
out to fulfill the business goal of the use case – in each use case step.
3.2. System responsibilities modelling with CRC cards: this is based on previous results from the use case
analysis. System responsibilities are distributed to previously identified classes. For a responsibility
there must be only one responsible class. Although, there may well be other classes involved in the
fulfillment of the responsibility. These classes are collaborator classes. A new class can be added to
the exiting model when there is no class found from the existing ones to take a responsibility.
4. Object interaction modelling (i.e. sequence diagrams)
4.1. Object interaction modelling with sequence diagrams: objFusion imposes that a sequence diagram
must have a reference use case and the objects in the sequence diagram must be those of existing
classes (i.e. previously identified classes).
4.2. Messages of the created sequence diagrams will be fed back to class modelling to flesh out previously
indentified classes with corresponding methods and relevant data (i.e. attributes). The predefined
24
class’s responsibilities are used as a basis when making a decision on which message should go to
which class. A new class can be added to the exiting model when there is no class found from the
existing ones to take a responsibility.
It is worth noting that in objFusion
the model traceability (i.e. semantic
links between models) is achieved
through enforcing model
constraints, in particular as in
functions no.3 and no.4 above.
Figure 2 illustrates that the model
traceability feature is built into the
objFusion system. Specifically, the
responsible class and collaborator
classes can only be chosen from
the existing (previously identified)
classes. The same constraints are
imposed on the options of the
sender and receiver classes of
messages when creating a
Figure 2: A screen shot of the Use case analysis – Add new Responsibility
sequence diagram.
3.2 Support for team collaboration
Students working collaboratively in teams will help each other to build better OO models for their team project
by complementing each other to achieve better understanding and skills of OO design. In particular they will
verify models created by other team members, maintaining the model traceability across their OO model
building process. objFusion supports collaborative work of students as in the following.
1. Allow tutors managing their classes, projects and project teams. Individual students can be added to or
removed from a project team.
2. Members of the same team can work on the project’s models concurrently in real-time.
3. In a project any member of the project team can view, modify the project’s model artefacts and create new
model artefacts.
4. Previous versions of the same model
artefact are maintained, rather than
overwritten and deleted. Tracks of
who created what are maintained and
can be seen by the team members
and tutors.
5. In a project any member of the project
team can make comments and
reflective notes on any model
artefacts, which other members of the
team can view.
6. Team members can chat online in
real-time with other team members
and tutors, while working in the
objFusion environment.
7. Online chat messages are logged by
the system for later review for student Figure 3: objFusion keeps tracks of versions of class diagram
reflection purpose, and for tutors’
instructional and assessment purposes.
Figure 3 illustrates that objFusion keeps tracks of all versions of “Class diagram” using time stamps, and allows
team members to edit existing version then to save it as new one, keeping the old intact.
3.3 Provision of data repository
In order to support the system administrator and tutors to manage their classes, students and projects
objFusion allows storing and retrieval of data as detailed below. These data are organised around the related
tutors, classes and projects. Within a project the data are structured along the created models, viz. use cases,
class diagrams, system responsibilities, CRC cards, and sequence diagrams.
1. User registration details, their usernames and passwords to log-in to the system.
25
2. Classes (aka courses) and class tutors.
3. Projects in a class
4. Project team membership, i.e. students of project teams.
5. All relevant data of the model artefacts created by the project teams and their tutors, including comments
and reflective notes on models.
6. All the online chat messages created by the users.
It is worth noting that these data are not only about the OO models of created by project teams. There are also
stored data of students’ comments and reflective notes, which indicate individual student contributions to their
team work and progress they make over the course of a project. As a result these data could be used not only
for student learning purposes, but also for tutor’s formative assessment of the student work.
4. TESTING AND EVALUATION

An implemented version of the objFusion system has been tested by the developer team, and by students in a
user acceptance test. The implemented system has served well as a proof of concept for the idea to build a
lightweight and collaborative modelling environment to support teaching and learning of OO design as
presented in [4]. Positive feedback received from our students via a survey shows that objFusion has potential
to enhance their understanding, appreciation and skills of OO design. In particular, nearly 100% respondents of
the survey agreed with (i) “the fact that the software is web-based and requires no local installation encourages
its usage”, (ii) “Collaborative modelling is really useful in keeping tracks on connections between OO various
models”, and (iii) “Enforcing and keeping tracks on the connections between the models during the design
process is instrumental in order to benefit from the OO design activity”. Last but not least, a presentation of the
pedagogic approach, on the basis of which objFusion was designed at the HEA-ICS 10th Annual Conference in
Kent [4] have attracted positive appreciation and encouraging interest from colleagues, who teach relevant
modules in OO programming, and analysis & design.
5. CONCLUSIONS AND FURTHER DEVELOPMENT

This paper has presented objFusion - a lightweight web-based collaborative environment to support an
approach to teaching introductory OO design. The approach is based on an emphasis on maintaining the
model traceability [4]. An evaluation of an early version of the system shows encouraging interest from
colleagues and positive feedback from students. Currently, we are working on the development towards an
enhanced version of the system. In terms of functionality, we are working upon the work to export the final
class models to class skeleton code in an OO programming language such as Java and C#. As to the system
usability, we are working on a better graphical user interface for the system, particularly, one with clickable
UML diagrams.
6. ACKNOWLEDGEMENT
Development of objFusion is supported with a grant from the Higher Education Academy subject network for
Information and Computer Sciences’ Development Fund. Their support is gratefully acknowledged.
7. REFERENCES
[1] Beck, K. and Cunningham, W.A Laboratory For Teaching Object-Oriented Thinking; the OOPSLA'89
Conference Proceedings, October 1-6, 1989, New Orleans, Louisiana.
[2] Booch, G. et al., The Unified Modeling Language User Guide , 2/E, Addison Wesley, 2005.
[3] Creately - Online Diagramming, http://creately.com/
[4] Dang, Q. and Wang, T., 2009, Enforcing Model Traceability for Effective Learning of Object-Oriented
Design, HEA ICS Annual Conference, 25 -27th August 2009, University of Kent at Canterbury (view full-
text in PDF), http://www.ics.heacademy.ac.uk/events/10th-annual-conf/programme_day2.php
[5] IBM Rational Modeler http://www.ibm.com/developerworks/downloads/r/modeler/
[6] Sparx Systems Enterprise Architect, http://www.sparxsystems.com/
[7] UMLet - Free UML Tool for Fast UML Diagrams, http://www.umlet.com/
[8] Violet - a UML editor, http://www.horstmann.com/violet/
[9] Wikipedia List of UML tools, http://en.wikipedia.org/wiki/List_of_UML_tools
26
Towards an Interactive Multimedia-based Environment
for Teaching Academic Writing
Alma Whitfield, Obinna Anya, Stewart Blakeway Nabil Sultan
Snr Lecturer, Research Assistant Lecturer Snr Lecturer
whitfia@hope.ac.uk
Computer Science, Liverpool Hope University, Liverpool L16 9JD,
ABSTRACT
This paper describes an ongoing research work to investigate novel ways of organising and storing resources
for academic writing and referencing in order to address the accessibility needs of students at the
undergraduate, diploma and pre-university levels. So far, most research in e-learning has focused on
resources for the teaching of various courses of study often targeted at distance and adult learning, while the
area of academic writing has received comparatively very little attention in spite of its important role as the
bedrock of academic creativity across disciplines. We report the results of the initial stage of our experiment in
applying Second Life as an interactive environment for teaching academic writing to students at the
undergraduate, diploma and pre-university levels. Our approach is to codify existing academic and reference
materials in the area of computer science using technologies, such as online databases and electronic
archives, in a way that will enable interactive access to them by students learning academic writing through
such entertaining and social media as Second Life. We suggest that exposing students to academic writing
using such emerging innovative and interactive multimedia technologies will lead to cross-disciplinary
improvement in learning among young students.
Keywords
Academic writing, e-learning environment, social network.
1. INTRODUCTION
A problem has emerged in my teaching during the last few years that requires attention. As
each semester progresses, a greater proportion of my students is reading less, referencing less
and writing with less clarity and boldness. Tara Brabazon [1]
In the book The University of Google: Education in the (Post) Information Age, Brabazon,[1] described a
concern that is reflective of the true state of academic writing among today’s students, particularly at the
undergraduate, and pre-university levels [1]. In January 2008 an article published in the Times Higher
Educational Supplement highlights the concerns held by many academics that students leave pre-university
schools poorly equipped with basic skills of academic writing, and consequently well under-prepared for
University level study [8]. The paper referred to current students as “The league-table generation” and
explored the divide between schools and universities, pointing to government policy in secondary education
as a major cause of the difficulties students face during their transition into University [2]. Increasing pressure
on students, from parents and society at large, emphasizes “high score” in exams often at the expense of
quality learning.
While students approach University level study ill-prepared, studies show that students spend a great deal of
their time in so many other non-academic activities that offer them entertainment and exposure to technology.
One case that buttresses this is the rising use of the Web and other devices such as iPod among students.
Advances in information and communications technology have led to an unprecedented increase in Web-
based information, online databases and e-resources plus a variety of electronic gadgets to access this
information. The use of these emerging technological tools often occurs in engaging, rich and interactive
multimedia environments that could potentially be directed to support learning.
While a lot of research work has investigated and designed multimedia-based tutors and online educational
environments, the area of academic writing and referencing has received comparatively very little attention in
spite of its important role as the bedrock of creativity across disciplines. Many students are hardly introduced
to the discipline of academic writing and referencing especially in the early stages of their education. Most
existing sources for teaching academic referencing are not interactive and tend to follow traditional
approaches, which many students at the pre-university, diploma and undergraduate levels do not find
copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first
page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission.
27
entertaining. As a result, the field has remained somewhat esoteric even to postgraduate students and
aspiring academics.
This research work proposes a framework for a multimedia-based environment for teaching academic writing
and referencing. Our hypothesis is that students exposed to academic writing and referencing using emerging
innovative technologies, such as interactive multimedia technologies, will lead to cross-disciplinary
improvement in learning among young students. A novel contribution of the research is to develop a new
model of integrating interactive multimedia resources into existing online resources for academic writing and
referencing in order to introduce students to the subject in a friendly and fun-driven manner and without losing
the pedagogical focus of the learning environment. We seek to focus, at this initial stage, on a methodology
for codifying existing academic and reference materials in the area of computer science using online
databases, electronic archives and multimedia systems.
2. LITERATURE REVIEW
A number of studies, such as [2] observe that defining academic writing is problematic. A preliminary finding of
our empirical study of understanding what academic writing is among students at the pre-university level
resonates with this view. Most students in our questionnaire responses describe academic writing as
"uninteresting" or "a form of writing for academics." As a result of this lack of appropriate understanding, most
works resort to defining academic writing by referring to the salient features of the term. After a survey of a
range of academic writing across and within disciplines [2] identifies a number of features of academic writing,
namely being analytical, critical and following a structured and clear argument supported by evidence and
logical reason as well as having a smooth and clear transition between ideas and arguments and careful and
complete referencing of the work of other writers and researchers.
The concept and development of academic writing has its origin in the Western historical-cultural tradition as
described by [6]. The authors cited by [2] place the genesis of academic writing in the seventeenth century,
within the “conduit model of language” (p.62) whereby knowledge and meaning are created in the mind of the
author and then transferred to the minds of others by means of language. This language conduit was required
to be plain, clear and distinct, reflective of a plain, clear and distinct thought process. Thus, structure,
argument and definition in academic writing were given priority. The conduit model of language, however,
detaches language from the process of knowledge and meaning creation [2]. Lillis and Turner [6], cited by [2],
describe a contrasting understanding of academic language in which “language and the construction of
knowledge go together.” (p.64). Under this model of language use, “the qualities of tightness of structure,
logical coherence and off-setting criticism by dealing with counter-arguments within the text” (pp. 64-65)
served to give language a pre-eminence in providing universality and persuasion to academic discourse. The
elements of both of these language models within academic discourse continue to impact students’ attitude
towards academic writing. What is needed is more research to enable sufficient understanding of the meaning
of academic writing both in principle and in practice, particularly among students at the undergraduate and
pre-university levels.
Over the last few years, a lot of research and design efforts have gone into the development of technology-
rich interactive environments to support e-leaning. Recent research efforts in this regard appear in the works
of [4] and [7]. As a result, a number of standards, certifications, techniques, design and development
protocols, and prototypes have been developed. However, most these have tended to focus on areas of
learning other than academic writing. Our research is therefore aimed at filling this gap by developing a
framework for using modern interactive technologies to teach academic writing and ultimately contribute to
promoting, among young students, the age-long tradition of learning and knowledge entrenched in the culture
of academic writing
3. RESEARCH APPROACH
This research work will involve:
 Review of relevant projects and literature
 Ethnographic (field) studies with quantitative (statistical) analysis of research data
 Conceptual design of the proposed framework
 Design of the online database
 System prototyping and implementation
 Validation of research results
 Discussion (for further studies)
We contend that this mixed methodology is appropriate in addressing the aims set out for this research study.
The research will adopt a testing out strategy, and a mixture of qualitative and quantitative methods will be
used. We briefly describe these as follows:
28
Activity 1: Testing Out
A sample of online resources and websites for teaching academic writing and referencing will be surveyed in
order to explore existing approaches to teaching academic writing to students at our target levels. The focus
will be on approaches used, database organisation methods, target students, level of application of state of
the art technologies, etc. A number of multimedia resources and data, as well as emerging devices such as
iPod, iPhone, iPad will be analysed to ascertain their suitability in supporting the learning of academic writing
and referencing. The process of triangulation will be applied by literature survey with the aim of validating the
testing out approach used.
This activity is on-going, and will continue until the end of June 2010. However, the literature review will
normally continue until the end of the project in order to incorporate the latest academic and research findings
and state of the art technologies into the research work.
Activity 2: Data Collection
First semi-structured questionnaires will be used to collect initial data from students. The questionnaires are
appropriate as the collected information will be fairly superficial and tailored towards eliciting opinions from
these students. The focus of the questionnaire will be to ascertain the degree of referencing, which students
incorporate in their academic writing at the undergraduate, diploma and pre-university levels as well as the
level of interactivity that the students will likely appreciate within a learning environment. The use of
questionnaires will be quickly followed up with short semi-formal interviews with members of the management
of a sample of schools and colleges in order to solicit their opinions on the teaching of academic writing to
students at the different target levels.
Semi-formal interviews will be held with a sample of teachers drawn from different schools. The aim of the
interviews will be to elicit their opinions about the teaching of academic writing to students. The focus of the
interviews with teachers and management will be to ascertain the degree of referencing required in the
academic writing of students at the undergraduate, diploma and pre-university levels as well as the level of
interactivity that will be allowed in a learning environment without negatively affecting pedagogy.
A form of group interview – focus group – will be held with the students and/or teachers to further collect more
data and validate the data already collected. This activity will begin as soon as activity 1 has been concluded.
The information obtained from the testing out process and literature review will inform the shape of questions
asked in the questionnaires and interviews. It is anticipated that this activity will take about four weeks.
Activity 3: Data Analysis
Data collected from the questionnaires and interviews will be analysed using statistical software, e.g. SPSS as
well as qualitative analytical tools such as NVivo software. Apart from the software-based data analysis,
triangulation processes will equally be employed at this stage to assess the effectiveness of teaching
academic writing and referencing under particular sets of identified and reflected-upon ground factors, as
elicited, ranked and prioritised in activity 2. It is anticipated that this activity will take about six weeks to
complete.
Activity 4: System Design and Prototyping
Based on the result of the analysis of data gathering of the two previous methodologies, a prototype of the
proposed interactive multimedia-based learning environment to support academic writing and referencing
among young students will be designed and implemented. The integrated learning environment will be Web-
enabled and database-driven, employing techniques such as profiling and ontologies to create a meaningful,
rich, and fun-driven environment. It is anticipated that this activity will take about eight weeks to complete.
Activity 5: Evaluation
The developed system will be deployed for evaluative use in sample schools. Further surveys will be used to
evaluate the system and ascertain the level of acceptance among the target students. Ultimately, the
prototype will be integrated into the Moodle learning environment, currently used by the university. It is
anticipated that some teachers and management staff may need to challenge their existing ideas about the
way academic writing and referencing is currently taught as a result of the expected outcome of this work.
4. PRELIMINARY FINDINGS
Based on our literature survey and a preliminary field study, we have the following findings:
 Most existing online resources on academic writing and referencing do not incorporate sufficient
interactive multimedia materials to enable them to act as powerful learning environments.
 Most young students who go online, most often visit Websites that incorporate interactive
multimedia features.
 Students will most likely embrace academic writing if they are introduced to the subject at an early
stage in a way that addresses their yearnings as children and young adults.
29
5. CONCEPTUAL DESIGN
The system architecture for implementing the proposed framework for integrating multimedia resources with
academic writing and referencing materials consists of a backend database, an integrated learning
environment and diverse interfacing devices (see figure 2). The integrated learning environment is the main
component of the system, and is responsible for combining indexed multimedia resources with the materials
for academic writing and referencing in order to create an interactive environment to suit the particular needs
of students. Materials are presented according to the subjects, levels and required referencing styles selected
by a user. The integrated learning environment can employ techniques such as profiling and ontologies in
creating an environment that is meaningful, rich and fun-driven.
As shown in figure 1, it is envisaged that students can access interactive materials through everyday gadgets
commonly used by young people, such as iPhones and iPods.
Link to online libraries and academic
writing and referencing e-resources
Pedagogy,
PC Learner Learning
Rules and
Profile Resources
Presentation Layer – HTML,
Ontology
Writing and Referencing

VRML, Presentation &
Animation packages
Content Builder
Mac
Virtual Learning Environment (VLE)
Browser Interface
Multimedia Indexing/ Multimedia
Query Classification Resource
Processor Module Aggregator
Mobile Handheld
Gadget
Multimedia Resource Processing Component
Figure 1: Conceptual Design of Proposed System Multimedia

Database
The proposed system will have/be:

 Subject-specific/Level-specific;
 Hyperlinkable;
 Tailored towards the special requirements of students;
 Able to provide guidance on the ethics of research, e.g. plagiarism;
 Rich and interactive;
 Enable students to build research and referencing skills at an early stage in their learning;
 Customisable;
 Enable students to cultivate good writing skills;
 Discourage the use of infamous writing aids online;
 Make online resources for academic writing available via gadgets, such as MP3 players, iPod and iPhones
The different presentation formats of the multimedia are Text, Graphics, Animation, Audio Format, Video,
Hypertext, Concepts Visualisation, Slide Presentation and Interactivity. Appendix A shows the referencing
guide for academic materials.
6. CONCLUSION AND FUTURE WORK

This research work will be evaluated by using the prototype system by way of piloting testing with students of
Liverpool Hope University. The system will be validated based on its effectiveness in supporting learning of
academic writing and referencing in a flexible and interactive manner as well as its level of satisfaction to
students at the pre-university, diploma and undergraduate levels. Ultimately, the prototype will be integrated
into the Moodle learning environment which is the university’s current VLE.
Our future work consists of field study of selected schools, which will involve 1) data gathering through the use
of questionnaires, interviews and observations and 2) data analysis using mainly statistical methods, the
design and development of a multimedia database for storing learning resources and system prototyping.
7. REFERENCES
[1] Brabazon, T. (2007). The University of Google: Education in a (post) information age, Aldershot: Ashgate
[2] Foxcroft, A. (2009) Academic Writing. A literature Review. Unpublished draft. (under Review)
30
[3] Hetherington, R., O’Doherty, M., Owens, T., Shannon, L., Beaumont, C. (2007). “Raising Standards: A
Dialogic Approach to Improving Computing Students’ Writing for Assessment”, Proceedings of the 8th Annual
Conference of the Subject Centre for Information and Computer Science: HEA Subject Centre for ICS
[4] Khine, M. S. and Fisher, D. (2003). Technology-Rich Learning Environments: A Future Perspective,
London: world Scientific.
[5] Lee, P. (1995). “The Use of Multimedia Support Materials in Engineering Education”, 17th International
conference on Computers and Industrial Engineering, Elsevier Science Ltd., Vol. 29, N0. 1-4, pp. 65-69
[6] Lillis, T. & Turner, J. (2001). Student writing in Higher Education: contemporary confusion, traditional
concerns, Teaching in Higher Education, 6 (1), pp. 57 – 68
[7] Ma, Z. (2006). Web-based Intelligent E-Learning Systems: Technologies and Applications, London:
Information Science Publishing.
[8] Thomson, A. (2008). “Why we can’t turn our backs on the league-table generation”, Times Higher
Education (internet) 10th January 2008, Available at
http://www.timeshighereducation.co.uk/story.asp?sectioncode=26&storycode=400071 [Accessed 7/10/09]
Appendix A
Material Type Referencing Channel Language

Journal Research Strictly required with Text Formal language
a specified style
Conference Research Strictly required with Text/Audio/Video Formal language/
paper a specified style Simulation/ Presentation
language
Book chapter Research/ Required often with a Text Formal language
Teaching specified style
Tutorial Research/ May be required Text/Audio/ Video/ Formal language/
Teaching Animation Slide Simulation/ Presentation
language
Lecture note Teaching May be required Text/Audio/ Video/ Formal language/
Animation Slide Simulation/ Presentation
language
Article Research/ May be required Text Formal language
Teaching
Websites Research/ May be required Text/Audio/ Video/ Formal language/
language
Videos, films Research/ May be required Audio/Video/ Formal language/
and Teaching Animation Slide Simulation/ Presentation
broadcasts language
Presentations Research/ Not required Text/Audio/ Video/ Formal language/
language
Appendix A: Referencing guide for academic writing
31
ON SECOND LIFE AS AN E-LEARNING ENVIRONMENT FOR
TEACHING ACADEMIC WRITING
Obinna Anya Alma Whitfield Janette Bonar-Law Stewart Blakeway

Research Assistant Snr Lecturer E-Learning Developer Lecturer
Dept of Computer Science Dept of Computer Science Centre for Learning and Dept of Computer Science
Liverpool Hope University Liverpool Hope University Teaching Liverpool Hope University
Liverpool L16 9JD Liverpool L16 9JD Liverpool Hope University Liverpool L16 9JD
Liverpool L16 9JD
whitfia@hope.ac.uk
ABSTRACT
Second Life is currently one of the most sought after 3D online virtual worlds on the Internet. One of the
reasons attributed to this is the fact that Second Life, unlike most virtual environments before it, is able to
integrate the technology of online gaming, the concept of avatars, social networking and e-learning into a
single framework for the purpose of education, entertainment and social networking. As a result, Second Life
has, in the last few years, led to the development of systems and tools for teaching and structuring our
networked learning environments. However, existing research and e-learning applications based on Second
Life tend to focus on subject-based learning, i.e. the teaching of various courses of study often targeted at
distance and adult learning, whereas the area of academic writing has received comparatively very little
attention in spite of its important role as the bedrock of academic writing across disciplines. In this paper, we
present the preliminary research and the conceptual design of a prototype interactive e-learning environment
aimed at exposing students in the pre-university, diploma and undergraduate levels to academic writing and
referencing using Second Life. We suggest that exposing students to academic writing using emerging
innovative and interactive multimedia technologies, such as Second Life, will lead to cross-disciplinary
improvement in learning among young students.
Keywords
Second Life, academic writing, e-learning environment, social network
1. INTRODUCTION
Over the last two decades, advances in information and communication technology have led to
unprecedented use of Web technologies in re-defining the way we teach and structure our learning
environments [14]. A major trend is towards the creation of collaborative and personalised learning
environments [19] based mainly on Web 2.0 technologies [8]. In this environment, intelligent tutoring agents,
avatars and other virtual characters are increasingly used to adapt to the characteristics of each student
through the construction and analysis of models that reflect both behavioural and cognitive aspects of the
students [3].
Second Life is an immersive 3-D online environment, where people can meet to chat, interact, play, study and
work using avatars [6], [7]. Unlike previous immersive environments, Second Life objects can be easily
integrated with web applications and can be flexibly rendered on relatively low-specification desktop
computers. This makes Second Life a good platform for implementing emerging forms of mobile learning,
anytime anywhere interactive education, flash simulations and virtual collaboration [9], [20]. Reports show that
an increasing number of users, particularly young students, access Second Life. Indeed, a large active
education community is thriving on Second Life [16]. The Open University and a number of universities in
Harvard, Texas State, and Stanford in the US as well as Lancaster University in the UK are just a few of the
many universities that have set up virtual campuses where students can meet, attend classes, and create
content together. While there are a number of research efforts towards the use of Second Life for educational
purposes, most of these tend to focus on other subjects and fields of learning than academic writing. Studies,
such as [4], observe that students embrace academic writing with far less enthusiasm that ever before – a
development that is highly detrimental to learning and pedagogy.
In this research paper, we argue that if students are exposed to academic writing and referencing using
emerging interactive and innovative multimedia technologies, such as Second Life, they will exhibit greater
interest in the subject and will eventually improve their academic performance across disciplines. This paper
32
reports our preliminary investigation and presents a conceptual framework for applying Second Life as an
interactive environment for teaching academic writing to students at the undergraduate, diploma and pre-
university levels. Our approach is to codify existing academic and reference materials in the area of computer
science using online databases and electronic archives, in a way that will enable interactive access to them
by students learning academic writing on Second Life.
2. RELATED WORK
As a result of their use of virtual humans and the ability to foster a kind of tele-presence that allows
participants to share knowledge, ideas and experiences in a technology-augmented setting, networked virtual
environments have, over the last three decades, been increasingly applied in learning and teaching [14]. A
number of virtual learning environments have emerged as a result, and include Blackboard, CyberExtension,
It's Learning, Saba Centra, WebCT, Moodle and WebTrain (). Other e-learning environments relevant to this
work include Virtual Life Network – VLNET [5]. VLNET was developed at the University of Geneva, and
incorporates different media – text, sound, 3D models, images, video, autonomous gesture and facial
interaction – into a single NVE. The VLNET system supports shared virtual environment that allows multiple
users to interact with each other and their surrounding environment in real time.
Though Second Life was developed less than a decade ago, the Internet based virtual world has already
been remarkably applied as a teaching and learning tool in various levels of education (see [21]). An
interesting feature of Second Life, which makes it highly attractive as a VLE is its use of avatars as well as its
unprecedented popularity as a social network media. Second Life has, to date, recorded the highest number
of registered users among all 3-d virtual worlds [6]. One interesting Second Life based e-learning project is
the Schome-NAGTY Teen Second Life Pilot [6]. This is an European enclosed island for teenagers on the
Teen Second Life sector of the 3D virtual world Second Life, developed by the Schommunity with the aim of
creating "a new form of educational system designed to overcome the problems associated with current
education systems in order to meet the needs of society and individuals in the 21st century" (Schome
Community, 2007). The Schommunity is a group of people which includes academics, parents, young people,
policy makers, educators, and other interested parties.
This virtual community has engaged with a wide variety of perspectives on educational practices and potential
educational futures, and uses a range of online media, including a wiki and forum, to explore a wide variety of
perspectives on educational practices and potential educational futures in order to broaden thinking and
debate about educational possibilities as well as to gather evidence about the effectiveness of alternative
approaches in different contexts. The project is affiliated in the UK to The Open University, which as a
distance learning higher education institution committed to exploring the potentials of flexible learning
environments. Additionally, SLOODLE [17] has gone some way to integrating external resources and learning
environments with Second Life, by establishing synchronous and asynchronous links between Second Life
and an external VLE, namely MOODLE.
Outside of virtual environments, other projects that have influenced this research work include virtual
reference projects of the Anglia Ruskin University Library and the University of Southampton. The approach
taken in this research work includes integrating online database approaches for codifying resources with the
ability to present such resources in an interactive virtual environment.
3. PRELIMINARY EXPERIMENT
As a preliminary requirement of this research work, a field study involving a sample population of the target
students at the undergraduate, diploma and pre-university levels was carried out. The purpose of the study
was to examine the extent to which students at these levels understand the discipline of academic writing and
its importance, their level of use of modern technologies, such as iPod, Second Life, etc., and the likelihood of
their embracing academic writing if it is presented using these technologies. The study was conducted using
questionnaires, and preliminary analysis of initial data, which is still on-going, was based on statistical
methods. A total of 90 students from three schools across Liverpool participated in the survey.
Preliminary results of the study are in line with our earlier claim that the subject of academic writing is under-
presented to young students. Over 65% of the participants tend to consider the discipline of academic writing
as an arcane domain reserved only for academics or people who intend to pursue that profession, whereas
over 95% of them are well versed in using modern technology based tools, mainly for entertainment
purposes. In a separate study, [4] made a similar observation, and notes that as each semester progresses,
a greater proportion of students is reading less, referencing less and writing with less clarity and boldness. As
more students tend to evade academic writing, they are very eager to experiment with and use emerging
technologies, such as iPods and Web 2.0 tools such as Second Life. As a result, integrating virtual worlds,
such as Second Life, into academic education will potentially bridge the apparent gap between theory and
33
practice by providing opportunities for enriching the student experience, providing authentic context and
activities for experiential learning [13] as well as for simulation and role play [1].
4. SYSTEM DESIGN
The results of the field experiment in section 3 will be used to inform the content design of our proposed
Second Life based e-learning environment for teaching academic writing. The content delivery system will
consist of a client system based on Second Life interface and a server side, which will be responsible for
coordinating learning and generating learning resources from a backend store. We describe a prototype
multimedia-based environment for teaching academic writing based on Second Life (see figure 1).
4.1 Conceptual Framework

The conceptual design of our proposed system is on the hypothesis that students exposed to academic
writing and referencing using emerging innovative technologies, such as interactive multimedia technologies,
will exhibit greater interest in the subject and will eventually perform better or as well as they do in other
subject areas. Our goal is to develop a novel methodology for codifying existing academic and reference
materials in the area of computer science using online databases, electronic archives and multimedia
systems.
Client Side Server Side
Social Network Learning

Module Module
Collaboration Manager
Database
Figure 1: System Architecture
The system consists of the server side and the client side. The server side is organised into three major
modules – the collaboration manager, the learning module and the social network module. The learning
module is responsible for maintaining the learning materials according the rules of pedagogy as well as
formulating and structuring lessons. This could be enforced by the system through references to known
learning archives, such as the Pearson Educational resources (e.g. see figure 3). The social network module
is responsible for maintaining the social activities within the NVE and for integrating learning into the social
environment offered by Second Life. The collaboration manager is responsible for managing interaction within
the NVE as well as managing learning and social sessions, and connecting learners to members with
knowledge. Learning resources organised and stored in the database. Existing implementations of databases,
such as mySQL or SQL server are used; materials in the database are stored in XML format. The client side
uses a typical Second Life interface, either using the main or the teen grid mode.
Interaction with NVE starts with a user being identified based on age and educational level as well as learning
goals. After this stage, the system searches for members online or prompts the user for queries. See figure 2.
The goal of our proposed Second Life based e-learning environment is provide young people with scaffolded,
experiential collaborative online learning online learning, i.e. to engage them in learning in an integrated
environment that is entertaining as well. Scaffolded edutainment has been viable form for education, mainly
for youngsters and people with learning difficulties.
Identify user based

Specify learning Enter NVE – Connect
on age and
goal to Members
educational level
34
Figure 2: Flowchart for using the Network Virtual Environment (NVE)
4.2 Generating Resources for Teaching Academic Writing

One of the most popular means of achieving meaningful learning is through conceptual change [22], [11], [18].
Different theories of conceptual change emphasize the reorganisation of conceptual frameworks, which result
from different forms and models of activity and, which lead to the repair of students’ mental models (Chi and
Roscoe, 2002). The prevailing perception of a majority of students, from our preliminary study (see section 3),
is that academic writing is a discipline for aspiring academics. A change of this mental model is necessary for
the teaching of academic writing to young students. Our choice of Second Life is one way of addressing this.
Another way is to generate learning contents that would have the potential to arouse the interest of the
students.
In creating contents for learning academic writing within the Second Life environment, we propose the use of
learning technology specifications, such as the Educational Modelling Language (EML), developed by the
Open University of the Netherlands [10], [15]. EML supports flexible, independent learning without the
constraints of time and place, especially in situations with minimal modulation, such as the Second Life. With
EML, we build models of required domain content, problems, experiences, or thinking processes using
different representational formalisms that depict different kinds of conceptual change and interests in
students. Jonassen (2004) argues that model building is a powerful strategy for engaging, supporting, and
assessing conceptual change in learners because these models scaffold and externalize internal, mental
models by providing multiple formalisms for representing conceptual understanding and change. Besides,
models would acts benchmarks for validating sources of knowledge within the Second Life environment (see
figure 3).
A model-based learning in an open and interactive environment as Second Life will provide a means for
situated construction of knowledge in a social Web-based setting. Recent studies performed on avatar in 3-D
virtual worlds have shown that the kind of social relations and interactions in Second life are able to convey
feelings of social presence in users while their awareness of mediation by technology recedes into the
background [6], [2]. Our model-based approach would provide the following services: 1) user model service,
2) learning object model service and 3) interaction model service. The user model service represents users,
their profiles and roles; learning resources are represented by the learning object model services; while the
interaction model service represents various forms of interactions between users and avatars.
Knowledge
References
provider
Validating
Knowledge source
provided
Facilitator
Support/query
knowledge source
Community of Learners
Figure 3: Validating Knowledge sources
5. CONCLUDING REMARKS
Second Life exists in two distinct and mutually inaccessible types. Second Life (also known as the Main Grid)
is designed for adults aged 18 and over, while Teen Second Life (also referred to as the Teen Grid) is
restricted to teenagers between the ages of 13 and 17. With each type, users interact with one another by
35
manipulating their character, known as an avatar, within and to different regions known as islands, which are
typically owned by third parties. As noted in the related work section, Second Life is already being used in
numerous ways for education and collaboration. However most of these involve largely adults and focus on
subject-based learning, e.g. arts, science, etc.
As noted by [4], [12] in Second Life one does not need to know something in order to teach it – one only
needs to connect people who seek knowledge (students) with people who have knowledge. This idea of
facilitating learning among young people in a social network environment requires an approach for ensuring
that learning is provided according the required standards. As a result, a number of questions, namely, how to
guarantee the validity of the sources of learning materials arise. One approach offered in this work includes
allowing the VLE to automatically validate or query knowledge sources. See figure 3. Information to validate a
learning material could be provided as meta-information or tags.
Future work on this research will focus on implementing a Second Life e-learning environment based on the
proposed concept as well as a real life evaluation of the e-learning environment using target student
population
6. REFERENCES
[1] Belei, N., Noteborn, G. and Ruyter, K. (2009) Cross-World Branding – One world is not Enough, In Wankel, C. and
Kingsley, J. (Eds.) Higher Education in Virtual Worlds, UK: Emerald Group Publishing
[2] Biocca, F., Harms C. and Burgoon J. (2003)Toward a more robust theory and measure of social presence: review and
suggested criteria. Presence 2003, vol. 12 no. 5 pp. 456-480
[3] Boff, E. and Vicari, R. M. (2008) A Collaborative Editor for Medical Learning Environments, In Kock, N. (Ed.)
Encyclopedia of E-Collaboration, New York: Information Science Reference.
[4] Brabazon, T. (2007) The University of Google: Education in a (post) information age, Aldershot: Ashgate
[5] Capin, T., Pandzic, I., Mageneat-Thalmann, N. and Thalmann, D. (1999) Avatars in Networked Virtual Environments,
New York: John Wiley and Sons
[6] Gillen, J., Twining, P., Ferguson, R., Butters, O., Clough, G., Gaved, M., Peachey, A., Seamans, D. and Sheehy, K.
(2009) A learning community for teens on a virtual island - The Schome Park Teen Second Life Pilot project, eLearning
Papers, No. 15, ISSN 1887-1542, June, From www.elearningpapers.eu
[7] Gorini, A., Gaggioli, A., Vigna, C. and Riva, G. (2008) A Second Life for eHealth: Prospects for the Use of 3-D Virtual
Worlds in Clinical Psychology, Annual Review of Cybertherapy and Telemedicine, Journal of Med Internet, vol. 10, no. 3,
pp. 16-21
[8] Graham, P. (2005) Web 2.0. Accessed on 21/01/10 from httpp://www.paulgraham.com/web20.html
[9] Heiphetz, A. and Woodill, G. (2009) Training and collaboration with Virtual Worlds, McGraw-Hill
[10] Hummel, H. Manderveld, J., Tattersall, C. and Koper, R. (2004) Educational modelling language and learning design:
new opportunities for instructional reusability and personalised learning, Int. J. Learning Technology, Vol. 1, No. 1, pp. 111-
126
[11] Jonassen, D. (2004). Model building for conceptual change: Using computers as cognitive tools. In M. Gregoriadou, A.
Rapti, S. Vosniadou, & C. Kynigos (Eds.), Information and communication technologies in education (pp. 3 -17). Athens,
Greece: New Technologies Publications
[12] Joseph, B. (2007) Global Kids, Inc.’s best practices in using virtual worlds for education. Second Life Education
Workshop 2007 Part of the Second Life Community Convention Chicago (pp. 12-19), retrieved August 29, 2007 from
http://www.simteach.com/slccedu07proceedings.pdf
[13] Kolb, D. A. 1984. Experiential Learning: Experience as the Source of Learning and Development. Prentice-Hall, Inc.,
Englewood Cliffs, N.J.
[14] Khine, M. S. and Fisher, D. (2003) Technology-Rich Learning Environments: A Future Perspective, London: world
Scientific
[15] Koper, R. and Manderveld, J. (2004), Educational modelling language: modelling reusable, interoperable, rich and
personalised units of learning, British Journal of Educational Technology Vol. 35, no. 5, pp. 537-551
[16] Lim, K. Theory Is the Reason. (2006) "How NOT to Teach in Second Life.", Interview with Milosun Czervik (RL: Ross
Perkins) founder of the Second Life ICT Library. Accessed on 24/01.2010 at http://theory.isthereason.com/?p=1363
[17] Livingstone, D. and Kemp, J. (2006) Putting a Second Life “Metaverse” Skin on Learning Management Systems, in
proceedings of the Second Life Education Workshop at SLCC, Livingstone and Kemp (eds.), San Francisco, August 2006,
p.13-18.
[18] Lundholm, C. (2008) ‘Students’ conceptual change- emerging issues’ Paper given at the 5th Professional
Development Conference for teachers in Higher Education, ‘Nurturing the learner; the central role of the tutor in higher
education’ Institute for Education Policy Research, Staffordshire University.
36
[19] Ninomiya, T., Taira, H. and Okamoto, T. (2007) A personalised learning environment architecture for e-Learning,
Proceedings of the sixth conference on IASTED Int’l Conf Web-Based Education, Chamonix, France, Vol. 2, pp. 517-521
[20] Ryan, M. (2009) 11 Ways to Use Second Life in Education, Lancaster University, Accessed on 23/01/2010 at
http://www.slideshare.net/millpubco/11-ways-to-use-second-life-in-education-michele-ryan
[21] Wankel, C. and Kingsley, J. (2009) Higher Education in Virtual Worlds: Teaching and Learning in Second Life,
Londom: Emerald Group Publishing
[22] Wichmann, A., Gottdenker, J., Jonassen, G. and Milrad, M. (2003) Developing a Framework for Conceptual Change
within Scientific Inquiry, Advanced Learning Technologies, 3rd IEEE Int’l Conf on Advanced Learning Technologies
(ICALT'03)
37
EDUCATION IN LIBRARY MANAGEMENT SYSTEMS (ELMS)
Alan Poulter
Computer & Information Sciences
Strathclyde University
26 Richmond St., Glasgow, G1 1XH
alan.poulter@cis.strath.ac.uk
http://personal.cis.strath.ac.uk/~ap/elms/page
ABSTRACT
In this paper, we describe the ELMS (Education in Library Management Systems Project), which virtualised
teaching on the ‘Digital Libraries’ module at Strathclyde University. It examines teaching all aspects of library
management systems and related topics MARC (MAchine Readable Cataloguing) and databases. Current
developments in adding teaching of mash-ups and simple scripting are outlined.
Keywords
Libraries, Library Management Systems, MARC, databases, mash-ups.
1. INTRODUCTION
The ELMS project grew out of teaching on the MSc Information and Library Studies course, at the Department
of Computer and Information Sciences at Strathclyde University. This teaching took a novel approach to
including core concepts in a practical manner. An explanation of library management systems will be given,
and then attendant pedagogic problems and solutions will be outlined. For more background see (Poulter
2009), a paper which was presented at the first UK conference on open source applications in librarianship.
2. LIBRARY MANAGEMENT SYSTEMS

Libraries have a long history of using computer systems to enhance their operations. Library automation
systems were early attempts, sometimes using completely home-grown software applications, to computerise
library operations. Nowadays there is little need for pioneer work in this area and libraries will typically use a
commercial library management system (LMS), except where their needs are unique (as in the case of
national libraries or some special libraries) or they have the technical know-how to use a free open source
LMS.
A library management system is essentially a relational database customised for use in running a library and
supporting its operations. Like software applications generally, an LMS comes in modules, all of which share
a common data format (or formats), but which perform different functions on that data. A useful directory of
library management systems is provided by Open Directory (Open Directory 2010).
Fundamentally an LMS does two things:
- supports all the ‘back-end’ operations of a library – acquiring stock/information, cataloguing
stock, loaning stock/disseminating information and reporting on these functions to enable
effective service management.
- enables users to find items or information in a library, via a catalogue, and then borrow/access
as appropriate their required items/information
3. ACQUISITIONS
Back-end operations are not as simple as they might seem. Acquiring stock is perhaps the most complex sub-
system of an LMS. Basically, a stock selection system should allow the input of enough basic information (e.g.
an ISBN, title and author) to identify for ordering purchasable items and to link to selection recommendation
systems and to an internal database of publishers and suppliers. However, there are many potential
complications. Firstly, the same item should never be acquired unwittingly more than once. Secondly, budgets
must be met so there is a need for a real-time, accurate summation of both monies spent and committed.
Third, purchased items may not arrive and items that do arrive may do so by mistake. There needs to be ways
of generating orders for suppliers automatically and send those orders. EDI and its successors are especially
38
helpful here.
Stock comes in two generic forms, ‘monographs’ or ‘serials’. A monograph, typically a book, comes
complete as is. A serial, typically a journal, does not come complete as is. It will appear, hopefully, at regular
intervals into the future. Thus each issue of a journal needs to be checked for receipt: if an issue is missing,
then a replacement copy needs to be obtained. The title or supplier or even the nature of a serial may change.
Post arrival, serials are typically not permanently stored as discrete items but bound together as annual
volumes: this needs to be recordable on the acquisitions system. The payments for serials, typically
subscriptions, are continual and also subject to change, so handling the financial side of acquiring serials is
more complex. It requires decisions need to be made with regard to future financial (and other)
circumstances.
3.1 Circulation
The ability to handle lending stock, in the form of a circulation system, is another complex sub-system of a
typical LMS. Items themselves need to checked on return or when they leave a library, to record the change
in their borrowed status. Optical barcode reading systems typically do this, although wireless barcodes (RFID
tags) are also used. Checking systems can be staff- or user-driven, like checkout tills in supermarkets. Most
libraries have their own loan policies and thus a circulation control system needs to be flexible enough to
accommodate a wide range of loan regimes (and to allow change in those regimes over time). Loans are in
themselves complex as they are an overlap between two core databases, one holding stock and another
holding user details, which is generally done using a third database which pairs items loaned with borrowers.
User details are required so that users can be authenticated. This is vital in permitting their access both to
items/information in library and for access to external resources that a library may acquire access to for its
users. While the user database needs to be able to hold an agreed set of identifying details, the stock
database needs to hold details of how many occurrences of each item a library holds and where they
currently are. For individuals, the circulation system needs to know how many items they have on loan, which
they must return, their contact details for recall notices and their current status in terms of overdue fine.
3.2 Cataloguing
The cataloguing module of an LMS is much less complex in its operational requirements than acquisition or
loan LMS modules, but can be very complex in the detail and richness and format of the data stored for each
item. Since cataloguing is very much a cooperative and shared task, some form of input and output function
would be required, to allow the import of records created elsewhere (e.g. in shared union and cooperative
catalogues) and the export of locally-created records into a central sharing pool. The standard record type
handled is MARC - MAchine Readable Cataloguing (MARC 21 Format, 2004).
Cataloguing is itself a complex task (Bowman 2002) .A complicating factor is the need for authority
file control. In individual item records there will be terms that appear in other item records: personal and
corporate names, subject terms and headings and possibly classification codes. The usage of these terms
must be standardised, for example the form of name used for the same author must be consistent throughout
a collection. Each authority file requires a separate database, linked to the cataloguing module, to both
validate individual names and terms and also to record connections between them (for example, the
pseudonyms used by an author and the subject terms that relate to one term, as in its narrower and broader
terms). Authority files will probably be sourced from external suppliers.
4. PEDAGOGIC ISSUES
It is important for students of librarianship to have the opportunity to:
- configure an LMS for a given library scenario as though a library systems administrator
- for their own configuration, experience its front-end capabilities as a user
- use the cataloguing sub-system to create MARC records
A new post-graduate module, ‘Digital Libraries’, was created in 2008. It was intended both to preserve content
from previous modules, but also to add new content, regarding teaching library management systems. Other
modules, notably ‘Organisation of Knowledge’, contained content on cataloguing (Anglo-American Cataloguing
Rules 2005), which was relevant to this module. The explicit linking was made clear to students, who
appreciated the chance to try out in one module, theoretical content given in another module.
Also included in the ‘Digital Libraries’ module as a necessary precursor, was material on relational databases.
While courses in librarianship typically include material on dealing with unstructured, chiefly textual,
39
information, it is rare to find database design content. In our librarianship Masters, for example, there is a
module ‘Information Retrieval’ for this type of content. Database design was approached in two ways. Firstly,
lectures on relational databases gave students a conceptual model of what a database does. Second, to
reinforce this understanding, an assignment was set which required students to store the references in a
bibliography from a previous assignment in a database. A previous assignment was used to ensure each
student had a different set of resources to deal with.It was felt that Access or Oracle would be too difficult to
master in a short time in lab practicals, so an online web 2.0 database, DabbleDB (http://dabbledb.com/) was
used. Students thus encountered problems like how to handle authors, when forms of name changed and an
item could have zero, one or more authors.
4.1 Koha
After dealing with databases students were presented with a real LMS. Since LMS are expensive it was
decided to use an open source LMS. Koha is the fully fledged library management system (LMS), available for
free as open source software, that was chosen. Currently version 3.2 is the latest. For its history and
development please see http://www.koha.org. Rather than use a single version of Koha, by running one copy
taking requests on different ports from different users, it was possible to give each student in a class of thirty
their own copy of Koha. The concept of open source was introduced through lecture content on: open
standards, the development of operating systems and the range of software available. The LAMP stack (Linux
+ Apache + MySQL + Perl) was also explained.
In the practical Koha exercise devised, each student had to set up library branches, item types, borrower
categories, loan and charging policies, book funds, currency handling and supplier accounts. The end result
was that each student should add themselves as a borrower and reserve an item, thus appreciating from a
users view the workings of their LMS. It may be considered easy to setup things like item types (e.g. adult
fiction, children’s fiction etc,) and loan and fines policies (e.g. children not being allowed to borrow from the
main collection) but the interplay between the set of characteristics defined is extremely complex, and gives
students a realistic experience of managing a complex entity.
4.2 MARC
If the mere ‘administrative data’ was complex, students soon discovered that the MARC structures within
Koha were even more complex. Entering different MARC ‘tags’ (three digit codes) brings up the entry forms
for them in Koha. Each tag has different ‘sub-field codes’ consisting of ‘$letter’.To keep the complexity
manageable for students, they were limited to the following set of common MARC tags:
100 Main author name
245 Title and author statement
250 Edition
260 Publication details
300 Physical description
440 Publication series
500 General notes
700 Subsidiary author name
One facet missing from the above list is item identifier. ISBNs cannot be used as not all library items have one
and not all ISBNs are unique. Using Koha, students had to create local id tags and realized how problematic
item identification could be. As part of the assessment for the ‘Digital Libraries’ module, students had to enter
6 MARC records of books from 6 different authors with the same surname as the student (dropping letters if
need be).
5. EDUCATION FOR LIBRARY MANAGEMENT SYSTEMS (ELMS)

We have tried to expand the teaching use of the facility described above by:
- sharing it remotely with other appropriate accredited MSc ILS cohorts in the UK
- adding other open source LMS packages, such as EverGreen
Koha is remotely available at:

https://elms.cis.strath.ac.uk:8200/cgi-bin/koha/opac-main.pl
https://elms.cis.strath.ac.uk:8300/cgi-bin/koha/mainpage.pl
40
The above administrative account requires a userid and password, which can be issued.
Evergreen is also available.
- developing a new teaching objective around user interface customisation, to allow students to
experiment with new service offerings, for example. linking to Amazon book reviews, allowing users to
tag etc (Engard 2009)
- exploring a potential crossover with the CS curriculum, through a new emphasis on using scripting,
simple programming etc in interface design and customisation
These last two issues relate to the user-facing aspect of a LMS which not be ignored, as it can be the ‘public’
face of a library service. The term for this is OPAC – Online Public Access Catalogue. This name stems
from the days when a library’s collection only was available from it. Early LMSs sported very limited and
difficult to use interfaces. Now a web front end is standard and LMSs look and act like internet search
engines. Searching is easy and so is the range of self-service options (reservations, inter-library loan
requests, stock acquisition requests etc). Usually the OPAC module within an LMS is customisable, in terms
of its appearance and facilities, so that each library can create its own look and feel, with colour schemes,
logos etc. Very often an OPAC can function as a single-stop portal, available at all times, to the online
services and digital information resources that a library offers. Perhaps the main remaining issue is that a
screen cannot give access to content still bound in physical items. Mashups and scripting are vital here for
delivering a customized feel.
Finally, some interesting implications for real applications of LMS were also uncovered, especially in providing
training for librarians in developing countries. Running an LMS remotely would deliver a much-needed
platform for accessible learning and training.
6. REFERENCES
Anglo-American Cataloguing Rules (2005), 2nd ed. 2002 revision: 2005 update, Facet Publishing, American Library
Association and Canadian Library Association.Publishing.
Bowman, J.H. (2002). Essential cataloguing. Facet Publishing.
MARC 21 Formats (2004). Guidelines for the Use of Field 856, Revised March 2003. Network Development
and MARC Standards Office, Library of Congress. http://www.loc.gov/marc/856guide.html
Open Directory (2010). Reference: Libraries: Library and Information Science: Software.
http://www.dmoz.org/Reference/Libraries/Library_and_Information_Science/Software/ (last accessed 21/5/10)
Engard, C. (2009) Library mashups. Facet,
Poulter, A.J. (2009) Koha in the University classroom. Open Libraries: Breaking the Barriers 2009.
41
INQUIRY-BASED LEARNING AND INFORMATION LITERACY WITH INTERNATIONAL
MASTERS STUDENTS
Briony J Oates Elisabeth Yaneske Gary Griffiths

School of Computing School of Computing School of Computing
Teesside University Teesside University Teesside University
Middlesbrough Middlesbrough Middlesbrough
B.J.Oates@tees.ac.uk E.Yaneske@tees.ac.uk G.Griffiths@tees.ac.uk
ABSTRACT
We discuss a masters module which was designed to address two significant problems: 1) How to introduce
international masters students to ‘learner-centred teaching’ and ‘inquiry-based learning’ (IBL) approaches,
and 2) How to develop research & information literacy skills in all our masters computing students. To solve
these problems we designed a new module which incorporates a structured sequence of IBL activities, which
act as scaffolding to help international students understand and adjust to masters-level research-informed
teaching approaches, including collaborative learning (e.g. via groupwork, presentations and peer review) and
reflective learning activities. The module embeds research & information literacy skills within the context of a
group project where students design and implement a website. The organizing scheme for the information
literacy skills development is the SCONUL Seven Pillars Model of Information Literacy. We discuss the design
and implementation of the module, and an independent module evaluation which used feedback from the
students, via an on-line survey, interviews and focus groups, and a reflective dialogue with the module leader.
We see the module development as an ongoing action research project and therefore conclude by suggesting
enhancements for the next iteration. In presenting this paper we hope to stimulate discussion about how we
can meet the needs of international students within the time constraints of a one-year taught masters
programme
Keywords
Inquiry-based learning, information literacy, research skills, web development.
1. INTRODUCTION
There are increasing calls to use ’learner-centred teaching’ (Blumberg, 2009) and ‘inquiry-based learning’
(IBL), and to move away from a didactic, ‘chalk and talk’ pedagogical approach (Audet & Jordan, 2005; Barell,
2007). In an IBL approach the tutor outlines an initial problem or topic and then adopts a facilitative role while
the students, individually or in groups, learn through investigation. IBL requires the students to (Audet &
Jordan, 2005; Hepworth, 2009; Lee, 2004):
1. Establish the area of investigation/problem/task
2. Identify key issues and appropriate questions
3. Investigate sources of information or evidence
4. Develop a plan and take action
5. Report outcomes
6. Reflect on the process so far, re-analyse the scenario, identify remaining gaps
7. Repeat until complete/out of time
In the information and computing sciences field a common example of IBL approaches is the final year
undergraduate project, where students manage a large task themselves and develop their ability to define a
problem, explore possible solutions, design, implement and evaluate a solution, and reflect critically on the
project product and process. However, many of our masters students do not have a UK undergraduate
degree and have not experienced IBL approaches.
In this paper we discuss the development of a masters level module, which was primarily designed to support
international (non-EU) students, who have little previous experience of IBL approaches and yet need to adjust
42
rapidly to this mode of learning in order to cope with masters level study. The module also aimed to develop
the research & information literacy skills of all the masters students, whatever their previous educational
experience, and to help computing students appreciate the importance of these skills. In the following
sections we analyse the problem, discuss our module’s design and implementation and evaluate the
outcomes.
2. PROBLEM ANALYSIS
Our concern was the learning experience of students at masters level in our School of Computing. In common
with many other UK universities, the nature of our masters cohort (around 100 students annually) has
radically altered over recent years. Where the cohort used to be predominately students with first degrees
from our own teaching-intensive institution, who have already experienced a wide range of IBL approaches,
the majority are now international (non-EU) students, with no previous exposure to such approaches. Instead
they have relied on their teacher to give them the necessary information, which is learnt parrot fashion for
exam-based assessment (e.g. Hayes et al, 2006; Rowland, 2006) They therefore need help to adapt to IBL.
A pre-masters course for international students can provide an introduction to the study skills required for HE
study in the UK, but at our institution the majority are admitted straight onto a masters course where they
study alongside UK graduates. Our first concern was:
How can we introduce international students to ‘learner-centred teaching’ and ‘inquiry-based learning’ (IBL)
approaches within a two semester taught masters programme?
An in-house survey of our international masters students found that, as well as difficulties adjusting to new
study methods and pedagogical practices, students had not settled well into the area (“Teesside is boring”, “in
our free time we just watch DVDs in our student houses”) nor into student life (“all the available student
activities involve alcohol, so we don’t go”). Our second concern was therefore:
How can we help international students to make the most of university life beyond their study time?
Also, we found resistance from our master students (of whatever nationality) to developing their research
skills and understanding of how knowledge is created in computing. They struggled to see the relevance of
research methods to them, expressing a strong preference for developing their technical skills (e.g. “I’m a
programmer not a social scientist!”). Our third concern was therefore:
How can we engage masters computing students in developing their research & information literacy skills?
To address these concerns we designed a new module, Project, Research and Development’.
3. DESIGN SOLUTION
The new module, ‘Project, Research and Development’, was based on group-work and IBL. It:
1. Incorporated a structured sequence of IBL activities which acted as scaffolding (Vygotsky 1978) to help
international students understand and adjust to masters-level pedagogical approaches. These involved
collaborative learning (e.g. via group-work, presentations and peer review), and tasks which helped them
adjust to the academic and social life at a UK university.
2. Embedded information literacy and research skills within the context of a group project where students
designed and implemented a website.
3. Developed students’ meta-cognitive skills (i.e. ability to reflect on how they learn) by incorporating group
and individual reflection on both IBL and the group activities.
Students were divided into groups. They were asked to develop and sign a team contract, which specified
how the group would operate and how conflicts would be resolved. They were also asked to record action
minutes for each group meeting.
The groups were asked to develop a website following the SWM method (Simple Web Method) (Griffiths et al,
2003; 2004). This gave them a structured process with specified deliverables for each stage. They had not
previously studied SWM, and were expected to find out about it from the resources provided. For example, for
the Planning stage, they had to produce a Quality Plan and a Project Plan, and for the Analysis stage they
had to address audience analysis, content analysis and market analysis – the meaning of each of these terms
first had to be investigated before the deliverables could be produced.
The website content comprised the outputs from the students’ IBL activities, which were designed to help
international students adjust to student life at Teesside and to develop all students’ research & information
literacy skills. The IBL tasks built up in terms of difficulty throughout the module, and moved from a focus on
settling students in to more formal academic tasks. For example:
 In week 1 the students had to produce a photograph of themselves standing in front of a Middlesbrough
landmark, the photo to be taken by a fellow student on the module, and they must find out the fellow
student’s name, background and masters route.
43
 Subsequently in groups they had to produce a guide for future international students describing two
activities they could do while living on Teesside that did not involve alcohol, and the guide must include
photographs of the group actually doing the activities.
 Final tasks involved the group carrying out a systematic review of the research literature on a topic they
chose, selecting the 10-30 most relevant papers (giving complete reference details) and analysing and
evaluating three of them. (A systematic review gives the tutor more insight into the students’ literature
searching skills than with a normal literature review, because the process has traceability: students
include a description of how they searched the literature: the initial research question, the keywords used
and databases searched and how their searches were modified and refined in order to narrow down to
the most relevant articles (Oates & Capper, 2009)).
Our intention was that the focus on developing web content meant that the students would see a need for
developing their research & information literacy skills, they were not simply completing academic exercises.
Students were also set meta-cognitive tasks where they reflected on how they were learning via IBL, and the
information literacy skills they were developing. Three organising schemes (Blumberg 2009) were provided to
act as conceptual frameworks as students integrated their activities with the underlying academic rationale:
the stages of IBL (see Section 1, derived from Audet & Jordan, 2005; Hepworth, 2009; Lee, 2004), the Seven
Pillars Model of Information Literacy (SCONUL, 2008) and the 6Ps model of research (Oates, 2006)
(Purpose, Process, Product, Participants, Paradigm and Presentation.
4. IMPLEMENTATION
4.1 The tutoring/facilitation approach
The module was implemented as a 15 week ‘short, fat’ 20 credit module. 50 students took the module in
2009-10. Students were timetabled for 2 hours of lectures per week (one focussed on the web development
process and one on web content and reflection), Staff used between a third and a half of these lecture slots,
but often used group-based activities within the lecture theatre rather than formal lecturing. For the remaining
lecture slots staff were available to groups for consultation in their offices, but did not lecture. Students also
had 2 hours per week in a computer lab (again one for process and one for content and reflection), where
staff facilitated their group activities and gave formative feedback.
4.2 The information resources and process support

Some lectures were given on web development and research methods, IBL and the SCONUL (2008) 7 Pillars
Model of Information Literacy. Library staff also led two sessions: using the online databases to search for
academic publications, and using the RefWorks bibliographic software tool.
The module staff provided resources via their personal webpages on SWM (Griffiths et al, 2003; 2004), a
structured approach to web development. A key resource for research methods and information literacy was
the book, Oates (2006) which explained the nature of research in computing and provided guides for
evaluating literature reviews, presentations and research publications based on a range of research
strategies. Examples were also provided of the critical analysis and evaluation of computing research
publications, and students were introduced to mindmaps as a way of summarising a paper’s content. To
support the group-work process students were given a team contract template and action minutes template.
Further online resources and feedback were provided via the institution’s Virtual Learning Environment (VLE)
(Blackboard). Student groups also used the VLE for email between members, file sharing and online
discussions. Emails between staff and students were also used on occasion.
4.3 The assessment

There were staged hand-ins throughout the module. The web development process carried 40% of the marks
and comprised three elements: a Planning and Analysis Document, a Design Document and a website
developed in Dreamweaver. Each element was handed in, but also presented to the tutor in a project review
meeting. All group members were expected to participate in the review meeting. The progress and quality of
the group’s work was assessed, together with group coherence and individual contributions to the work, the
presentation and the discussions at the meeting.
The web content also carried 40% of the marks, and it comprised two elements. The ‘Learning to do research’
element asked for photographs of the group members in front of Middlesbrough landmarks, the guides to two
social activities (with photographs of the group members participating), a summary of an academic research
paper and a group reflection on the research they had to do in order to complete all these tasks, and how that
related to inquiry-based learning and the SCONUL 7 pillars model of information literacy. The ‘Research using
44
the published academic literature’ element asked for a systematic search of the research literature on a
computing topic chosen by the group, and analysis & critical evaluation of three of the journal or conference
papers found in the systematic search.
Finally students submitted an individual reflective essay at the end of the module, worth 20% of the marks, in
which they discussed how the work the group did related to IBL and the SCONUL (2008) 7 Pillars Model of
Information Literacy, the effectiveness of their group, their reflections on their personal contribution to the
group-work, how they had developed or changed as a learner and an IT professional by doing the module,
and how what they had learnt would be carried forward into their masters project and dissertation.
5. EVALUATION
An Independent module evaluation was conducted by a colleague using feedback from the students, via an
on-line survey, focus groups, and a reflective dialogue with the module leader. Overall students were very
enthusiastic about the module and the way it had been taught. In both the survey and focus group responses
most students felt the IBL methods used in the module had motivated them to engage actively in their own
learning, increased their confidence in research & information literacy skills and enabled them to become
autonomous learners. International Students recognised IBL as a different way of learning for them where the
student rather than the lecturer takes control of what is learned. As one focus group put it ‘You make the
questions and get the answers’. Students found the IBL method to be a more interesting and social method of
learning compared with the teacher as expert model. They also felt that peer-peer support was more
encouraged with IBL. Students had particularly enjoyed the early IBL tasks which were focussed on the social
side of student lives. The staff felt that using non-academic activities was less daunting for students new to
higher education in the UK, allowing them to learn about IBL itself before moving on to more formal academic
work.
Students were able to articulate the inquiry based learning process in the context of a computer systems
development project. e.g. an online student survey question: “If you were asked to find out about the history of
the internet and then display that information as a website what steps would you take?” (N= 19) Most students
answered that they would first have to do some research into the history of the web (14) and most then
recognized they would then have to focus on building a website to display that information (11). Some
students also identified evaluating the information (4) and identifying key questions (2)
The collaborative nature of the IBL process and the skills gained through working on a computer systems
development project in teams was particularly valued by students. All of the focus groups discussed how
important it was for the groups to meet regularly to discuss their work or ‘think together’ as one group put it.
This was very different to most of the students’ previous experiences of group work (if any). For many
students group work had involved allocating tasks to each member then working remotely on an individual
piece of work. This work would then be collated at the end and handed in. Communication was done remotely
by phone or email. In this module they felt they could not succeed without meeting for discussion.
One negative point discovered in the evaluation was that students felt the module was really two half modules
running in parallel – one on IBL and research & information literacy skills, and on web development. Our
delivery pattern (see above) reinforced this perception.
The reflective dialogue with the module leader revealed how her IBL skills had been developed through
teaching this module. “It has refreshed my teaching. I used to do a lot more of this kind of thing. A lot more of
setting them a task and then they go off and do it. As we’ve become more tied down – less teaching contact
hours, more we must follow what is in the module guide, you must put a schedule up at the beginning with
exactly what you are going to do, it‘s been easier to just package it up– that’s a lecture, that’s a tutorial and
you can just pick it up next year and use it again. My teaching has been forced into particular boxes. This
module gave me an opportunity to go back to – what am I trying to teach here? What am I trying to get the
students to do? To move away from a content focus.”
6. CONCLUSION
This paper has described the development and implementation of a module designed to introduce
international (non-EU) masters students to IBL and to develop all masters students’ research & information
literacy skills. We see the module development as an ongoing action research project and therefore conclude
by offering two suggestions for how the next iteration will be improved.
Firstly, we need to consider whether to continue using the SCONUL (20008) Seven Pillars Model of
Information Literacy as one of the organising schemes for the module. This has two dimensions, one
concerned with information literacy skills (searching, evaluating etc) and the second concerned with current
level of expertise (from novice through to expert). This is a useful model, but the first dimension is very similar
45
to the model of IBL stages we gave the students (see section1). Examination of the students’ reflective essays
shows that the majority saw the SCONUL model as simply a rewording of the IBL stages. Only one student
(out of 50) discussed the second dimension of the SCONUL model and how his learning had moved him
along the novice-expert dimension for each of the information literacy skills. If we use the SCONUL model
next time we will need to spend more time showing the students how it is different from the IBL stages.
Secondly, we need to ensure better integration between the two module strands: web content and web
process. We plan to provide students with a better client brief so that it becomes more obvious why the
website requires the specified web content. Also after the first iteration of the module each staff tutor has a
better understanding of what the other is doing with the students, so next time it will be easier to refer to the
other tutor’s work during lectures and tutorials, leading to greater module cohesion.
The evaluation shows that overall the module was successful. Staff found the module enjoyable and it re-
invigorated the module leader’s teaching. Students also found the module enjoyable, and they acquired both
IBL and research & information literacy skills, which they could take forward into their other taught modules
and the masters project and dissertation.
ACKNOWLEDGEMENTS
This project was partly funded by a CILASS Subject Centre IBL External Project Grant – 2009.
REFERENCES
Audet, R. H. & Jordan, L. K. (eds.), (2005), Integrating Inquiry across the Curriculum, Corwin Press,
Thousand Oaks, CA
Barell, J., (2007). Problem-based Learning: An Inquiry Approach, 2nd edition, Corwin Press, Thousand Oaks,
CA.
Blumberg (2009), Developing Learner-Centred Teaching. A Practical Guide for Faculty, Jossey-Bass, San
Francisco.
Griffiths, G., Lockyer, M. A., Oates, B. J. & Hebbron, B. D., (2004). SWMing at Teesside, In Proc. JICC 8, (8th
Java and the Internet in the Computing Curriculum), 26 January 2004, London.
Griffiths, G., Lockyer, M. A., Oates, B. J. & Hebbron, B. D., (2003). Simple techniques for complex web
applications, In Proc. IADIS (Int. Association for Development of the Information Society) Int. Conf.
WWW/Internet 2003, , Algarve, Portugal, 5-8 November 2003, pp 1068-1071.
Hayes, N, Whitley, E. Introna, L., (2006), ‘Power, Knowledge and Management Information Systems
Education: The case of the Indian Learner’, In Proc. International Conference on Information Systems
(ICIS), December, 20006, Milwaukee. Available online: http://www.lums.lancs.ac.uk/files/icis2006.pdf
(accessed 21 April 2010).
Hepworth, M., (2009), Developing academic information literacy for undergraduates through inquiry based
learning, ITALICS, 8 (2), pp 2-13
Lee, V. S. (ed.), (2009). Teaching and Learning Through Inquiry. A Guidebook for Institutions and Instructors.
Stylus Publishing.
Oates, B. J., (2006). Researching Information Systems and Computing. Sage, London.
Oates, B.J. & Capper, G., (2009) ‘Using systematic reviews and evidence-based software engineering with
masters students’, IN Proc. EASE 2009, 13th International Conference on Evaluation & Assessment
in Software Engineering, Durham, 20-21 April, 2009.
Rowland, S., (2006). The Enquiring University. Compliance and Contestation in Higher Education, McGraw-
Hill/Open University Press, Maidenhead.
SCONUL, (2008), ‘The Seven Pillars of Information Literacy’, Society of College, National and University
Libraries, http://www.sconul.ac.uk/groups/information_literacy (accessed 21 April 2010).
46
ENHANCING STUDENT ENGAGEMENT THROUGH PAIR
PROGRAMMING
Sonya Coleman Eric Nichols
University of Ulster University of Ulster
Northland Road Northland Road
Londonderry Londonderry
sa.coleman@ulster.ac.uk Nichols-E1@email.ulster.ac.uk
ABSTRACT
Pair programming is an approach often used in industry to increase productivity and reduce programming
error. Although it typically takes longer to program when working in pairs the resulting product is often of
better quality than that obtained when working alone. This paper extends this concept to pair programming in
the classroom by introducing it into a first year, second semester programming module to encourage and
increase student attendance and therefore enhance student engagement. This paper presents the
methodology used and initial results obtained which indicate that attendance at practicals has increase in this
academic year and also that practical assessment marks have increased in comparison to previous years.
Keywords
Pair programming, first year module, engagement, attendance
1. INTRODUCTION
The concept of pair-programming has recently been introduced in the United States, both in companies and
universities. Within an industrial setting, software developers spend 50% of their time working with one other
person and this is due to the fact that it is known that working in teams of two (or more) improves productivity.
Although it typically takes 15% more development time, pair programming improves many aspects of
programming including design quality, reducing defects, enhancing technical skills and improving team
communications; these are all skills that we also require in our students.
Much research has been carried out and published on the benefits of pair programming within both industry
and academic. For example, within an industrial setting, Arisholm et al., [3] carried out a controlled experiment
in pair programming using a combination of junior, intermediate and senior Java consultants within Europe.
The results suggested that the most benefit was gained by the junior developers in that they could complete
tasks in less time and more successfully than normal with the support of a partner. The research in [5]
indicated that, given the correct pairing, pair programming could be very successful.
Within academia, pair programming has been introduced but primarily in the US rather than the UK. The work
of Braught et al., [4] determined that students were more likely to successfully complete their course using
pair programming and are also strengthened as individual programmers. In [6, 7] this was also found to be
true as introducing pair-programming was proven to increase student engagement and confidence
(particularly in female students and weaker students) and thus increase their programming skills. This is turn
leads to more students successfully completing the programming modules and therefore improves first year
retention.
Student retention and deep learning is an ongoing issue throughout the UK. In this paper, we present the
initial evidence for increased student engagement in Algorithmic Programming II now that pair programming
has been introduced to the module. We analyse the module results for another module over two years as a
control measure to establish consistency between the two groups being compared thus ensuring that any
improvement in performance is not simply due to the ability of the student cohort. We demonstrate that
working in pairs encourages students to attend practical classes by comparing and contrasting attendance to
date with the equivalent attendance records for the module in 2009.
47
We also illustrate that overall, the practical assessment marks have improved over those obtained in 2009,
hence demonstrating increased engagement and motivation, which will ultimately lead to improved module
results.
2. METHODOLOGY
Previously in the Algorithmic Programming II module, attendance at practicals and practical assessments has
been poor. Generally these students failed their coursework, resulting in a high failure rate in the
programming module. By introducing pair programming, in which the students are working and assessed in
pairs, the students are more likely to engage in the module, learn from each other and motivate one another
to participate in the practical and assessment aspect of the module. This should in turn improve the pass
rates within programming modules, support deeper learning and produce better, more informed programmers
by the end of the module.
Pair programming was therefore implemented in the following way. With respect to carrying out practicals and
practical assessments, the students worked in pairs. The students were allowed to choose their own
partners, as randomly selecting partners can lead to many problems during the course of the module. In their
pairs, the students completed the practicals and at specified times throughout the semester practical
assessments where conducted with each student pair. However, assessment was still marked on an
individual basis in the following manner:
a) both students must be present at assessment in order to obtain their individual mark;
b) both students were asked individual questions about the practical in question, hence both students
had to demonstrate their own understanding.
By conducting the assessment in this way, no individual student received credit for their partner’s work.
3. RESULTS
At the end of the module, the success of pair programming was assessed. This initial assessment was two
fold: a full statistical analysis was conducted to assess the overall improvement of practical assessment
marks and attendance at the practicals sessions in contrast to last year; and the students were given a
questionnaire to collect and analyse their views on pair programming. As this project was carried out in
second semester, the first year students had already completed one programming module in first semester
without using pair programming, so they will be able to compare and contrast the advantages and
disadvantages of both learning styles.
3.1 Statistical Analysis

In order to conduct the statistical analysis thoroughly, we firstly compared the attendance with that of the
same Algorithmic Programming II module in the academic year 2008-2009 to ascertain if attendance had
improved. In addition, to ensure that any improvement in attendance was not dependent on the specific
cohort, we analysis the attendance of a Mathematics module in semester one of academic year 2009-2010 to
obtain expected attendance profiles.
In order to obtain the expected results, the total possible attendance from the first semester's module was
obtained by multiplying the number of students by the number of weeks. Next the attendance every week of
the semester was summed and divided by the total possible attendance to obtain the attendance percentage.
This was carried out for the current and prior academic years. The comparison of these first semester
attendance records was then used against the previous year's Algorithmic Programming 2 module attendance
to achieve the expected attendance for the current Algorithmic Programming 2 module. The same method
was used to obtain expectations of practical assessment results. The benefit of using this method is that a
difference is quantified and applied to the same students in different years for attendance, assessments and
class tests. This way an amount of improvement or decline in the students without pairing is found, leaving the
48
difference in the analysis as the result of pairing; not ability.
(a)
(b)
Figure 1. (a) : Relation between the 2008-2009 attendance, the expected 2009-1020 attendance and the actual 2009-2010
attendance (a) Practical attendance; (b) Lecture attendance
Firstly analysing the change in attendance in comparison to the previous academic year, we found an
increase of 2.5% in practical attendance. Additionally, when we cross-correlated the second semester data
with first semester data for the same cohort we found that the attendance was significantly higher than
expected. In fact the overall practical attendance was 27.8% better than expected and the overall lecture
attendance was 10.35 better than expected as illustrated in Figure 1. This implies that working in pairs
encouraged students to attend in second semester, however as demonstrated by the graphs in Figure 1,
there is still room for significant improvement in the overall attendance.
Secondly we analysed the overall practical assessments for the module to determine if pair programming had
improved the practical assessment results. Analysis showed that the practical assessment marks had
increased by 31.2% in comparison to last year’s practical assessment marks and were 26.2% higher than the
expectation. These results are presented in Figure 2.
49
Figure 2. : Relation between the expected practical assessment marks and the actual marks
3.2 Student Feedback

At the end of the module, the students were issued with a questionnaire in order to obtain their views and
opinions on pair programming. They were asked to compare and contrast the pair programming technique
used in this project with the solo programming approach used in their first semester programming module.
This provided us with the student’s overall view of pair programming. It should be noted that the students
were allowed to select their own partners in order to eliminate any major conflicts between each pair
throughout the module. In total, 21 students completed the questionnaire. The questions and the percentage
of students that answered ‘yes’ to each question is summarised in Table 1.
Question % response of ‘yes’
Do you enjoy working in pairs? 76.2

Would you have preferred to work on your own? 28.6
Would you have preferred to work in larger groups? 28.6
Do you feel working in pairs has helped your 57.1
understanding of programming?
Do you feel you have learnt from your programming 57.1
partner?
Do you feel your programming partner has learnt 42.9
from you?
Did you and your partner work together on 47.6
programming practicals outside the assigned time?
Table 1. Summary of student feedback
As illustrated in Table 1, 76.2% of the student cohort enjoyed working in pairs. The general feedback
suggests that knowledge transfer within the partnerships did occur. The evidence also indicates that the
students prefer working in pairs rather than larger groups which could potential have impact on many other
modules that incorporate group work as means of assessment.
4. SUMMARY AND FURTHER WORK

The analysis to date implies that the pair programming has enhanced student attendance and practical
assessment performance and furthermore the students have enjoyed learning and working in pairs. To date
the analysis is only preliminary as the exam component of the module has not yet been completed. After the
exam, additional statistical analysis will be carried out to ascertain if the overall exam performance has
50
improved in comparison to the previous year, thus providing evidence that pair-programming leads to deeper
learning and understanding. However, this will also have to be cross correlated with other module marks to
determine if an overall improvement is unique only to the Algorithmic Programming module, or if indeed the
student cohort is academically superior to the previous year.
Verbal feedback from some of the students implied that they felt that one student always helped the other to
gain knowledge but this was rarely a two way partnership. This is an outcome of the students being allowed to
select their own partners for the pair programming. Therefore in the academic year 2010-2011, pair
programming will form part of the module assessment. However, the pairing will be pre-allocated and
dependent on the student’s academic ability.
5. REFERENCES
[1] Cockburn A., Williams, L., “The Costs and Benefits of Pair Programming” The XP Series, Extreme
Programming Examined, pp. 223-243, 2001
[2] Sanders, D., “Student Perceptions of the Suitability of Extreme and Pair Programming” Computer Science
Education, 2003
[3] Arisholm, E., et al., “Evaluating Pair Programming with Respect to System Complexity and Programmer
Expertise” IEEE Trans on Software Engineering, Vol. 33, No. 2, Feb 2007.
[4] Braught, G., Eby, L.M., Wahls, T., “The Effects of Pair-Programming on Individual Programming Skill”
SIGCSE, March 2008.
[5] Dyba, T., et al., “Are Two Heads Better than One? On the Effectiveness of Pair Programming” IEEE
Software, Vol. 24, No. 6, pp12-15, Dec 2007.
[6] McDowell, C., et al., “Pair Programming Improves Student Retention, Confidence and Program Quality”
Communications of the ACM, Vol. 49, No. 8, August 2006.
[7] Simon, B., Hanks, B., “First-Year Students’ Impressions of Pair Programming in CS1” ACM Journal on
Educational Resources in Computing, Vol. 4, No. 7, Jan 2008.
51
TEACHING MEDIA AND INFORMATION LITERACY TO
POSTGRADUATE RESEARCHERS
Andrew Whitworth Steve McIndoe Clare Whitworth
School of Education John Rylands University Library CWA
University of Manchester University of Manchester 41 Nutclough, Hebden
Oxford Road, Manchester, M13 9PL Oxford Road, Manchester, M13 9PL Bridge, W Yorks, HX7 8HA
andrew.whitworth@manchester.ac.uk steven.mcindoe@manchester.ac.uk clare.whitworth@manchester.ac.uk
http://83.170.93.38/~unimanc/
ABSTRACT
This paper reports on the early stages of a project funded by the HEA-ICS, which aims to create an open
educational resource to help develop media and information literacy skills in postgraduate research students.
The resource uses a distinctive and holistic approach to the teaching of these subjects, which brings together
Bruce et al’s ‘Six Frames of Information Literacy’ model (2007), Egan’s division of literacy into conventional,
emergent and comprehensive (1990), and Whitworth’s objective, subjective and intersubjective domains of
value (2009b). We describe here the structure and methods of the course, and look forward to the evaluation
which will take place over summer 2010.
Keywords
media literacy, information literacy, postgraduate research, open educational resource, holistic.
1. INTRODUCTION
This paper reports on the early stages of a University of Manchester project funded by the HEA-ICS. The
objective is to create a resource to help develop media and information literacy skills in postgraduate research
(PGR) students. We first describe the project background and the motivations behind it. We then explore the
distinctive model of media and information literacy (hereafter, M&IL) education in use, before reviewing how
this model has been translated into practice within the constraints of available resources.
Piloting of the course is taking place through May, June and July 2010, but at the time of writing (May 2010)
results of the pilot are not available, though they will be presented at the conference in August. However, we
outline at the end the questions we need to ask about this pilot.
2. BACKGROUND: EMBEDDING M&IL

Information literacy is recognised as fundamental to the health of the information society. UNESCO endorse it
as a basic human right, essential to national development, active citizenship and economic well-being as well
as educational standards (Catts and Lau 2008, pp. 9-11). Its importance is due to a recognition that digital
tools cannot be “readily… assimilated to existing practices of representation and communication… they are
fundamentally changing what it means to communicate, make meaning, think, work and learn” (Beetham et al
2009, p. 14). Hence, traditional notions of literacy must now be enhanced by an additional understanding of
the ways technology changes how we absorb information from the environment, then use it to sustain our own
learning, and that of the communities, organisations and society of which we are part (Whitworth 2009a).
Although the 21st-century learner now exists within a world saturated by media and ICT - even in developing
countries, where there has been a recent, rapid penetration of mobile communications - OfCom’s Media
Literacy Audit (2008, cited in Beetham et al 2009, p. 23) found that “[e]nthusiastic take-up of new media by
young people was not necessarily accompanied by an understanding of how new media content is produced,
i.e. by a capacity to read it critically, or play a role in collaborative co-creation... Their confidence in using the
internet is similarly not complemented by critical thinking or appropriate care in use of web sites, potentially
exposing them to risks relating to unsuitable material or abuse of their personal information.”
52
These difficulties are compounded for postgraduate research (PGR) students. They must spend more of their
studies working alone, without the filtering that takes place in a classroom. They may often be older and/or
from countries in which either the Internet infrastructure is less developed, censored or otherwise not freely
available, or whose education systems, largely for cultural-historical reasons, may not emphasise critical
thinking. Many are returning to education after a long gap. In any case (Beetham et al 2009, p. 24, citing
JISC/British Library, 2008): “research-behaviour traits that are commonly associated with younger users –
impatience in search and navigation, and zero tolerance for any delay in satisfying their information needs –
are now the norm for all age-groups”.
Despite the recognised importance of M&IL, it has not proven easy to embed it within UK HE institutions,
either at undergraduate or postgraduate level. There are two main reasons for this: limited definitions of IL
(see §3); and difficulties in locating M&IL institutionally.
Beetham et al’s report (2009) developed from the project, Learning Literacies for a Digital Age (LliDA), which
undertook an audit of practice across 15 institutions and also included insights from around 40 nominated
exemplars. Though there were many positive things drawn out by their review, they also identified (p. 63)
much scattered, inconsistent provision. Where practice was strong in certain departments or central services
(like libraries), these were “silos”, which found it difficult to disseminate their good practice, and/or make their
work relevant to different contexts. There are also disputes over the ownership of the agenda. IL (here
omitting the ‘M’) has “been strongly influenced by the idea that it is the province of librarians” (Whitworth
2009a, p. 98). Yet this view does little to connect IL with other literacies such as media literacy; nor does it
easily make connections between the library’s mandate to offer support to all disciplines, and the needs of
students who must work within disciplinary traditions and meet expectations that differ between subject areas.
Also, there is the wide variety of individual research projects and theses to contend with.
Moving from undergraduate study into PGR work has already been recognised as a difficult transition in terms
of information skills (JISC/British Library 2008, cited in Beetham et al 2009, p. 26). Policies vary across
institutions departments as to how PhD students are trained. Could M&IL classes be made compulsory?
Could upgrading be made conditional on performance in a skills unit that may be run in a library: if not, does
the training lack credibility? The supervisor serves as the personal contact between the student and the
expectations of an institution and/or a discipline, but there may be little integration between what a library or
exemplary department can offer, and the professional teaching development of research mentors, as well as
the academic development of their students. We address these questions of integration further below, in §4.3.
3. THE HOLISTIC MODEL OF INFORMATION LITERACY

There is not the space in this paper to discuss this model in detail. Detail will be found mainly in Whitworth,
McIndoe and Fishwick (forthcoming), Whitworth (2009b) and, in a more general way, Whitworth (2009a).
Nor is there scope to conduct a full analysis of all the many national and international models for M&IL
education, as outlined by Beetham et al (2009, pp. 30-38). However, it is possible to make some criticisms of
the general form of M&IL provision. Egan (1990) categorised literacy in three ways: conventional, emergent
and comprehensive, and it is Whitworth’s argument that these categories can also be applied to contemporary
multiliteracies such as information, media or environmental literacy (Whitworth 2009b, pp. 32-4). Broadly:
• A conventional view of any literacy treats it as skills which can be defined objectively, and then applied to all
situations consistently. The skills are quantifiable and measurable, and the general policy objective is to
increase literacy rates.
• An emergent view of any literacy treats it as personal and subjective, applied by the learner in local contexts
to meet specific needs. Skills are developed through a reflective understanding of these needs and differ
from person to person, and the general policy objective is to increase individual appreciation of the benefits
of literacy.
• A comprehensive view of any literacy sees it as intersubjective, recognising that it can be defined, and thus
wielded as a political tool, to exclude certain groups from informational spaces, and hence be a source of
power. Skills are developed through negotiation, critique and action, and the general policy objective is to
transform practice and challenge the status quo.
Whitworth (2009b, p.28) illustrates the relationship between these three perspectives with the diagram
reproduced as Figure 1. This also connects the model to the “six frames of information literacy” proposed by
Bruce et al (2007). Bruce et al derived the six frames model from research which showed the wide range of
approaches to information and IL education that were in use by their respondents. The five frames mentioned
in the diagram - content, competency, learning to learn, personal relevance and social impact - represent a
progression from functional skills (where to look for information, how to find it effectively), through more
53
individual and reflective approaches, to a critical, comprehensive view in which IL constitutes the skills needed
to transform the environments of communities. It is important to note that these latter skills depend on, and do
not supersede, functional, conventional ones. Yet the model also suggests that it is right to criticise the
tendency, also noted by Beetham et al (2009, p. 63), to reify and privilege the “skills agenda” and separate it
from “learning and living”. The three domains are interdependent. All three are essential to the development of
fully information literate learners, cognisant not just of how to find information, but how to evaluate it and then
produce and reproduce it, so that informational resources remain open, dynamic, accessible and of good
quality. Whitworth (2009b) shows how omitting any one of these three domains of value from the evaluation of
information will lead to pathologies, which may include counterknowledge (belief systems, which may be
widespread, but which have no scientific basis: Thompson 2008), relativism, or groupthink.
Figure One: Domains of value and approaches to literacy

Hence the existence of a sixth frame of IL, the relational frame, where the aim is to “explore variation” (Bruce
et al 2007, p. 51) in the ways students engage with information. The aim is not to privilege one frame over the
others, but familiarise students with all frames, empowering them to select between techniques as appropriate
to given - and dynamically changing - situations and contexts, to evaluate their needs and performance
against a range of criteria that include not just subjective but objective and intersubjective values as well.
Teaching in the relational frame thus requires a holistic approach (Whitworth 2009b); it is this approach which
has driven the development of the M&IL resource for PGR students.
4. THE PROJECT
The School of Education in Manchester has run the course unit Media and Information Literacy since 2008 as
part of the MA: Digital Technologies, Communication and Education (www.MAdigitaltechnologies.com). This
unit has been designed around the holistic model (Whitworth 2009b; Whitworth et al forthcoming). It was cited
in the LliDA report as an exemplar of IL teaching, and the only exemplar to address media literacy alongside
IL (Beetham et al 2009, p. 42). Two of the authors of this paper have worked on the unit since it began, A,
Whitworth as the course leader and C. Whitworth as a teaching and development assistant.
In 2009, McIndoe began work on a JRUL project aimed at developing and supporting students’ information
skills at all levels within the University of Manchester, UG, PGT and PGR. As an early step in this work an
audit was conducted of extant practices across the university. This enquiry made the JRUL and the existing
M&IL team aware of one another’s active interest in this sphere.
The winning of funding from the HEA then catalysed a collaboration to convert the existing M&IL materials for
use by a generic PGR audience. The course as it stood was too complex to be useful in this context. It was a
full-semester course with an involved assessment task (compiling a portfolio of M&IL teaching activities),
aimed at students who would go on to teach M&IL skills to others, in a variety of educational settings. Hence,
the first practical objective of this project was to create a pilot version by significantly simplifying the course for
a generic PGR student audience. Materials were converted in March and April 2010. A selection of M&IL
practitioners has been approached to act as academic reviewers of the pilot version, in addition to PGR
students from a range of disciplines. At the time of writing (mid-May 2010) this review process is under way
and scheduled to be complete by late June 2010. We will then analyse the pilot in the following ways:
54
* through e-mail feedback and follow-ups with academic reviewers (who for reasons of anonymity we should
not name directly, but who include some authors whose work has already been cited in this paper);
* through a focus group conducted with the PGR student reviewers;
* through analysis of Moodle log files, checking records of reviewers’ actual as well as reported engagement
with the materials.
This analysis will feed into the final version of the resource and the guidebook (see §5.3). The resource and
guidebook will be made available as an open educational resource.
5. THE HOLISTIC MODEL IN PRACTICE
5.1 Syllabus
Applying the holistic model to the needs of PGR students has led us away from a focus on technology and
information searching skills. Certainly these form part of the M&IL resource, but in accordance with the model
illustrated in Figure 1, the skills are located in a wider setting. The general theme is the role of research, and
the researcher, in a media- and ICT-saturated environment.
The structure and content of the resource are as follows:
• Introduction: why six frames of IL?
• Content frame: What is research? The role of information. Where to look. Citation guidance.
• Competency frame: Information literacy procedures and standards. Credibility of information.
• Learning to learn frame: How information becomes knowledge. How our tendency to seek patterns and
support for existing beliefs can prevent us learning.
• Personal relevance frame: Reflecting on information searching. Scientific method and counterknowledge.
• Social impact frame: Bad science (Goldacre 2009), and other examples of how the media use and abuse
information and research. Media literacy. The information commons, intellectual property and enclosure
(Hess and Ostrom 2007).
• Conclusion: objective, subjective and intersubjective value, and why each one matters.
Remember that the sixth, relational frame represents the course viewed as a whole; the aim being to have
students move between the other frames as appropriate.
In designing the resource we have been mindful of the need to address employability as much as students’
academic skills. Employability is part of institutions’ strategic vision for student and researcher development,
but nevertheless is rarely addressed in most IL schemes (Beetham et al p. 42). Yet only a limited number of
PGR students actually enter research as a profession following graduation, and by no means all of these do
so in HE (Jeffrey 2010) - and those that stay in HE may not all become researchers. Whether we have
successfully addressed this wider context will not be confirmed until results from the evaluation are in.
5.2 Pedagogical methods
The resource has been created on Moodle, and the pilot is at http://83.170.93.38/~unimanc/. The intention is
that it can be completed in around 7-10 hours of work, whether in one go, or smaller chunks corresponding to
the seven topics listed above, spread over a few weeks. Students will follow the resource individually. Each
topic contains two audio files, averaging 5-6 minutes in length, which are the principal content delivery
method: there are also some optional readings, additional web-based resources, and three activities which
test participants’ understanding. Note that at this point, the question of how students’ work on this resource
will be accredited has not been resolved, even at Manchester: it is almost certain this will vary from institution
to institution, and in any case cannot be properly established until after the pilot phase has been completed.
Some justification is required. The project brief required us to create a resource which stood alone and was
not part of any wider programme of teaching. Hence there are at present no collaborative spaces such as
discussion boards or wikis included in the resource. Instead, the emphasis is on content, which may seem to
give it a didactic feel. Yet once adopted by other institutions (see §5.3) it may be that students are guided by
an instructor, supervisor, or a community of researchers: it may be that students will continue to work on the
resource independently. The resource should be viewed not as a finished course of teaching but as a base on
which may be overlaid other activities, forms of interaction, and so on. Ultimately it should be embedded in
other research support and development activities, such as group seminars or research methods courses
which are discipline-specific and more oriented towards developing communities of researchers.
VLEs such as Moodle have also been criticised as an “undead” technology (University of Wolverhampton
2009): that is, one which has expired as a useful technology but which still continues to exhibit some lifelike
55
characteristics. There is certainly scope to use other, more personalisable and/or collaborative technologies,
such as e-portfolios or blogs, in developing research skills. The audio files could be disseminated as podcasts
via iTunes, with supporting downloads for non-audio materials. Again we stress that these are decisions that
can eventually be made by adopting institutions. The project specifications allowed neither the time nor other
resources to innovate pedagogically, and materials were created in Moodle because this was the VLE in use
in the original M&IL taught course unit.
5.3 Integration
For both reasons outlined in §5.2 there is a need to release the materials with accompanying guidance on
how they can best be integrated into the research development work already under way in other locations.
This connects with a general need for universities to consider how they integrate IL into other aspects of their
teaching and research work. Seeing IL as the responsibility of the library can easily lead to a dislocation here.
Including some kind of IL awareness training in the induction of new academic staff can help, but continuous
support is more helpful: the problem is that such CPD work is usually voluntary except for new academics
(Beetham et al 2009, p. 55). Simple provision of new materials is very unlikely to lead to their adoption, even if
they are accompanied by guidance on integration.
An interesting potential solution to this problem has been tried by Vedvik Tonning et al (2010). Their paper
reviewed efforts in a Norwegian university, which brought central pedagogical support services together with
teaching staff to help them integrate IL into new and existing courses they were developing. In a similar way,
then, it may be that the M&IL course could be included both in teacher training for new academics and
ongoing support for supervisors, though once again there is an issue with compulsion. This can only be
resolved in the adopting institution or department, however.
5.4 Evaluation and follow-up
Researching the manner in which this resource is eventually adopted, whether at Manchester or elsewhere,
will be an interesting follow-up project. Will it appeal more to some disciplines than others? How will it be
adapted, and what changes will be made?
The evaluation of the pilot will also need to measure the impact of the resource on students’ behaviour,
though with such a holistic approach, such measures will not all be quantitative and/or easily seen. The focus
group. and perhaps follow-up interviews after a few months, could use questions similar to those used by
Whitworth et al (forthcoming) in their evaluation of the original M&IL course: students have been asked before
the pilot started what their views are of information literacy, and we will check whether these have changed
and expanded after completing the pilot. This aspect of the evaluation will also inform the guidebook, as it will
help other institutions determine what role the resource can play in their own information literacy programmes
for PGR students (and possibly PGT).
6. CONCLUSION
Information literacy is important; that much is broadly accepted. But there remains a risk that it will turn into
another functional skill, with students’ success measured only against quantitative criteria such as how quickly
they can find information and from what (approved?) sources. Such a view will turn students into mere
information-processing machines. But the research student must be creative, autonomous and individualistic.
IL for PGR students must therefore pay respect to the particular nature of research work. This is the approach
we have taken in developing this resource with the help of the HEA-iCS.
7. REFERENCES
Beetham, H., L. McGill and A.Littlejohn (2009), Thriving in the 21st century: Learning Literacies for the Digital
Age, Glasgow Caledonian University, Glasgow.
Bruce, C., S. Edwards and M. Lupton (2007), Six frames for information literacy education: a conceptual
framework for interpreting the relationship between theory and practice. In Andretta, S., ed., Change and
Challenge: Information literacy for the 21st century, Adelaide: Auslib (pp. 37-58).
Catts, R. and Lau, J. (2008) Towards Information Literacy Indicators, UNESCO, Paris.
Egan, K. (1990). Romantic Understanding: The Development of Rationality and Imagination, Ages 8 – 15.
London: Routledge.
Goldacre, B. (2009), Bad Science, Harper, London.
Hess, C. and Ostrom, E., eds. 2007. Understanding Knowledge as a Commons. Boston, MA: MIT Press.
Jeffrey, L. (2010), Generic versus subject-specific researcher training, paper presented at LILAC 2010,
Limerick, Ireland.
JISC/British Library (2008), Google Generation
http://www.jisc.ac.uk/whatwedo/programmes/resourcediscovery/googlegen.aspx
56
OfCom (2009), Report of the Digital Britain Media Literacy Working Group,
http://www.ofcom.org.uk/advice/media_literacy/media_lit_digital_britain/
Thompson, D. (2008). Counterknowledge. London: Atlantic.
University of Wolverhampton (2009), The VLE is Undead, at
http://www.learninglab.org.uk/eventdetails.php?Plv=&Pid1=&Pid2=&id=15.
Vedvik Tonning, A. S., T. Rullestad and T. Skagen. 2010. Integrating information literacy within the university
curriculum: Cooperation between University of Bergen Library and the Centre for University Pedagogy. Paper
presented at LILAC 2010. Limerick, Ireland: Librarians’ Information Literacy Annual Conference.
Whitworth, A. (2009a): Information Obesity, Chandos, Oxford.
Whitworth, A. (2009b): Teaching information literacy within the relational frame: the Media and Information
Literacy course at Manchester, Journal of Information Literacy 3/2.
Whitworth, A., Fishwick, I. and McIndoe, S. (forthcoming): Teaching information literacy in a holistic way, in
Teaching Information Literacy Online, ed. T. Mackey and T. Jacobson, Neal-Schuman, New York.
57
ANYTIME-ANYWHERE STUDENT SUPPORT THROUGH
VIRTUAL MENTORS
J. C. Augusto, P. McCullagh, A. McRoberts, V. McNair

University of Ulster
Jordanstown campus, UK
{jc.augusto, pj.mccullagh, a.mcroberts, v.mcnair }@ulster.ac.uk
ABSTRACT
Higher Education institutions in the UK rely on mentoring programs as one of the ways to support students at
different stages of their training. The reality is that, good as this resource could be, it is not available to
students all the time. Students can contact their advisers, and support resources like Career Development
Centres, at scheduled times, but sometimes the problems motivating the students to seek advice may require
an immediate answer. Anytime-anywhere student support is an emerging area of development as indicated
by its inclusion in a range of University policies. This article proposes a holistic approach which blends
emerging technologies into a ‘Virtual Mentor’. This paper provides the motivation for our work, technical
details of the software system, and initial validation exercises.
Keywords
Student support, virtual, mentoring, avatar front-end
1. INTRODUCTION
Higher Education institutions in the UK rely on mentoring programs as one of the ways to support students at
different stages of their training. This advice provides students with orientation on how to develop their talents
and also (probably more often) to help them cope with difficulties related to the student’s academic
experience or even sometimes to their social context (family and personal issues). The academic support
provided by mentors is very valuable as students, particularly in their early years of university education, face
a wide range of problems which can limit the quality of their experience, inhibit learning and ultimately
influence their career direction (National Audit Office (2002)). Sometimes these problems influence the
student to interrupt or even leave education. Finding additional ways to support students facilitates retention
and is incumbent upon universities (Wilcox et al. (2005), and Thomas (2002)).The reality is that, good as this
resource could be, it is not available to students all the time, and has to be balanced within an ever increasing
academic workload. Students can meet their academic advisers at scheduled times, but sometimes the
problems motivating the students to seek advice may require an immediate answer. Other services like
Student Support and Career Development Centres have similar accessibility restrictions. Documents may be
available through internet access, but they are passive repositories of information and students are used to
interactive ways of engagement.
Anytime-anywhere student support is a growing area of development as indicated by its inclusion in a range
of University policies and hence it has become an area of pedagogic research. There is a range of
technologies (e.g., podcasting of lectures, the development of reusable learning objects, and SMS reminders)
which can be applied to support teaching and learning, however they have limitations in providing students
with anytime-anywhere support. This article proposes a holistic approach which blends available technologies
into a ‘Virtual Mentor’ (VM). A VM is not intended to replace a human mentor but can offer support when a
human mentor is not available. The system under development involves an avatar at the front end which can
take questions from the students, interrogate a knowledge base, and either verbalise an answer, point the
student to a source of relevant information or redirect the query to a studies adviser. The approach is intended
to engage the student and enhance confidence in the support system. In a 2004 survey by Nottingham
University, the most popular words used to define engagement were: ‘interest’, ‘enjoyment’ and ‘interaction’
and the commonly cited constraints to effective engagement were the method or quality of teaching and a
lack of student confidence (Carlton et al. (2004).
© 2010 Higher Education Academy Subject Centre for Information and Computer Sciences
58
A Prototype has been developed using requirements derived through questionnaires given to students. This
paper provides the motivation for our work, technical details of the software system, and an analysis of the
answers collected from the students. Section 2 provides more details on the academic context motivating this
development, followed by a state-of-the-art supporting interfaces and systems; Section 3 explains the concept
of Virtual Mentoring system and Section 4 describes some preliminary assessments of the system. Section 5
provides an analysis of its implications.
2. PEDAGOGICAL MOTIVATIONS
The academic support provided by mentors is valuable as students, particularly in their early years of
university education, face a wide range of problems which can limit the quality of their experience, inhibit
learning and influence their career direction. Figure 1 highlights student experiences that may require extra
support from the academic environments. These important aspects of a student academic life cannot be
ignored as they contribute to student satisfaction.The University of Ulster’s policies on induction, transition
and retention, aim to enhance the experience of students in Higher Education, especially through the
transition in the first year. Courses make key information available (e.g., on their courses, evaluation
methods, etc.), provide extra support to new students (e.g. explaining the induction process and how to
participate in it) and offer students opportunities to meet with peers and members of staff. Other steps include
the reinforcement of learning skills, and promoting engagement of students with university life.
These support systems help close the gap between University expectations and student experiences. For
example, the STAR project (Student Transition And Retention) is aimed at increasing student retention,
particularly at the important transition point between school and university. The program found that among the
elements that impact on students entering University were social changes such as the way they interact with
students/teachers, changes in curriculum, study methodology and assessment. STAR proposed the following
measures to make students to feel more comfortable in the new system: induction as a process rather than as
an event; well managed group work, field work, social events; cohort identity (e.g., teaching in small groups);
mentoring to assist rapid integration; rites of passage; and more contact with fewer staff. For example, Cook
and Lowe [2009] show that meeting with studies advisers was highly important for students entering university
and played a role in their retention while Cuseo (2009) highlights the importance of out of class contact which
goes beyond didactic issues. Similarly, Levitz (2009a) and Levitz (2009b) highlight the importance of academic
advice and the provision for clarification of questions students may have. Findings suggest that the effort
placed on supporting students in their initial phases of Higher Education is appreciated by students.
Fig. 1 Progression of student related support elements
3. VM: A VIRTUAL MENTORING SYSTEM

Existing solutions rely on a human being available to read enquiries and process them either in real-time or
off-line. Our use of “Virtual Mentor” however refers to a fully automatic system that can autonomously interact
with the student and find ways to assist. The system and the student still have the option to forward a query to
a human mentor but in principle the VM will assess the problem and will provide a solution/advice on its own.
The VM development merges technology with students’ need. In Figure 2, the student (portrayed at the top)
can interact through a computer which will allow (optionally in a future prototype) an input through a webcam
and a microphone which will add to the VM’s perception. This input will be stored in databases (DB) from
which a software agent (Wooldridge (2009)) equipped with general reasoning and pedagogical skills will offer
personalised advice. This advice will be stored in internal databases which the avatar will use to communicate
with the student. The student can request that part of that interaction is passed to her/his tutor or may decide
to send an email to the tutor so a channel between the system and human mentor is also available.
59
Fig. 2 Basic architecture of the Virtual Mentor system
The VM utilizes specialized software (Media Semantics) that allows the definition of an avatar, linked to a
conversational database (accessed through Program O1), which has been updated with specialist information
of relevance to the student. PHP has been used an integrating technology. The intelligence of the system is
at the moment rather primitive and consists of a set of rules that define what types of queries it can process
and what are the answers associated with them. The quality of the service is basically dependent on the
coverage of those rules. For the time being that is enough to illustrate the functionality and potential of the
VM. The user interface is shown in Figure 3. Several features of the avatar can be personalized by the user.
In contrast the set of rules has to be maintained by the central development team and is the same for all
students accessing it.
Fig. 3 One of the possible (customizable) looks of the avatar

The VM uses an avatar Cassell et al. (2000)) to take advantage of spoken-dialogue technology (McTear
(2004)) but supported with a Knowledge Base that provides interesting answers to the users. The VM extends
“blended learning approaches” (Mentors Online (2009)) like those based in WebCT which have passive
recipients and rely on asynchronous communication. The intelligence of the approach depends again on
when the teacher is available. The VM is a complement to a Smart Classroom environment (Augusto (2009))
supporting the student on less technical topics. In our project the student can bring part of the learning
environment (mentoring element) with her/him wherever she/he is. The VM also differs from the mobile
platforms (Tavangarian (2006)) in the type of interaction sustained by presenting a natural interface and
1
http://www.program-o.com/
60
reasoning engine that allows the student to interact and obtain feedback and guidance f on a variety of
academic life related issues.
4. VALIDATION
Trials were conducted with students through seminars and questionnaires. We have been informed by
students at different stages on preferred aspects of a VM system. We used optional and anonymous
questionnaires which was effective to gather a larger number of opinions simultaneously and without
pressure. The questionnaire combined closed questions for some issues (e.g., whether a VM can be useful)
and open questions (in the form of free text response) which allowed students to participate with their opinion
on shaping the services they need.
The questionnaire was passed to two different cohorts of Mathematics and Computer Science students; one
was part of an undergraduate course and the other part of an MSc course. There were 24 responses from a
total of approximately 40 students. There was a diversity of the students’ ethnicity (approximately half of them
come from outside the UK) and gender (approximately 30% Female and 70% Male). The most important
issues obtained by the results of the questionnaire were: (a) that 2/3 of students faced situations where they
needed support but it was not available, (b) 3/4 of students reported that they believed a Virtual Mentoring
system would have helped in those situations, and (c) informed on a diversity of topics the students would like
the system can help them with: availability, socialization, support to prepare for assessment, easiness of
access, facilitation of contact with the human mentor, study advices, guide on preparation of assignments,
and general guidance. They provide a good indication that, at least for a cohort of students with a technical
background, there is a perceived potential usefulness associated with the type of system to be described in
the next section.
Another questionnaire was developed for Mentors seeking their opinion on the pedagogic use. The vision is
that mentors see the VM system as a complement to their role and an optional tool for supporting students.
The questionnaire provided enough flexibility for mentors to address topics which were characteristic of the
cohorts they were mentoring. The opinion of two course directors was requested as they are mentors with
significant experience in the role. Their comments were informative and complementary to those provided by
the students. Reasons they cited for students’ visits were: students who were struggling to perform as
expected or to make decisions due to a poor and/or uninformed previous choice, circumstances that result in
poor attendance, non-submission (or late submission) of coursework, nonattendance at exams, poor
academic performance, choice of modules, transfer to other courses either during or on completion of current
course, problems arising from team work. One mentor observed that at the root of those with the more severe
problems were: poor choices of course, work commitments, and the perception that attendance at lectures is
unnecessary. These surveys indicate that students and mentors are happy to receive and give, respectively, a
greater range of support strategies and highlighted a range of issues of interest which are being taken into
consideration for the system reported in this paper.
5. CONCLUSIONS
Human mentors are irreplaceable but they may not always be available when and where the student needs
help. A VM can assist students in some of those cases when a mentor is not available. Cases in which a VM
may be the preferred option are still to be fully explored. The extent of support provided depends on many
factors, including the type of technology deployed. This article focused on the initial stages of development
which involved the scoping of the technology, the early detection of problems (and potential “bottlenecks”) as
well as the identification of opportunities and potential areas of application.
There are several elements to be explored in this system and have been left untouched during this first
prototype. On one side there is the pedagogical dimension with a difference of approaches to this
technological opportunity, whilst students see it as another option some human mentors see it as a remedial
option that can be helpful when they cannot address specific problems. This can lead to a misuse of the
system and information has to be put in place to ensure all sides share a view on when and how the system is
to be used. Another important element to address at all stages of development of the successive prototypes is
privacy. The student has to be adequately empowered to decide what information can be shared outside the
virtual mentor, with whom and when. There is an enormous potential in this concept and different areas can
be progressively covered as the VM is enriched with a more sophisticated internal logic to support
pedagogical advice and more powerful sensing devices (e.g., through spoken dialogue and image
recognition) to understand more accurately the student’s needs.
61
Other challenges are related to the way the system engages with the students and how the advice that it
provides is consistent with the educational environment the student is immersed in. It will be interesting to
investigate: how successful is the communication of the student and the VM, how useful the advice of the VM
is perceived to be by the students, how the system can contribute to provide individual and global views of
specific students’ and cohort’s perceptions of a module and how that can lead to improvements in
pedagogical content delivery. These provide a very interesting intersection amongst pedagogical/social
elements and technology deployment which can be investigated by teams of the Social Sciences and
Engineering faculties of this university and also provide fruitful interaction amongst other different units of
universities across the UK.
A prototype is currently being extended, which will provide a platform for experimentation. This platform can
provide an interesting opportunity for the academic environment to increase the options to support students, a
valuable and flexible tool for students, and a stimulating challenge for researchers investigating the way
technology can be used to improve Teaching and Learning.
Acknowledgement
This project has been supported with funding from JISC TechDis HEAT Scheme Round 3 and a Development
Fund grant from the Centre for Higher Education Practice of the University of Ulster.
6. REFERENCES
J. Augusto. Ambient intelligence: Opportunities and consequences of its use in smart classrooms. Italics,
8(2):53–63, 2009.
J. Cassell, J. Sullivan, S. Prevost, and E. Churchill. Embodied Conversational Agents. MIT Press, 2000. A.
Cook and H. Lowe. Hitting the ground running: The role of induction in preparing students for university.
http://www.ulster.ac.uk/star/resources/hgr.pdf (last accessed in Jan. 2009).
M. Charlton, G. Langmack, J. Peirce, T. Sach. Perceptions of student engagement in a research-driven
university, 2204. http://www.nottingham.ac.uk/pesl/resources/participation/perceptx217/, accessed May 2010.
J. Cuseo. The importance of faculty-student contact outside the classroom, 2009.
http://www.ulster.ac.uk/star/resources/faculty\ student.pdf (last accessed in January 2009).
N. Levitz. Fundamental retention theory, 2009a. http://www.ulster.ac.uk/star/resources/Noellev1.pdf (last

accessed in January 2009).
N. Levitz. Other suggestions from noel levitz as provided in retention planning workshop, January 22-23,
2009b. www.ulster.ac.uk/star/resources/NoelLev2.pdf (last accessed in January 2009).
M. McTear. Spoken Dialogue Technology. Springer-Verlag, 2004. Mentors Online. Mentors online, 2009.
http://online-mentor.blogspot.com/ (last accessed in January 2009).
National Audit Office. Improving student achievement in english higher education, 2002. Report by the
Controller and Auditor General HC486 London, The Stationery Office.
D. Tavangarian. From mobile to pervasive learning, 2006. IEE: Informatics Education Europe, Montpellier,
France. www.ics.heacademy.ac.uk/education_europe/mobile_To_Pervasive_Tavangarian.pdf
L. Thomas. Student retention in higher education: the role of institutional habitus. Journal of Education Policy,
17(4):423–442, 2002. University of Bradford. Mentoring program, 2009.
http://www.brad.ac.uk/admin/conted/guidance/mentoring/ (last accesed on 2009).
P. Wilcox, S. Winn, and M. Fyvie-Gauld. It was nothing to do with the university, it was just the people’: the
role of social support in the first year experience of higher education. Studies in Higher Education, 30(6):707–
722, 2005.
M. Wooldridge. An Introduction to MultiAgent Systems. John Wiley & Sons; 2nd Edition, 2009.
62
TEACHING PROBLEM SOLVING AND AI WITH PAC-MAN
Jim Smith Steve Cayzer
Department of Computer Science Department of Computer Science
University of the West of England University of the West of England
Bristol, BS16 1QY, UK Bristol, BS16 1QY, UK
james.smith@uwe.ac.uk steve.cayzer@uwe.ac.uk
http://www.bit.uwe.ac.uk/~jsmith http://www.bit.uwe.ac.uk/~jsmith
ABSTRACT
We describe a series of resources and practical exercises supporting the teaching of introductory topics in
Artificial Intelligence using the metaphor of the well-known arcade game “Pac Man”. They are aimed at level
one students from a range of disciplines and motivated by a view of Artificial Intelligence as a means of
automating the problem solving process. The first set of exercises start with search strategies and gradually
build up via rule-based and expert-system approaches to create a Pac-Man player based on “traditional AI”.
Subsequent apply computational approaches such as artificial neural networks and evolutionary computation
provide a radically different approach to generating and improving controllers
Keywords
Artificial Intelligence, Problem Solving, Groupwork, Coding.
1. INTRODUCTION
Systems containing some kind of Artificial Intelligence (AI) form the state of the art in many applications of
computing, across the spectrum of social and economic activity, and the future is certain to contain more,
rather than less such systems. Moreover, since AI fundamentally concerns itself with problem solving, it
should provide an ideal medium to explore issues and provide generic transferable skills. Despite this, many
undergraduates view AI as a rather dry subject matter, a situation that is not helped by many of the learning
exercises to be found in textbooks and on-line coursework materials. Several authors have attempted to
address these using robotics-based tutorials (Kumar and Meeeden 1998, Karapetsas and Stamatis 2008,
Parsons and Sklar 2004) but these tend to be specific to certain aspects, and often require programming skills
beyond a typical level one student. More importantly, a series of informal conversations with current and
potential students revealed a view of anything using robots as somewhat “techie” and off-putting. Certainly
there is a concern that they can reinforce a view of AI as primarily relevant to robotics rather than to all
disciplines within computing. A second problem facing the would-be teacher of a general course on AI is that
books are often specialised, and/or often use different examples, and different software to illustrate and teach
different types of AI. This raises issues where either students face the overheads of learning many different
software packages, or are required to code their own algorithms, which risks turning AI classes into coding
tutorials. This analysis lead to the identification of the following “wish-list” to which any new set of activities
should adhere:
 Use of a “metaphor” which is well-known and liked across a range of backgrounds;
 Emphasis on problem-solving skills that can inform and assist other topics.
 Use of a common programming environment, supporting tutor-provided scripts / functions to create a
“naturalistic” style of programming with as low a learning overhead as possible;
 Coverage of the syllabus elements of search strategies, rule-based systems, expert-systems, artificial
neural networks, evolutionary computation and swarm intelligence (multi-agent systems);
 Use of a free, lightweight, and simple to install software environment, preferably one that also contains or
supports a range of other examples to stimulate the more able students.
In this paper, we describe how this analysis lead us to design a series of practical exercises using the
metaphor of the well-known “Pac Man” game, and implemented in the Netlogo environment (Wilensky 1999).
63
2. BACKGROUND AND CONTEXT
These materials were developed for the UWE level one module “Introduction to Artificial Intelligence”. This
module is compulsory on several degrees such as the BSc’s in Computer Science, Robotics, and Games
Technology, and optional on several others. The syllabus of the module is fairly conventional. In the first
semester the students briefly cover “philosophical issues” such as Turing’s test, and Searle’s Chinese Room
problem, but primarily as a means of focussing on what “AI in practice” can or might involve, in opposition to
what Hollywood might portray. From there a number of scenarios (driving a car unaided, acting as a tour
guide, playing table tennis) are used to highlight how “AI” behaviour can be broken down into groups of
related tasks, and the “input-model-output” paradigm of computing is used to group these and introduce the
idea of all learning as a search through a space of possible inputs (or sequences), models, or outputs. The
formal lectures then move on to cover blind and informed search strategies as means of automating search
(depth-/breadth-/best- first, hill-climbing, A*) then knowledge representation: moving through first order logic
and rule-base systems, via expert systems to the Semantic Web.
Semester two starts with by recapping the idea of learning as search, the three principle classes of search
space and some examples of relevant Machine Learning technologies, and then covers three areas in more
depth. The first is Artificial Neural Networks: perceptrons and simple MLPs with back-propagation as a hill-
climbing search method. The second is Evolutionary Computation as a general-purpose search method,
illustrated by Genetic Algorithms for optimisation and Genetic Programming for model building. Finally the
cellular Genetic Algorithm is recast as an emergent process and used to introduce the topic of “swarm
intelligence” illustrated by optimisation (Ant Colony Algorithms), model building (ant-based clustering) and
simulation (“Boids” (Reynolds 1987).
The majority of the activities designed to support this learning had the students working in groups of three or
four. Typically, after the task was introduced, everyone would be asked to think about it individually for five
minutes before the group activities commenced, and at the end of the session a “wrap-up” discussion would
see the groups working together either in competition or collectively to come up with ideas how this work
could be used to solve some larger task.
The Netlogo environment is an open-source multi-platform package for developing and running simulations of
multi-agent systems. Based on a variant of the logo programming language, it combines a simple interpreted
scripting tool with an interactive interface for running, pausing and manipulating scripts. Scripts can also be
saved as applets embedded in a web page containing the instructions. Netlogo has been used for both
research and teaching in a range of disciplines covering the physical sciences, sociology, economics etc. A
screenshot is shown in Figure 1. In addition to the very well written, user-friendly environment, Netlogo has
the advantage of a small language and syntax, and great extensibility. Thus the students only need to learn
how to use Boolean constructs (which end in “?”), and the syntax of if, ifelse, while and foreach. By writing
suitably named procedures the tutors can hide all the rest of the detailed syntax and coding niceties such as
list operations, observer functions, and exception handling.
Pac-Man is a classic arcade game in which the player steers a character (the pacman) around a maze eating
pellets of food and trying to avoid a number of mobile “ghosts” which attempt to eat the pacman – reducing
the number of “lives” left. As extra complications “power pellets” temporarily change the appearance of the
ghosts, rendering them edible to the pacman. While conceptually simple, the game involves search (finding
the valid paths to uneaten pellets), classification (recognizing the edibility of ghosts), tasks hierarchies
(avoiding being eaten) and planning (e.g. not eating the “power pellet” if no ghosts are within range).
3. TOPIC-SPECIFIC EXERCISES
3.1 Search Methods
In order to accustom the students to the idea of blind search, how people use memory and clues, and how
making assumptions can help search, the first tutorial takes the form of an interactive applet which presents
the students with an initially blank grid and asks them to manoeuvre the pacman to find a gold square. The
students are encouraged to write down the strategies they are using, and an on-screen display of the number
of moves taken forms a natural basis for competition and comparing strategies. One powerful teaching tool
is the ability to toggle the maze’s visibility, either by displaying a limited ‘window’ around the pacman, or by
simply displaying the whole maze at all times; the latter allowing strategies which employ ‘look ahead’.
Another equally important parameter is memory. This is achieved by colouring visited cells green, and hence
facilitating movements that prioritise unexplored directions. The students were asked to formalise how these
factors affected their planning strategy, and to document their “solution” online sufficiently unambiguously that
another group could follow it.
64
The role of the tutors in these sessions is critical to it becoming more than just a play session. We used two
or three tutors per session, who moved between the groups asking questions, prompting and commenting.
The other factor critical to success is the use of a twenty-minute “wrap-up” session with plenty of student
interaction. During this stage a number of common points should be drawn out and then related back to the
more formal language of the lectures. As mentioned above, this module is core to several different degrees,
so in addition to the obvious points about memory and the value of heuristics (e.g. most students do “turn
towards the middle”) the exercise could facilitate discussion tailored to different degrees – e.g. global vs. local
models for Robotics students, storage and run-time analysis for Computer Science students etc. At the end
the students are asked
In the second session the students are asked to modify the scripts from the first tutorial to implement either
depth-first of bread-first search. The scripts are heavily commented to show where the new automated code
should go. They are provided with examples of “if”, “if-else” and nested versions of these, to aid them with
syntax. Most importantly, they are given a set of primitive constructs from which to build their code. These
include Boolean tests (e.g. “can-turn-left?”, “wall-ahead?”) and functions to turn the pacman (“turn-left”, “go-
ahead”, “turn-around”, “turn-north”,…). The surrounding code deals with the movement of the pacman, so
their code only needs to decide which way the pacman should face at each timestep.
...
if need-to-pick-another-pacman?
[
set current-pacman-id get-oldest-pacman
;;set current-pacman-id get-youngest-pacman
;;set current-pacman-id get-closest-of-all-pacmen
;;set current-pacman-id get-furthest-of-all-pacmen
;;set back-tracking-allowed? FALSE
;;set current-pacman-id get-closest-child-of-last-
parent
;;set current-pacman-id get-oldest-child-of-last-
parent
;;set current-pacman-id get-youngest-child-of-last-
parent
update-parent-id
]
Figure 1: Left: Screenshot of Breadth-first search. Current pacman is coloured yellow. Two other
pacmen at same depth are white, three at the next depth is red. Some pacmen are slightly overlaid.
Right: corresponding code fragment allowing students to create different search strategies.
Most students correctly identified their previous wall-following strategy as a version of depth-first chose to
implement that, and recognised that they should use relative rather than global co-ordinate systems. They
were given various mazes so that the more able students could think about how to modify their code to cope
with recursive or looping mazes. Most groups successfully built the controller to navigate the loop-free maze.
In session three they are then shown a new Netlogo script in which, rather than a single pacman moving
around the web, the current pacman moves along the maze to the next junction whereupon it spawns one
child facing along each unexplored path (if there are any) and then dies, whereupon a decision has to be
made which of the extant pacmen to move next. Each pacman contains variables recording their “depth” and
the initial “distance_to_goal”. This latter records the Euclidean distance to the goal, which is of course not
necessarily the “path” distance but it is at least an optimistic heuristic, which is a prerequisite for the optimality
of A* search. Manhattan distance is another alternative that can be used. All these details can be hidden from
the students but are available for discussion if appropriate. Figure 1 shows a screen shot of breadth-first
search with this model, and the relevant code fragment. The learning task is to apply the conceptual ideas of
search to this particular scenario and thus create implementations of depth/breadth and best-first search, Hill-
Climbing and A*. A number of mazes are provided, and an informal competition between the groups
65
encourages experimentation. At the end of these sessions all groups had working implementations of
procedures that could be built into a final pacman controller to provide different search mechanisms, and
some understanding of the characteristics of each search method.
3.2 Knowledge Representation

The decision to ask the students to build a controller that decided which way to face at each timestep, while
hiding the looping and movement phases, allowed a deliberately focus on conditional logic. Once the lectures
move on to consider knowledge representation, rule-based and expert systems this is further extended in a
series of exercises which build on this rather stylised form of propositional logic.
In the first exercise the students are asked to create and save new ghost shapes in netlogo’s “turtle shape
editor”, then to create some classification rules that will assign this ghost shape to the class “edible” or
“inedible” based on visible characteristics. They are asked to add the ghost details into a “world map” - a
spreadsheet that contains maze and ghosts characteristics. They need to instantiate new ghosts, label them
as edible or inedible and finally create new attributes describing their ghosts. The students are also provided
with a script, which loads the map and calls a stub function that classifies each ghost in turn. The primary
learning task is thus to write the code that correctly classifies all the ghosts. Depending on how the ghosts are
labelled, and the number of attributes used in the classification rules, this activity provides scope for
discussion of rule-hierarchies, default rules, disjoint and conjoint logic. Of course there are many equivalent
ways of doing this, providing scope for a “wrap-up” session where the different versions produced by students
are shown to be formally equivalent.
The penultimate set of tutorials focussed on the specification, creation and maintenance of expert systems.
Using the maze from the previous tutorial, the students were asked to devise a set of rules to control the Pac-
man using a range of (supplied) primitives. Choice functions include returning a set of directions to test and
condition clauses include simple tests (clear, safe), ones that involve lookahead (ghost ahead), classification
(edible ghost ahead), heuristics (nearer to edible ghost) and memory (unexplored). Higher level planning for
the more advanced students was provided through the use of route planning (choose next edible ghost). The
rule production was done as a “pyramiding” activity – each member of the group was asked to come up with
single rule describing a particular situation, and then the group collectively refined the combined rule set and
implemented it – iterating as necessary to achieve the desired behaviour. The groups were free to use the
procedures provided, or any other ones they had developed or used previously.
In the final practical of this section, the students were invited to put their classification, search and expert
system work together in a series of provided mazes, with challenges such as dead ends and moving ghosts.
Supporting material has been made available for motivated students to extend this work into the full pacman
game with pellets and power pills. As hoped, this activity provided a valuable experience in the “naturalness”
of expert systems, but also of the difficulties of ensuring correctness and maintaining the rule bases. It also
showed how increasing levels of knowledge could be built into rules that deal with environments at a range of
complexities. The classification, expert system and route planning activities also showed how AI techniques
could be employed at different levels of abstraction.
3.3 Artificial Neural Networks

The lectures for this topic cover the use of perceptrons as a stylized representation of the action of biological
neurons, Hebbian learning, and the strengths and weaknesses of single perceptrons – e.g. the XOR problem.
They move on to cover simple multi-layer perceptrons (MLPs) and back-propagation. Among the set of
models that come provided with Netlogo are a simple perceptron and a three-input, two-hidden node MLP.
These come with train/test routines for learning and testing simple functions such as or/and/xor. A particularly
nice feature is that the values of the weights are shown graphically by varying the width of the link between
nodes. Next, using training and test sets built from variations on the set of four ghosts from Figure 4, students
learn how these formal functions relate to practical problems by showing that neural networks are capable of
learning rules that correctly classify ghosts. A range of exercises, and careful placement of (static) ghosts on
mazes is used to illustrate issues such as over-fitting, problems of feature selection/sparse datasets, etc.
3.4 Evolutionary Computation

Like many agent-based systems, netlogo comes complete with a model of a simple genetic algorithm using
one-point crossover, bitwise mutation and fitness-proportionate selection to evolve a solution to the binary
“OneMax” problem, where the fitness is simply the number of bits in a string. The students begin with a
structured series of exercises, using sliders in the interaction tab to explore the effect of changing population
size, selection pressure, mutation and crossover rates so as to illuminate the effect and value of these
operators and settings. The second exercise uses a fitness function that periodically switches between
66
“OneMax” and “ZeroMax”. Students are asked to first design on paper, and then evaluate, strategies which
maximize the response time, and time-averaged vales of the mean and maximum population fitness. This aids
discussions of issues such as the need for multiple runs with stochastic algorithms
The final set of exercises concerns the task of designing an EA that will evolve a controller for a Pac-Man to
find the “gold” in a maze. Solutions are interpreted as a set of rules governing motion, so at each time step the
“state” of the neighbourhood is queried and the evolving solution is queried to provide the relevant action. The
students are given a choice of two representations – in the first the “state” is the timestep – so a solutions is
represented as a sequence of moves. In the second, the state is inferred from the Pac-Man’s current location
and orientation, so the binary representation for this problem is effectively a Pittsburgh-style classifier system.
The occupancy (wall/empty) of the three squares (left, ahead, right) provides for 8 states, which with four
directions requires a sixteen-bit representation. The extension to consider ghosts etc is trivial. They are
required to select appropriate parameters (mutation rate, population size etc.) for their chosen representation.
The final design decision requires the students to create a fitness function based on weighting factors such as
final distance from goal, number of cells explored, the reward for reaching the goal, and the number of steps
taken to reach it. In practice most explored both representations, and the particular choice of maze nicely
illustrated issues such as over-fitting – the rule based version first “evolved” depth first, then “refined” it to
exploit characteristics of the specific maze used.
4. CONCLUSIONS
This paper reports an attempt to enliven the teaching of artificial intelligence by a coherent set of providing
practical activities illustrating the main elements of an introductory course to AI via the metaphor of PacMan.
These activities have been running since September 2009 and have been well received. Attendance, already
high in previous years has been extremely high and activity of discussion board sand between classes has
been higher than usual. Of 50 students filling in an anonymous module evaluation form, more than half used
the “extra comments” section to comment positively on the use of Pac-Man.
5. ACKNOWLEDGEMENTS
The authors would like to thank Doctor Gordon Downie for his considerable input to this project. The work
was supported by grants from the Faculty of Environment and Technology at UWE (“Development of
interactive small group tasks to be embedded within larger group tutorials”) and the Higher Education
Academy Subject Centre in Information and Computer Sciences (“Teaching Artificial Intelligence with
PacMan”).
6. REFERENCES
Kumar, D, and Meeden, L. “A Robot Laboratory for Teaching Artificial Intelligence”, Proceedings of the
Twenty-ninth SIGCSE Technical Symposium on Computer Science Education (SIGCSE-98), Daniel Joyce,
Editor, ACM Press, 1998.
Karapetsas , E. and Stamatis, D., “Teaching AI Concepts Using a Robot as an Assistant”, in Fasli, M. (ed.)
Proceedings of the 4th Artificial Intelligence in Education Workshop, Cambridge, UK. HEA-ICS, 2009.
Parsons, S. and Sklar, E. Teaching AI using LEGO mindstorms, Proceedings of the AAAI Spring Symposium
on Accessible Hands-on Artificial Intelligence and Robotics Education, Stanford, 2004.
Wilensky, U. NetLogo. Available from http://ccl.northwestern.edu/netlogo. Center for Connected Learning and
Computer-Based Modeling. Northwestern University, Evanston, IL. 1999
Smith, J.E. “Re-usable Online Assessment Materials for Teaching Artificial Intelligence” in Fasli, M. (ed.)
Proceedings of the 4th Artificial Intelligence in Education Workshop, Cambridge, UK. HEA-ICS, 2009.
Reynolds, Craig (1987), "Flocks, herds and schools: A distributed behavioral model.", SIGGRAPH '87:
Proceedings of the 14th annual conference on Computer graphics and interactive techniques (Association for
Computing Machinery): 25--34, doi:10.1145/37401.37406, ISBN 0-89791-227-6
67
STUDENT ENGAGEMENT AND MODES OF STUDY:
WHAT CAN VLES TELL US?
Nick Efford Roger Boyle Royce Neagle

School of Computing School of Computing School of Computing
University of Leeds University of Leeds University of Leeds
Leeds LS2 9JT, UK Leeds LS2 9JT, UK Leeds LS2 9JT, UK
N.D.Efford@leeds.ac.uk R.D.Boyle@leeds.ac.uk R.J.Neagle@leeds.ac.uk
ABSTRACT
Virtual Learning Environments (VLEs) are used widely as platforms to standardise the provision of teaching
materials and implement institutional strategies for blended learning. As such, they provide unique
opportunities for the collection of data not only on who is accessing online resources, but also on when, from
where and how they choose to do so. Analysis of these data is potentially of great value in assessing the
learning habits of the digital natives who enrol on today’s university courses. Such information can have
significant implications for how we deliver and resource the teaching of computing in the future.
In this paper, we consider a recent project to deploy a market-leading commercial VLE solution across a large
UK university. We highlight the limitations of this market-leading system for the large-scale analysis of
student engagement and describe software developed locally at Leeds that runs outside of the VLE itself,
performing extensive data mining of server log files to obtain the information it needs.
Our paper addresses the technical challenges of performing this type of data mining exercise and discusses
the variety of insights that are possible from the analysis of such data. We also present some preliminary
results from running our system on two academic years of VLE logins – several million in total, around
100,000 of which are from computing students. Our results illuminate distinct patterns of use amongst these
students and shed new light on when and where they prefer study.
Keywords
VLE, learning environment, data mining, server logs.
1. INTRODUCTION
Virtual Learning Environments or VLEs – also described as Learning Management Systems (LMS) or Course
Management Systems (CMS) – have become commonplace in HE institutions and are seen increasingly as
critical to cross-campus blended learning strategies. A DfES research report from 2004 found that half of its
survey sample had an institutional VLE that was seen as “encouraging a more mixed range of learning and
communication activities, as well as more interactivity” (Lewis & Goodison, 2004). A more recent case study
of VLE deployment in a UK university highlighted the significance of a VLE’s ability to “simultaneously perform
more than one function such as the delivery of course notes... email... online chat and discussion boards...
online tests and assignment submission” (Grewal & Harris, 2008).
A new VLE represents a major investment for an institution, in terms of hardware, software licenses (in the
case of commercial solutions), training and technical support. It also demands a significant investment of time
from instructors, who have to transfer module materials to it and learn how to make best use of its various
features. Effective tools for assessing student engagement are therefore essential if we are to make
evidence-based judgements of the return on these investments.
2. EXISTING TOOLS AND APPROACHES

Blackboard Inc. is currently regarded as the leading commercial provider of VLE technology, with an
estimated 80%-90% of the US market (Griffiths, 2007) and widespread adoption in the UK. The University of
Leeds recently completed a two-year project to deploy Blackboard across the institution, during which the
need for detailed analysis of modes of use was identified. Version 7.3 comes with a built-in statistics tracking
capability, but it operates on a per-module basis only, requires labour-intensive configuration by module
leaders and is clumsy to use; furthermore, it offers no facility for aggregating results across modules, cohorts
Subject Centre for Information and Computer Sciences 68
of students or organisational units such as schools or faculties, and it suffers from significant limitations with
regard to both the amount of information that can be collected and the way in which that information is made
accessible.
Two different home-grown data mining approaches are possible for a system such as Blackboard. One is to
query Blackboard’s internal database directly (Griffiths, 2007), which simplifies data analysis greatly but does
not capture every variable of interest; the other is to operate entirely outside of Blackboard, mining data
sources such as web server logs (Ai & Laffey, 2007; Hung & Zhang, 2008). For various reasons that are not
relevant to this paper, we have been constrained to follow the latter approach.
VLE data mining is regarded as a relatively new field of study (Hung & Zhang, 2008). Published research has
tended to focus on a narrow cohort such as students on a single course (Ai & Laffey, 2007; Hung & Zhang,
2008) and on making specific predictions of student behaviour (Luan, 2002; Ai & Laffey, 2007). A literature
survey by West et al (2007) counted 164 published articles on VLEs, but fewer than ten of these described
data-driven studies that performed a more general evaluation of VLE impact – across schools or faculties, for
instance. Our work falls into this particular category but is unique for its scale, its longitudinal aspects and the
particular emphasis it places on time and place in the analysis of student VLE access.
3. IMPLEMENTATION AND TECHNICAL CHALLENGES

The Blackboard VLE at Leeds is fed daily with almost 180 MB of data from the university’s student information
system, Banner. Our system taps into this data feed on an irregular basis to ensure that it has relatively
accurate information on student users of the VLE: specifically, the type of student (undergraduate, taught
postgraduate, research postgraduate), their year of study and their programme of study. From the latter, we
are able to determine the school and faculty to which a student belongs.
Each day, our system acquires and processes two sets of new log files. The first set records logins and
contains user IDs, timestamps and IP addresses; we use the latter, in conjunction with campus network
details and a GeoIP database, to determine the student’s location during their VLE session. The second set
of data consist of standard Apache web server logs, which we process to derive additional information such
as session duration, number of HTTP GET/POST requests per session and user agent details. The latter are
useful in identifying the technology used by students to access the VLE (hardware platform, operating system
and web browser).
Each VLE session associated with a student is tagged with that student’s type, year of student and
programme of study and the resulting record is stored in an SQL database. We deliberately avoid storing
user identifiers, as we are interested only in aggregate behaviour and not in the activities of individuals. We
record session details for students only; staff use is largely ignored (except for counting the total number of
staff logins per day).
Leeds is a large university and VLE uptake has been almost universal, so the daily data volumes faced by our
system are significant. 17 March 2010, for example, generated 2.2 million lines of web server logs – 1.8 GB
of raw, uncompressed data in total. For the academic year up to that date, we had collected 324 million lines
of web server logs, totalling over 250 GB. At the time of writing (May 2010), our database had accumulated
details of over 3.9 million student VLE sessions for the current academic year, had counted nearly 330 million
hits against the VLE’s web servers and was occupying over 400 MB of disk space.
Interaction with the data is facilitated by a web-based front-end, implemented in Python using the Django web
development framework. Each view into the dataset can be restricted to a specific period of the university
calendar or to an arbitrary date range. Results are cached for performance reasons and are presented in the
browser in tabular/graphical forms, as appropriate. Selected data can be downloaded as CSV files or Excel
spreadsheets, and there is a limited facility to generate summary reports as PDF documents and email them
to interested users.
4. RESULTS
4.1 Overall Patterns of Use
Figure 1 plots daily login totals during the current academic year, for both students and staff. The effect of
Christmas and Easter vacations is clearly visible. Most strikingly, a weekly cycle is evident for both student
and staff VLE use, in which logins peak on a Monday before gradually declining over the course of the week,
falling invariably to a minimum on Saturday and recovering somewhat on Sunday. Griffiths (2007) detected a
similar cycle in his analysis of activity at Brigham Young University, but did not venture an explanation. We
hypothesise that this distinctive rhythm of VLE use reflects a preference that both staff and students have for
69
taking a day off from work on Saturday and then using Sunday and Monday to prepare for the working week
ahead.
Figure 1: daily login totals for 2009-10 (upper line students, lower line staff)
Figure 2 is a coarse-grained comparison of student location

during VLE sessions. Interestingly, over 50% of VLE access
takes place from outside the campus network. The next
largest location category is halls of residence. These two
categories can be regarded collectively as representing
access from “home”. On this basis, we can say that nearly
three-quarters of VLE use is from home rather than from
computer clusters on campus. This is consistent with the
DfES finding that VLEs are valued because they enable
students to “work at their own pace, in their own time and
chosen location” (Lewis & Goodison, 2004).
Our system is able to classify off-campus access to the VLE
by country of origin. The value of such analysis is limited, Figure 2: location of VLE sessions
given that the vast majority of access (over 97%) is from the
UK; nevertheless, it is interesting to note that students at Leeds have accessed the VLE from 144 other
countries – a fact that underlines the truly international nature of the institution. Some of the 60,000 VLE
sessions from outside the UK that we have observed this year represent overseas students keeping up with
their studies while at home during vacations, but a significant proportion are seen to occur during term-time
from Spain, France, Germany and other countries where students from the School of Modern Languages are
spending their year abroad. These students are using the VLE to keep in touch with tutors and report on
experiences.
As a final general observation, we note that our system provides insights into the technology employed by
VLE users. We see, for example, that Microsoft Windows systems have been the source of 76% of all VLE
use so far this year but that the Internet Explorer browser (IE) has been responsible for only 52% - the
remainder coming from Firefox, Safari and Google Chrome. Moreover, it is clear that IE is losing ground
relative to these alternatives, despite the fact that our version of Blackboard is demonstrably less reliable in
some respects when accessed from these other browsers and is therefore officially unsupported on them.
One of the most intriguing results concerns access from mobile devices, currently representing less than 1%
of all VLE access but clearly trending upward (Figure 3). 90% of the observed traffic comes from Apple
iPhone and iPod Touch devices.
70
Figure 3: VLE access from mobile devices, Oct 2008 – Apr 2010
4.2 Computing Students

What do VLE access patterns look like for computing students? Figure 4 shows which of the 46 schools at
Leeds were the top five users of the VLE during the 2008-9 and 2009-10 academic years. Our normalised
measure of comparison is SSD, the average number of VLE sessions per student per day. Computing was
the biggest user during 2008-9, reflecting the ease with which its students adapted to VLE features such as
discussion forums; a key reason for this, aside from the students’ natural enthusiasm for web technology, was
the pre-existing culture of supporting every module with a web site and USENET-style newsgroup.
By 2009-10, however, the picture has changed, and other schools have overtaken Computing. Figure 4 also
shows how level of use has increased across the board, with the average student in some schools logging in
twice as often as in the previous academic year. This is partly due to a roll-out process that has activated
modules in the VLE gradually, rather than all at once.
2008-9 2009-10
School SSD School SSD
Computing 0.526 Biol UG 0.935
Biol UG 0.519 Business UG 0.768
Business UG 0.369 Psychology 0.636
Medicine 0.319 Computing 0.634
Jt Hons Sci 0.317 Jt Hons Sci 0.614
Figure 5: VLE use by different types of Computing

Figure 4: top five VLE users
student in 2008-9 (left) and 2009-10 (right)
Figure 5 shows the breakdown of VLE use in Computing between types of student in the 2008-9 and 2009-10
academic years. We see a relatively even split between undergraduate (UG) and taught postgraduate (TP),
with the latter group overtaking the former this year for reasons that are unclear as yet.
We have also looked at the breakdown of VLE use between different UG year groups in Computing and
observe that Year 4 students (those who have returned from a year in industry) appear to access the VLE
more frequently than Year 2 students, despite studying fewer modules as a result of their final-year project.
Another puzzling observation is that VLE use has reversed amongst Year 1 and Year 2 students, being
somewhat lower for the former during 2008-9 but significantly higher in 2009-10. The reasons for these
observations are again unclear and merit further investigation.
There are clear differences between computing students and other students with regard to both time and
place of VLE access. Figure 6 shows the locations of VLE sessions for students in Computing (left) and the
UG Business School (right). We see that the libraries are relatively important locations for business students
but are of minimal significance to computing students – reflecting, no doubt, the greater importance of
physical textbooks and reading assignments within the Business School. We also see that off-campus
access is much more significant for business students and that access from campus computer clusters is
much more significant for computing students. This reflects the lab-based nature of computing; a similar
location distribution is observed in other schools where students do a large amount of practical work on
computers (e.g., Electronic Engineering). We note that, even for lab-based disciplines, VLE access from
home still exceeds 50%.
Figure 6: VLE session locations for computing students (left) and business students (right)
71
Adding a time dimension to the analysis sheds further light on the variability of access patterns. Figure 7 (left)
shows how VLE sessions of computing students are distributed over the day. Lab-based access dominates
between 10 am and 4 pm but declines rapidly thereafter. There is a sustained level of home-based use
throughout the evening, with more access occurring between midnight and 1 am than between 9 am and 10
am. Medical students exhibit a rather different profile (Figure 7, right), with lab-based access much less
significant than home-based access and with much less of a tendency to be “night owls”.
Figure 7: hourly profile of VLE access in 2008-9 for computing students (left) and medical students (right)
5. CONCLUSIONS
We have shown that it is possible to extract valuable data on how students are engaging with a VLE across
an institution and within a computing department via web mining techniques. The significant technical
challenges posed by the scale of our study mean that this research is still in its early stages; we are pursuing
it further by investigating the working habits of computing students using focus groups (Boyle et al, 2009) and
by expanding our system to encompass more detailed analyses of specific VLE tools such as discussion
forums, quizzes, etc.
6. REFERENCES
 Ai J & Laffey J (2007) Web mining as a tool for understanding online learning, Journal of Online
Learning and Teaching, 3(2), pp.160-169
 Boyle R, Efford ND & Neagle RJ (2009) Evolving patterns of working: do they matter?, submitted to
Computers and Education
 Grewal SK & Harris LJ (2008) Beyond critical mass: a case study investigating the use of WebCT for
course delivery by faculty in a campus-based UK university, Journal of Online Learning and Teaching,
4(3), pp.380-390
 Griffiths ME (2007) Patterns of User Activity in the Blackboard Course Management System Across
All Courses in the 2004-2005 Academic Year at Brigham Young University, Masters thesis
 Hung J-L & Zhang K (2008) Revealing online learning behaviours and activity patterns and making
predictions with data mining techniques in online teaching, Journal of Online Learning and Teaching,
4(4), pp.426-437
 Lewis D & Goodison R (2004) Enhancing Learning With Information and Communication Technology
(ICT) in Higher Education, DfES Research Report RR533
 Luan J (2002) Data mining and its applications in higher education, New Directions for Institutional
Research, 113, pp.17-36
 West RE, Waddoups G, Kennedy MM & Graham CR (2007) Evaluating the impact on users from
implementing a course management system, International Journal of Instructional Technology &
Distance Learning, 4(2), http://www.itdl.org/Journal/Feb_07/article01.htm (27 May 2010)
72
ENHANCING STUDENT ENGAGEMENT AND SUPPORT WITH
DIGITAL VIDEO
Dr Paul Newbury Dr Phil Watten

University of Sussex University of Sussex
School of Informatics School of Informatics
Brighton BN1 9QH Brighton BN1 9QH
P.Newbury@sussex.ac.uk P.L.Watten@sussex.ac.uk
www.mtllive.com www.mtllive.com
ABSTRACT
The School of Informatics at the University of Sussex has two television broadcast studios, which support
teaching on our technical digital media degrees. In the first instance these facilities are used to give hands-on
experience of aspects of broadcast technology, live video productions and digital content creation. However,
over the last few years the School has been experimenting with several different methods of enhancing
student learning, engagement and support on other courses using these studios, specialist techniques and
associated systems. This paper reviews lecture and laboratory attendance, download rates and qualitative
student feedback and shows that using these techniques has a significant effect on the student learning
process. Other types of on-line support such as enhanced podcasts are also investigated and a tangible
improvement in engagement over these techniques is shown.
Keywords
Digital video, video streaming, multimedia, engagement, attendance, podcast
1. INTRODUCTION
It is now common for Universities to record lectures with video cameras or as enhanced podcasts [1,2] and
there are several off-the-shelf systems, which Universities can purchase to provide this type of functionality
(Echo360 [3], etc). There are also several distribution outlets available, such as iTunesU [4] and YouTube
EDU [5], which can be used to share this media with the students. However, although the recording of audio
and slides (often referred to as podcasts, screencasts or enhanced podcasts) has been around for several
years the use of mainstream video is a fairly recent innovation [6], the vast majority of higher education
material being produced from a single camera placed at the back of the lecture theatre with the possible
addition of a separate capture of the slides. Although several studies have shown that this is generally
advantageous to the students [7,8] this type of material is best for students revisiting the lecture (after
attending) to refine their notes or for revision [9]. There are several key issues with is type of media:
• The material lacks focus, the presenter, audience and screen are all included in the capture;
• Lighting is often an issue. It is hard to light both the presenter and the screen in a reasonable manner;
• Screen/board is often hard to read. Text and diagrams are often too small to see.
These problems (as can be seen in Figure 1) combine to often make the video lectures little use in terms of a
teaching medium, and can undermine the additional engagement that might be considered implicit with this
type of support material. In an effort to address these issues this paper reviews a pilot scheme that uses high-
quality digital video resources to support student learning for both lectures and laboratory sessions.
The specialist studio facilities in Informatics enable a significant improvement on standard video capture and
the creation of high-quality tutorial materials, which enhance both the student learning experience and
engagement.
73
Figure 1 - Echo 360 capture of a lecture
Studio-based work includes live multiple-camera lecture recording, the creation of video-based feedback
materials in response to student requests and the distribution mechanism for this support that incorporates
multiple platform coverage extending to mobile platforms such as the iPhone.
2. PRODUCTION FOR DIGITAL VIDEO

There are essentially 3 key parts to the production pipeline outlined in this paper:
a) Materials development. As the consumer of this material may be viewing it on a range of possible
resolution devices it is key that all material is clear at the lowest resolution. At present the lowest resolution
target device is the iPhone (Figure 2b). This requires special handing of textual material and its replacement
with graphical material wherever possible. There are specific requirements as to minimum font size, use of
colours and safe view area for all presentations.
b) Live recording. The objective of the proposed system is that presentation material is captured as efficiently
as possible, and that means in real-time. The system discussed in this paper produces high-quality engaging
output by using multiple cameras to cut between camera views recording material in real-time, without the
need for postproduction editing.
c) Streaming. As the material is recorded live and does not require any post recording editing it can be
encoded and uploaded to the web within minutes of the presentation ending. Our system includes a video
streaming site using the Wowza media server [10] and FlowPlayer video player [11], which enables students
to view the video from a range of devices (on or off campus) as shown in Figure 2.
Figure 2 - (a) Web page streamed video. (b) iPhone
3. THE STUDIO
The Media Technology Lab in the School of Informatics consists of two broadcast studios and associated
control galleries and non-linear editing facilities. However, for the majority of this work a single studio and
gallery is used as shown in Figure 3. Generally 3 cameras and slide capture is used in presentations,
although it is possible to get reasonable results using just a single camera and slide capture. The studio has
been built in a highly configurable manner, which enables quick conversion and setup between different
presentations.
74
Figure 3 - (a) One of multiple camera views of the presenter and screen. (b) The control gallery
4. SUPPORT FOR STUDENT ENGAGEMENT

The MTL studio system provides support for student engagement by providing a rich learning environment
and material for many different subjects and teaching scenarios. However, the two key areas covered in this
paper are digital video for lectures and tutorial lab support.
4.1 Lecture Engagement

The significant amount of pre-production work put into the development of slides for video means that these
slides are media rich and engaging. The use of the studio also means that full multimedia facilities are
available so that sound, video and animation are common components of lectures. When reviewing lectures
through the web-streaming framework, students are presented with a clear representation of the material
covered through a selection of views (including multiple views of the presenter and screen and also slide
capture). This is much like the experience of television viewers of the Royal Institution lectures rather than the
back of the lecture theatre video recordings which are common (Figure 1).
Student engagement often suffers if key lectures are missed which make it hard for them to absorb the taught
material following that lecture even if they have access to the slides and notes. This research found that
students who missed a lecture receive a very similar experience to those who attended if they subsequently
viewed the streamed lecture. The issue of attendance is covered in section 5.4.
4.2 Tutorial Engagement

Students are provided with written tutorial sheets to follow in computing laboratory sessions. However,
increasingly we find that some students have trouble progressing through this written material. The studio
facilities enable the production of a significant amount of video tutorial material, which the students can use in
laboratory sessions. Students have been overwhelmingly positive about the provision of this new material and
they engage with it more keenly than reading the traditional written material. All courses in Informatics are
supported by StudyDirect, the University’s Moodle-based VLE [12]. Students commonly engage well with the
discussion forum part of this system and this enables the production of targeted video tutorial material aimed
at common problems. This in turn leads to reduced need for teaching support in labs.
5. MEASURE OF ENGAGEMENT
As cohorts differ in size, makeup and aptitude between years it is hard to make definite claims about student
engagement from one course to the next. However, it is generally considered that qualitative student
feedback, download rates and attendance rates provide an insight into the student experience [8,13] Thus this
paper assesses students’ engagement using the following 4 key criteria:
 Download figures for the support material (traditional vs. new);
 Qualitative student feedback;
 Qualitative faculty feedback;
 Lecture attendance rates.
5.1 Download Figures

The following data shown in Figure 4 represents the material downloaded by students for each of the past 3
years of a year 2 multimedia course. In 2007/8 the only material available for download was the lecture slides.
In 2008/09 an enhanced podcast was available (slides and audio). In 2009/2010 a complete video was
available for each of the lectures. Figures have been normalised for the slight variations in cohort size
between years.
75
Figure 4 – Number of student downloads per year of course.
Download rates for the video are significantly higher than for the other two forms of media. Interestingly the
podcasts don’t fare particularly better than the static slides when compared to the video downloads.
5.2 Students’ Feedback

Students were polled on their feelings concerning the digital video material. From those results: 92% had
watched the video material online and 23% rated this material as “Invaluable” with the remaining 77% rating it
as “Very Useful”. Qualitative feedback followed these results with students believing that this was a positive
aid to their learning that they would wish to see used in all their subjects. Some example student comments:
“I find it useful to replay a lecture. It helps consolidate the information. The videos will be useful for revision.”
“We can see the lecturer himself not just hearing his voice, have a feeling of realism and like watching TV”
“If you didn't quite catch something that was said or don't fully understand it, you can go back and watch it
again online rather than stop the lecture mid-flow and ask them to repeat it.”
“They are an excellent resource - at revision they will be so much more useful than just slides and audio”
“You get to hear and see the full explanation of the slides and this is much more interesting than podcasts.”
“The ability to see the entire lecture again and to pick up points I didn't understand first time round was great”
5.3 Faculty Feedback

Those taking part in the filming found it a rewarding experience. An unexpected consequence of this work
was that the ability to review the lectures themselves was found to be very useful in revising, updating and
improving the delivery of the material. Faculty do not normally get the opportunity to view their own lectures
and modify their delivery based on this view. The faculty impression of the students’ engagement was that
they were more interested by the material and were excited by the studio atmosphere. Inattention issues were
reduced and there were no issues with disruptive behaviour. In laboratory sessions the students’ progression
through the work was smoother with significantly less questions asked and those questions that were asked
tended to be at a higher level than in previous years.
5.4 Attendance Figures

Although only limited research has been performed on the effect of podcasts on student attendance [8,14,15]
it is a fairly common perception that providing podcast material is likely to have a detrimental effect on
attendance. The common question “Why would students attend lectures when they could watch the lecture at
home” is often posed. However, previous research suggests that provision of podcast material has limited
effect on attendance [14,15]. In cases where there is a drop in attendance this is often not converted into a
drop in student achievement in the course as the additional support material alleviates this issue [15].
Although the enhanced digital media created in this research could be considered more likely than standard
podcast technologies to result in a drop in attendance the results were found to be broadly inline with previous
studies [14,15]. Attendance is taken at all lecture and laboratory sessions. The percentage attendance figures
for the above 3 years of courses can be seen in Figure 5.
Figure 5. Percentage attendance figures.
76
Several factors affect attendance on courses from year to year. One of the key factors in attendance on the
above course was the move to 9am lectures for the 08/09 cohort. Another factor is the gap between this
lecture and the next. Unfortunately, for the 08/09 cohort there was a 4 hour gap between this lecture and the
next lecture of the day. This may well be the reason for the slight drop in attendance at this point. The 09/10
cohort has the same lecture times as 08/09 and attendance has rallied slightly. Although a small subset to
extrapolate from, it seems that attendance has not been affected significantly by the videoing of the lectures.
When asked “Has the fact that you can view the lectures on-line changed the number of lectures that you
have attended?”, 54% said no, 38% said they had attended less and 8% said they had attended more.
6. CONCLUSION
The student response to the provision of lecture and laboratory support material as digital video has been
overwhelmingly positive. The student progress in the laboratory sessions is faster and the students are overall
more engaged with the taught material. There seems to be a key difference in the student response to
podcasts than there is to digital video material. Not only are download rates significantly increased, but the
students comment that they find the video interesting and “like TV” rather than the slides and audio podcasts,
which they often refer to as “boring”, or hard to watch for any length of time. Although a significant concern of
many academics it seems that the provision of this high-quality digital material has only a limited effect on
attendance and the student performance overall is increased. So far this digital videoing pilot study has only
been used in a few courses, and future work will involve the rolling out of this framework to more of the
subjects in the School of Informatics, with the associated monitoring of student feedback and attendance. By
increasing the number of courses covered it is hoped that quantitative analysis of student results on these
courses compared to previous years can be made and significant improvements in attainment shown.
7. REFERENCES
[1] MIT Open Courseware, http://ocw.mit.edu/OcwWeb/web/home/home/index.htm (May 2010)
[2] Imperial Colledge Online Lectures, http://www3.imperial.ac.uk/media/onlinelectures (May 2010)
[3] ECHO360 Lecture Capture System, http://www.echo360.com/ (May 2010)
[4] iTunesU, http://www.apple.com/education/itunes-u/ (May2010)
[5] YouTube EDU, http://www.youtube.com/education?b=400 (May2010)

[6] Copley, J. (2007) “Audio and video podcasts of lectures for campus-based students: production and
evaluation of student use”, Innovations in Education and Teaching International, vol. 44, 387-399
[7] Woo, K, Gosper, M, McNeill, M, Preston, G, Green, D, Phillips, R (2008) “Web-based Lecture
Technologies: Blurring the boundaries between face-to-face & distance learning”, ALT-J. vol. 16-2, 81-93
[8] Bradley, P, Summerside, C, Agar, M, Ansell, P, Humphrey, R Knight, J, Mohammed, A, Moss, J, Watts,
C, Wheeler, J, & Wolfendale, D (2009), “Large scale implementation of a lecture capture system: a value
added initiative?”, ALT-C 2009 presentation
[9] Bongey SB, Cizadlo G, & Kalnbach, L (2006), “Explorations in course-casting: podcasts in higher
education”, Campus-Wide Information Systems, vol. 23, issue 5, 350-367
[10] Wowza media server, http://www.wowzamedia.com/ (May 2010)
[11] FlowPLayer video player, http://flowplayer.org/ (May2010)
[12] Moodle Course management system (CMS), http://moodle.org/ (May2010)
[13] Day, J and Foley, J, (2006), ”Evaluating Web Lectures: A Case Study from HCI.” Conference on Human
Factors in Computing System 2006, 195-200
[14] Traphagan, T, Kucsera, J V, and Kishi, K, (2009), “Impact of class lecture webcasting on attendance and
learning”, Educational Technology Research and Development, Volume 58, 18-37
[15] Harpp, D N, Fenster, A E, Schwarcz, J A, Zorychta, E, Goodyer, N, Hsiao, W and Parente, J, (2004),
“Lecture Retrieval via the Web: Better Than Being There?”, Journal of Chemical Edu, vol. 81 n5, 688
77
TAGS AS A FEEDBACK MECHANISM IN PROGRAMMING COURSES:
ANALYSIS TO SUPPORT JUST-IN-TIME TEACHING
Stephen Cummins Liz Burd Andrew Hatch

Durham University Durham University Durham University
South Road South Road South Road
Durham Durham Durham
s.a.cummins@durham.ac.uk liz.burd@durham.ac.uk andrew.hatch@durham.ac.uk
www.durham.ac.uk/s.a.cummins www.durham.ac.uk/liz.burd www.durham.ac.uk/andrew.hatch
ABSTRACT
This paper evaluates the use of a new technique for generating, disseminating and analysing feedback for
programming courses at undergraduate level. The technique incorporates ideas from the popular Web 2.0
tagging paradigm and explores the resulting feedback generated. This interdisciplinary project between
software engineering and education investigates how analysis of programming feedback can be used to direct
remedial teaching. To illustrate the interdisciplinary nature of the project this paper utilizes a technique found
primarily in social sciences called Thematic Analysis along with sentiment analysis in order to explore the
feedback given to a cohort. The results presented here show experiences of using the technique and how the
analysis of the resulting feedback data influenced teaching on a particular programming course. This paper
concludes by identifying possible extensions to the research.
Keywords
Feedback, Programming, Just-in-time teaching, Sentiment Analysis, Tags
1. INTRODUCTION
This paper utilises a novel alternative to traditional feedback mechanisms by adopting the Web 2.0 idea of
using tags to organize information, in order to provide feedback for programming assignments. This approach
known as the “SoftWare Assessment Through Tagging” or SWATT approach, involves one or more
examiners electronically annotating students’ source code with short comments in the form of tags. The
resulting collections are used to generate clouds of feedback tags, which represent an individuals’ assignment
feedback or indeed the feedback of an entire cohort. The ability for students to share these feedback clouds
along with the associated source code in a Web 2.0 style empowers them to make the most of the feedback
generated.
The SWATT system was developed (Cummins et al, 2009) to explore the use of tag based feedback. The
system enables examiners to annotate student projects within the Eclipse Integrated Development
Environment which was a regularly used environment for examiners. It also provides an online portal that
enables examiners and students to view and analyse the feedback tags. Students may view a summary of
their feedback as a tag cloud similar to that shown in Figure 1. The feedback can also be viewed alongside
the students’ original source code, thereby preserving the context of each feedback tag and allowing students
to relate the comments directly back to their work.
A key feature of SWATT is that it allows students’ to share their feedback and associated code. This enables
individuals to browse their peers’ feedback and provides an opportunity for them to identify instances where
another student has had similar feedback. This supports students in being able to gain a deeper
understanding of their feedback and that of their peers. Students can also opt to remove anonymity and allow
peer discussion concerning their work or feedback, thereby building a social community around feedback.
A motivation for using this approach to feedback generation is that the resulting feedback data inherently
lends itself to analysis techniques such as frequency analysis and co-occurrence analysis of tags. These
techniques are easily implemented when using tagging systems and results of these may provide insights into
patterns of feedback which occur in a cohort. This information could assist lecturers in modifying their
teaching to better support the learning needs of a particular cohort. This paper focuses on presenting one
approach to feedback analysis aimed at supporting Just-in-Time Teaching (JiTT) using feedback tags.
78
Figure 1 An Example Feedback Tag Cloud
The research questions (RQs) considered by this paper are as follows:
 RQ1: What are the most common themes exhibited by the feedback tags issued for a given
assignment?
 RQ2: Can sentiment analysis be used along with thematic analysis to identify topics of a course that
could benefit from additional teaching?
2. RELATED LITERATURE
Feedback is the most important aspect of learning, it is how students are given the information that they need
to improve (Laurilard, 1993). Feedback is given in various formats from written annotations to summary
sheets pre-appended to the students work. However, the feedback normally intended for student usage can
also be used by lecturers in helping to understand the common gaps in the knowledge of students in a
particular cohort.
Just-in-Time Teaching (JiTT) (Novak et al, 1999) is a strategy where by lecturers use feedback from a cohorts
assessment to enable them to adjust their teaching in lectures to provide extra support on the learning
difficulties detected by the assessment. It is called just-in-time teaching because the instructor reads the
cohorts feedback just-in-time to adjust their lecture content, tailoring it to suit the needs of the learners.
2.1 Sentiment Analysis & Thematic Analysis

Sentiment analysis is a technique that enables the analyst to extract the opinion or underlying connotation of
text. That is, it allows the recording of whether a text is perceived as positive, negative or neutral. The
sentiment of feedback is an important consideration as it has been shown that too much negative feedback
can actually adversely affect students’ future performance (Gee, 1972). Some assessment tools include
sentiment analysis as part of them to enable examiners to verify the sentiment of feedback before its release
(Gillam et al, 2009). Automated tools can provide an estimate of how a human may perceive the sentiment of
feedback. These estimates are not always correct especially if the use of technical terminology is common
throughout the feedback (Cummins et al, 2010). This paper utilises one freely available automated sentiment
analysis tool provided by the National Centre for Text Mining (NaCTeM), (NACTEM, 2010).
Themes within feedback can be detected through a manual process of Thematic Analysis. Thematic Analysis
is a technique often associated with social sciences and analysis of large narratives such as interviews. It is
used to highlight frequently occurring themes or patterns from a particular dataset. Thematic analysis is a
several staged technique sometimes requiring multiple iterations in order to refine the themes and ensure that
they are clear and repeatable. Independent reviewers are often used to confirm the repeatability of the
analysis. The steps involved in the thematic analysis process are summarised below;
1. Derive initial themes based on software engineering theory and examiner experiences
2. Code the feedback tags into the themes
3. Give a 30% sample of feedback tags to reviewers to encode
4. Calculate reviewers agreement rates
5. If agreement rate < 80% threshold then reduce or redefine initial themes and repeat from step 2.
79
Burn describes a similar application of the thematic analysis method on configuration management change
logs in Software Engineering Group projects (Burn, 2008). It is a similar process that has been adopted here.
The initial step of a thematic analysis is to generate a set of codes or themes representing general topics
within the dataset. The second phase is to assign these themes to the data. After the entire dataset has been
encoded by the principal researcher the validation phase begins. A sample of the data and themes is given to
one or more reviewers in order to confirm that the codes have been correctly assigned. The aim is that the
reviewer is able to repeat the same encoding if the same themes and data were given to them. A desirable
agreement rate is often decided in advance and used as a metric for successful analysis since it is accepted
that a 100% agreement rate is unlikely since human opinion is by its nature subjective.
This technique is used to group feedback tags into meaningful themes and the sentiment analysis is used to
identify which have high positive or negative feedback associated with them.
3. INVESTIGATION METHOD
This paper presents the results of an exploratory study conducted with students involved in a large scale
Software Engineering Group (SEG) project using Java to create an application on a Personal Digital
Assistant. The module was part of a 2nd year undergraduate Software Engineering course at Durham
University. The cohort was comprised of 67 mixed ability students divided into 12 ‘company’ teams. This
project gave rise to large repositories of source code for which some form of assessment and feedback was
required.
Normal feedback practice for this type of assignment would be to provide a summary sheet with general
feedback relating to the quality of the source code overall. Very rarely would the source code ever be
annotated either electronically or on paper due to the large volume and complexity of the submitted work.
In order to test this new approach of feedback analysis, the feedback to this SEG assignment was generated
using the SWATT system. Discussion about how the groups used the new feedback tags system is omitted
from this paper but has been investigated in (Cummins et al, 2009).
The feedback students received was generated by two examiners who independently annotated the same two
files from each groups’ software project. These two files were selected based on number of revisions made to
them in the configuration management system. This was used as a naïve and simple mechanism of
identifying interesting source files and was selected entirely for convenience.
In order to generate the sentiment analysis data required to determine the areas of strength and weakness for
the cohort, the NaCTeM automated tool was used to analyse each feedback tag. This resulted in each tag
being associated with one of three sentiment values; positive, negative or neutral.
The process described in Section 2.1 was followed for the thematic analysis and resulted in each feedback
tag having an associated theme as well as sentiment value. The initial themes were generated using software
engineering theory and evidence from feedback given in previous programming assignments.
4. RESULTS & EVALUATION

The initial themes generated did not yield a high enough acceptance rate during the review phase as only
66% agreement on tag themes was reached. The desirable agreement rate for this study was 85% as it
represents a significant agreement rate (Burn, 2008). A refinement and reduction of themes took place for the
second iteration and an average agreement rate of 90.45% was achieved between the two reviewers and the
researcher. This yielded an agreement rate sufficient for the analysis to end.
A description of the final set of themes used is as follows.
 Completeness – This theme represents feedback tags that identify how finished the students work is
in terms of the functionality described by the assignment task.
 Comprehension – refers to any tags that highlight issues surrounding the understanding of source
code, for example comments, documentation or ease of extendibility. This is also a theme that
encompasses tags that comment on the layout, order and style of code.
 Design – is any tag that refers to design elements reflected in the source code. Any tags relating to
the use of design patterns or pre-implementation phase issues are design tags.
 Miscellaneous – represents tags that cannot fit into any other theme. Often tags that are very
general or too specific are included in this theme. An example of this is ‘Good’, when isolated from its
associated code fragment it does not indicate any meaningful information.
80
 Programming Standards – is a theme that refers to tags that identify how students’ code aligns to
accepted standards within the field. It can also refer to tags that suggest improvements to code
involving use of more appropriate language features or software engineering techniques. Examples
include tags such as ‘make threadsafe’, or ‘don’t hard code’.
Table 1 shows the proportions of feedback tags in each theme. A total of 103 unique feedback tags were
analyzed over the two review cycles.
Theme % of all Positive Negative Neutral
feedback % % %
Completeness 5.42% 0.00 37.50 62.50
Comprehension 32.20% 28.42 45.26 26.32
Design 5.08% 20.00 0.00 80.00
Miscellaneous 4.07% 41.37 16.67 41.67
Programming 53.22% 10.19 13.38 76.43
Standards
Table 1 shows the distribution of positive, negative and neutral tags for each theme according to the
NaCTeM sentiment analysis engine
From the results, students in this cohort received the highest proportion of negative feedback from the
Comprehension theme. The data suggests that the cause for this result was the feedback tags “needs
refactoring” and “bad commenting”. “Needs refactoring” implies there is something structurally or
organisationally wrong with the students’ programming submission. This extremely general tag does not help
pinpoint exactly what was wrong with the student’s code and so limited analysis can is available.
The other tag “bad commenting” is an indication that students have either failed to comment their work or the
comments they have included are not meaningful to the examiner. Based on the analysis results of this
theme, it is clear that the importance of good commenting, code organisation and code documentation should
be reinforced in lectures for this cohort. This is especially due to the importance of having high
comprehensibility of software for easing the burden of software maintenance.
The themes with the next highest amounts of negative feedback associated with them are “completeness”,
“Design” and “Miscellaneous”. These themes are small and each has less than 6% of tags associated with
them. This means that whilst these themes may highlight problems in students learning, they are not
widespread and perhaps could be addressed in smaller group teaching. For example, in the Design theme
one of the higher frequency negative tags is “class too big”. This is an interesting tag as it could mean either
students are unable to decompose the problem properly using objects or perhaps they have engaged in a
process of copy and paste coding.
The most frequent negative tag within the programming standards theme is “improper exception handling”.
This is clearly a crucial topic to reinforce in lectures as failure to handle exceptions and errors in software is a
fundamental skill. Interestingly, the tag with the highest frequency in this theme is neutral and is “use
constants”. This tag suggests students have been creating fields or variables which could be declared as
being final and for whatever reason they have not used this java feature. Reminding students that this feature
exists and the benefits of protecting constants from inadvertent changes may also beneficial.
RQ1 questions what the common themes are within the feedback, after the process of thematic analysis
these have been shown in Table 1, with the theme having the highest frequency being “Programming
Standards”. The use of thematic analysis has enabled a higher level of exploration and analysis of the
feedback, however the time required for thematic analysis may be a limitation to its use for just in time
teaching, where as automated sentiment analysis was instant.
The answer to RQ2 is clearly positive, as the use of sentiment analysis has identified a number of learning
deficiencies which can be corrected through remedial teaching activities. The benefit of sentiment analysis to
simply ad-hoc analysis of the cohorts feedback tag cloud is that the negative tags are highlighted and there is
less chance that these could be missed. The benefit of thematic analysis to this process was that it became
apparent that the entire topic of software comprehension needed to be reinforced to the software engineering
students, with specific emphasis on commenting and javadoc documentation.
5. THREATS TO VALIDITY
The results are from a specific usage of the SWATT system and the analysis process defined. Therefore, the
results cannot be used outside of the context of this investigation.
81
It has been noted that automated sentiment analysis tools are not full proof and as such another threat to
validity is the fact an automated sentiment analysis tool was used to generate the sentiment analysis results.
The use of alternative tools would no doubt yield different results.
6. CONCLUSION & FUTURE WORK

This paper discusses how SWATT feedback can be used with sentiment analysis and thematic analysis to
help examiners determine which aspects of programming students require additional support with.
The process of automated sentiment analysis when combined with frequency analysis has been
demonstrated to be a very quick and useful process for detecting common problems in a cohorts learning.
This process when automated is quick enough that the analysis can be used to direct JiTT strategies.
However, as discussed in (Cummins et al, 2010) it is important that the automated sentiment analysis results
are reviewed by the examiner for correctness and not exclusively relied upon.
The use of thematic analysis provides a high level overview of what aspects of the programming course
students are struggling with. This when combined with the lower level frequency analysis allows lecturers to
focus in on particular learning issues highlighted by assessment feedback. Unfortunately, due to the overhead
involved in manual thematic analysis it is a much slower process which may reduce its usefulness as a JiTT
teaching tool. Perhaps a simplified or automated version of the thematic analysis process may be more
beneficial for use in conjunction with the JiTT.
The results presented identify that in this particular group project the comprehension theme had the highest
proportion of negative feedback associated with it. The tags usually highlight very specific problems that
students encounter, with a minority identifying high level programming standards such as encapsulation or
coupling. This may be due to feedback tagging as a system encouraging examiners to focus on lower level
source code functionality or due to the fact that the design and higher level issues were assessed in a
different deliverable of the group project.
Using feedback tags for programming assignments has proved on this occasion to produce data that can
support and can direct future teaching. The feedback clouds alone highlight specific strengths and
weaknesses for both individuals and the cohort. However, by using automated sentiment analysis these can
be easily categorised and used to make quick decisions to direct subsequent teaching as soon as the marking
process has completed. Using thematic analysis can identify general weaknesses in learning and teaching for
a given course, however it is clear that sentiment analysis has proved the most productive and informative
part of the process.
This study adds to a growing body of research investigating the use and analysis of feedback tags as an
approach to providing feedback for programming assignments. Whilst the process described could be used
as a means of analysing feedback delivered in different formats, it is the ability for feedback tags to be reused
in different contexts that enables detection of how widespread problems are within a cohort.
A possible extension to this research could look at profiling students based on their feedback tags. For
example do students with a low attainment have a particular set of tags or tags of a particular theme? Can this
be used to target areas of weakness in their learning?
7. REFERENCES
Burn, A. (2008) Thematic Analysis of group software project change-logs. In the 10th Higher Education
Academy Subject Centre for Information and Computer Sciences Conference. Liverpool Hope University.
Cummins, S., Burd, L., and Hatch, A. (2009) Tag Based Feedback for Programming Courses, SIGCSE
Bulletin 41, 4 (Dec. 2009), pp 62-65
Cummins, S., Burd, L. & Hatch, A. (2010) Using Feedback Tags and Sentiment Analysis to Generate
Sharable Learning Resources. In the 10th IEEE International Conference on Advanced Learning
Technologies. Sousse, Tunisia, (to appear)
Gee, T. C. (1972) Students' Responses to Teacher Comments. Research in the Teaching of English, 6, 212-
221.
Gillam, L., Qin, G., Bush, D. & Newbold, N. (2009) Automating Feedback: The CAFEX2 Project. In the10th
Higher Education Academy Subject Centre for Information and Computer Sciences Conference. University of
Kent, Canterbury.
Laurillard, D. (1993) Rethinking University Teaching: a framework for the effective use of educational
technology. Routledge, London
National Centre for Text Mining, NACTEM (2010), http://www.nactem.ac.uk/, Accessed 22/04/2010
82
Novak, J., Gavrin, A., Christian, W. & Patterson, E. (1999) Just-in-Time Teaching: Blending Active Learning
with Web Technology, Addison Wesley
83
TURNITOFF – DEFEATING PLAGIARISM DETECTION SYSTEMS
Lee Gillam John Marinuzzi Paris Ioannou

Department of Computing Department of Computing Department of Computing
University of Surrey University of Surrey University of Surrey
Guildford, Surrey, UK Guildford, Surrey, UK Guildford, Surrey, UK
l.gillam@surrey.ac.uk jm00217@surrey.ac.uk pi00002@surrey.ac.uk
http://tinyurl.com/leegillam http://www.cs.surrey.ac.uk http://www.cs.surrey.ac.uk
ABSTRACT
Defeating plagiarism detection systems involves determining effective approaches for greatest impact at
lowest cost with the least likelihood of detection. Relatively simple techniques have been applied elsewhere
for avoiding plagiarism detection, demonstrated at the last HEA-ICS conference. In this paper, we discuss
defeats for seven plagiarism detection systems, including Essayrater, Seesources, PlagiarismDetector, and
the popular Turnitin. We report on initial results of human experiments undertaken on visual similarity to
assess the risk of human detection of changes. The systems evaluated are variously susceptible to sufficient
numbers of small alterations to characters or words in the text Our results suggest, at minimum, to use at
least 2 such systems in combination to reduce the likelihood of failed detection and increase the difficulty for
the determined, and yet somehow lazy, plagiarist – otherwise, the discovery and dissemination of simple
defeats for plagiarism detection software may mean that we may as well just “Turnitoff”.
Keywords
Plagiarism, detection, defeat, visual similarity, risk.
1. INTRODUCTION
Plagiarism detection becomes a key distraction from assessing written work, with an inherent cost in
investigating suspicious material. While we actively and strongly discourage “cut+paste” or buying off-the-
shelf, pressure to perform makes such strategies a likely last resort. To help identify suspicious writing,
student work may be systematically run through plagiarism detection systems, attracting legal and ethical
discussion in the process, with outputs used as an indication of need for further (human) investigation. Here,
there is a danger of becoming reliant on the system. However, those knowledgeable of detection strategies
may also become adept at avoiding risk of detection by suitable adaptation of the sourced material, bringing
the material below any arbitrary threshold of suspicion. Techniques such as essay spinning [4] may be
effective if the machine translation system produces results that are sufficiently divergent from the original –
there may be an advantage to using machine translation systems that produce lower quality results. However,
subsequent efforts required to rewrite such spun material may be significant, and may also introduce the risk
that the rewrite lowers the required divergence. Techniques requiring less subsequent effort involve thesaural
substitution and character substitution [2] and can be highly effective against plagiarism detection systems
because of algorithms that are either highly sensitive to very minor variations, or rely on a specific string span.
In this paper, we discuss simple techniques of character and word substitutions that we have used, following
on from [2] and [4] to variously defeat a reasonable number of plagiarism detection systems. The systems
evaluated are Plagium, Seesources, PlagiarismDetector, Plagiarism Checker, EssayRater, Plagiarism Detect
and Turnitin. As systematic changes should also attempt to avoid suspicions being raised on reading, we
have also considered that visual (and semantic) similarity should be retained in character and word changes,
and in this paper we assess the risk of human detection in relation to character changes. Visual similarity can
be an issue for both website domain names [6] and webpage/site phishing [5]; semantic similarity is beyond
the scope of this paper. Part of our work involves developing a prototype to automate the generation of test
texts that will suggests extent of changes required to pass a detection threshold with a known risk of
detection. We have identified systems that are not defeated in the same way as the apparently de facto
Turnitin, so some of these might be used, in the way that they were designed, in parallel with Turnitin in order
to improve the likelihood of plagiarism detection. Consequently, those wishing to risk plagiarism and
attempting to avoid detection might find the efforts required to defeat such systems becoming increasingly
costly.
84
2. PLAGIARISM IN THE WILD
There have been a variety of high profile cases of plagiarism in recent years. The Downing Street “dodgy
dossier” is perhaps one of the most significant. The dossier was highly similar to an article by Ibrahim Al
Marashi, “Iraq’s Security and Intelligence Network: A Guide and Analysis” published in September 2002 by
the Middle East Review of International Affairs (MERIA), which the journal editor has suggested, possibly on
reflection, endorses the quality of their published articles1. An example of the similarity is shown in Table 12
with differences identified in bold; even “grammatical mistakes made on the internet version ended up in this
February 2003 document”3.The case is further discussed on the FamousPlagiarists.com /
WarOnPlagiarism.org website, amongst others.
Marashi British Intelligence
Its internal activities include: Its internal activities include:
Spying within the Ba’th Party, as well as other monitoring the Ba’th Party, as well as other political
political parties; parties
monitoring other grass roots
Suppressing Shi’a, Kurdish and other opposition suppressing Shi’a, Kurdish and other opposition
counter-espionage counter-espionage
targeting threatening individuals and groups inside targeting threatening individuals and groups inside of
Iraq Iraq
spying on foreign embassies in Iraq and monitoring foreign embassies in Iraq
foreigners in Iraq monitoring foreigners in Iraq
maintaining an internal network of informants maintaining an internal network of informants
Table 1: Similarities and differences (in bold) for fragments from the Marashi article and the dodgy dossier
The FamousPlagiarists.com / WarOnPlagiarism.org website features further alleged, and proven, cases of
plagiarism in a variety of professions; amongst these are, in no particular order, Al Gore, Martin Luther King,
Osama Bin Laden, J.K. Rowling, Madonna and Britney Spears. The site also includes an unfortunate number
of academics4. Most notable of these is Bindu Ganga, who was eventually sacked by her University (Argosy)
and stripped of her (plagiarized) doctorate5 but was allowed to submit a new project for, and be awarded,
another doctorate. This case is notable because Argosy initially defended Ganga against accusations made
by Argosy student Marla Decker, who was “disciplined, in part because she pushed the charge”6 until her
allegations of plagiarism were substantiated by a Turnitin originality report.
While plagiarism cases are typically focused on quantities of duplicated text, there is an intriguing case of a
phantom paper [3]. Here, there have been may citations by academics and practitioners of a supposed
landmark reference in information retrieval, but never been an article with that particular title. At minimum, this
suggests that referring to spurious articles will help to identify the diligent researchers.
In part, the three cases discussed above set unfortunate precedents. Acts of plagiarism that appear
acceptable and go unchallenged, or are defended at senior levels, and may even be described as
unintentional/mistakes7 with second chances offered, make penalties for small or moderate quantities of
plagiarism in educational settings harder to justify.
2.1 A note on why plagiarism detection systems can fail

Plagiarism detection systems typically rely on the selection of (a specific set of) strings of a particular length
from a source document, and the relationship of these strings to a set of target documents. Strings may be
selected as a number of words or number of characters, and be sampled from the source document or taken
in various increments across the entire document. Frequent occurrences of similar patterns in target
documents adds to the indication of suspicion. These strings may be hashed, and hash collisions used to
indicate similarities; low probability of collision is required to avoid false positives.
1
MERIA Editor’s response posted at: http://meria.idc.ac.il/british-govt-plagiarizes-meria.html (Accessed 20 May 2010)
2
Tony Blair, Colin Powell and the Case of the “Sexed Up” British Intelligence Dossier - A Linguistic Analysis by Dr. John P.
Lesko: http://www.famousplagiarists.com/MLSsexedupdossier.ppt (Accessed 20 May 2010)
3
See footnote above.
4
http://waronplagiarism.org/academia.htm (Accessed 20 May 2010)
5
See, also, the (somewhat similar) articles at http://www.ask.com/wiki/Argosy_University#Plagiarism_controversy and
http://en.wikipedia.org/wiki/Argosy_University#Plagiarism_controversy (Accessed 20 May 2010)
6
http://www.ibhe.state.il.us/newsdigest/newsweekly/010408.pdf, p18, (Accessed 20 May 2010)
7
“Ganga admits there were many mistakes in her paper. [..] she said her errors were "unintentional" and claims she
shouldn't be seen in the same light as plagiarists who take credit for the work of others” -
http://www.ibhe.state.il.us/newsdigest/newsweekly/010408.pdf, p17, (Accessed 20 May 2010)
85
Requiring exact string matching, rather than computing string distance, will immediately be susceptible to
character or word variations within the string. Such variations would carry through to use of hashes.
Modifications within a sufficient number of such strings maybe enough to avoid detection, and the longer the
string, the fewer changes required to the text as a whole. Word changes, insertions, or deletions, can be
detected if using overlapping strings (“shingling”). Shorter string lengths, smaller increments for shingling, and
producing patterns against the entire text will all tend to favour detection; consequently, systems tuned for
efficiency by using larger strings, fewer samples, and small or no overlaps, are likely to be less accurate.
2.2 Defeating plagiarism software?

There appear to be relatively few software applications directly “marketed” at defeating plagiarism detection
systems. However, searching for “article rewriter” reveals a number of software applications8,9. A notable free
piece of software which undertakes the latter, but with a title towards the former, is the “Anti-anti-plagiarism
detection system” (AAPS)10, a Perl script that contains a number of substitutions (and their reverse) intended
to maintain “grammer [sic.] and spelling”. A sample of AAPS substitutions is shown in Table 2. The
substitutions do not necessarily have the coverage required to make a significant impact on a document: run
against the “dodgy dossier”, the full text of which already had several 100% hits at a number of Universities in
Turnitin, the reduction was a mere 2%. The extent of impact possible due to other article rewriters remains to
be evaluated. However, replacement by alternative phrases, use of “intentional” mistakes, systematic use of a
thesaurus, and similar resources11 could increase the degree of change substantially. The key to this, with
reference to the section above, is knowing how many changes to make, and where.
Phrase Replacement Phrase Replacement
in other words alternatively then there is next comes
a plethora of many took part in participated in
Table 2: Example replacements in AAPS
3. EXPERIMENTS IN DEFEATING PLAGIARISM DETECTION SYSTEMS

Three experiments were undertaken. The first involving character substitutions within Unicode, the second
using thesaural substitutions, and the third assessing the chances of (human) visual detection of character
substitutions. For the first and second experiments, a 266 word text12 was used; for the third, we made use of
20 intelligent humans for segments of a relevant Internet Engineering Task Force (IETF) document13.
3.1 Experiment 1: Character Substitutions

A set of character substitution tests was formulated around changing, for example, Latin “e”, with code
U+0065 to Cyrillic “Ie” with code U+0435. These characters look incredibly similar, even at 20pt font (Table 3).
Latin e Cyrillic e Latin a Cyrillic a Latin o Greek o
(U+0065) (U+0435) (U+0061) (U+0430) (U+006F) (U+03BF)
e е a а o ο
Table 3: Character similarities in Unicode
The tests were:
 Test 1: Change Latin o,c,p,y,a,e to similar Cyrillic and Greek letters;
 Test 2: Change Latin i and v to similar Vav and Greek letters;
 Test 3: Test 1 plus change Latin A,B,C,E,H,I,J,K,M,N,O,P,T,Y to similar Greek and Cyrillic letters;
Results for the seven systems show six are defeated by these changes (Table 4). Plagiarism Detect appears
not to recognize plagiarism for the original article, suggesting its database does not include this. The only
8
The free Article Changer: http://www.articlechanger.net/ (Accessed 20 May 2010)
9
The Best Spinner, priced at US$7: http://thebestspinner.com/ (Accessed 20 May 2010)
10
http://sourceforge.net/projects/aaps (Accessed 20 May 2010)
11
For example, the Plain English Campaign’s A to Z of Alternative Words http://www.plainenglish.co.uk/files/alternative.pdf
(Accessed 20 May 2010)
12
266 words of “History of London, History of England a unique and stimulating site”.
http://www.historyofengland.net/content/view/119/49/, (Accessed 20 May 2010), starting at “London 1500 Years Ago”
and ending “made London their winter HQ”.
13
Request for Comments (RFC) 4690 on Internationalized Domain Names (IDNs), available at: http://www.rfc-
archive.org/getrfc.php?rfc=4690 (Accessed 20 May 2010).
86
system that appears to perform well is Turnitin, though Test 2 suggests it may be possible to push a text
below a threshold if it were only a number of percentage points above it.
Plagium Seesources Plagiarism Plagiarism EssayRater Turnitin Plagiarism

Detector Checker Detect
No change 100% 100% 100% 100% 100% 100% 8%
Test 1 0% 0% 0% 0% 0% 100% 0%
Test 2 0% 0% 0% 0% 0% 60% 0%
Test 3 0% 0% 0% 0% 0% 99% 0%
Table 4: Plagiarism Detection Systems Tests 1 to 3
3.2 Experiment 2: Word Substitutions

Words were manually substituted using the most suitable synonyms at every 5th (Test 4), 6th (Test 5), 7th (Test
6), and 8th (Test 7) word of a sentence using a combination of thesaurus.com and Microsoft Word. Such a
process could be automated, to an extent, using an electronic resource such as Wordnet14; post-editing may
still be needed to retain meaning. Results for the seven systems show five are defeated by changes at the 5th
word, though two recover at the 6th word, and two systems remain defeated by substitutions at 8 words (Table
5) - Plagiarism Detect was starting from a low threshold anyway. These results show the likely string span
used by the systems that recover from the defeat. Interestingly, Seesources appears to be unaffected by
these changes, while PlagiarismDetector has a change in performance relative to length.
Plagium Seesources Plagiarism Plagiarism EssayRater Turnitin Plagiarism
Detector Checker Detect
Test 4 0% 100% 85% 0% 0% 0% 0%
Test 5 0% 100% 93% 0% 100% 0% 0%
Test 6 0% 100% 98% 0% 100% 0% 0%
Test 7 0% 100% 98% 100% 100% 87% 0%
Table 5: Plagiarism Detection Systems Tests 4 to 7
3.3 Experiment 3: Visual Similarity

A small crowdsourcing experiment was undertaken with a group of 20 undergraduate students, predominantly
from the Department of Computing, who were each briefed to detect as many changes as they could to a set
of paragraphs and given both the original and a revised version containing zero or more character
substitutions. For a set of substitutions, intuitively selected, we were looking for approximate detectability for
an informed, but moderately motivated, audience.
Replacement letters e-е h-һ v-ν l-Ɩ u-υ i-ί p-ρ k-ĸ
Found 0/20 0/20 3/20 4/20 6/20 9/20 12/20 14/20
Risk of detection 0% 0% 15% 20% 30% 45% 60% 70%
Table 6: Risk of detection based on rates of discovery for visually similar characters
To demonstrate these replacements, the first four, with up to 20% risk, have been applied to a segment of the
Marashi article - 16 replacements of “e”, 2 of “h”, 2 of “v” and 4 of “l” (Table 7). Visually, these appear highly
similar, and this is also likely to pass detection by most of the systems tested in this paper.
Marashi (original) Marashi (with substitutions)
Part Two gives up to date details of Iraq’s network of Part Two giνеs up to datе dеtaiƖs of Iraq’s nеtwork of
intelligence and security organisations whose job it is intеƖƖigеncе and sеcurity organisations wһosе job it is
to keep Saddam and his regime in power, and to to kееp Saddam and һis rеgimе in powеr, and to
prevent the international community from disarming prеνеnt tһе intеrnationaƖ community from disarming
Iraq. Iraq.
Table 7: A segment of the Marashi article with, and without, character substitutions.
For a small collection of recently published news articles15, we assessed the average number of changes
attributable to these substitutions to determine the average impact on a document of making such changes,
and hence to gain a broad understanding of risk versus alteration (Figure 1). Over 50% of words are changed
in an apparently risk-free manner by the first two substitutions. We are initially assuming 100% plagiarism, so
14
http://wordnet.princeton.edu/wordnet/download/ (Accessed 20 May 2010)
15
8 texts taken from BBC News, e.g. http://news.bbc.co.uk/1/hi/world/asia-pacific/8681833.stm
87
amounts already at a distance below this might well be brought under any threshold for suspicion. A similar
experiment could be carried out relating to the retention of meaning following thesaural substitutions.
100%
90%
80%
70%
60%
50% Average Reduction of
40% Plagiarized Content
30%
20% Risk of Using the
10% letters
0%
Figure 1: Risk of detection of substitutions versus proportion of words altered in the text (n=8).
4. CONCLUSIONS AND REMARKS

The experiments discussed demonstrate how the determined, but lazy, plagiarist could learn to avoid
detection. Most of the systems evaluated are susceptible either to character substitutions, or to systematic
word substitutions; some are susceptible to both. Given the relative ease of making such substitutions with
typical word processing software, manually or via a macro, or using or buying an article rewriter, reliance on a
single plagiarism detection system may be risky. Our results suggest that using Turnitin together with either
Seesources or PlagiarismDetector would help to avoid the weaknesses discussed. However, if such systems
used certain pre-processing strategies, likelihood of detection could be improved. For example, an initial test
for contiguous strings being within the same Unicode code point ranges could flag insertions, and a simple
language model may act as confirmation; systematic thesaurally-driven reduction of both source and targets
could deal with thesaural variations. Without such pre-processing, the burden of detection remains with the
human reader, who has to become increasingly adept at spotting stylistic variations and any other flags
relating to such kinds of trickery as may have been used in order to avoid detection.
While evaluations of plagiarism detection systems are reported [1], the majority have not assessed the
specific weaknesses of these systems. Relatively recently, an international workshop hosted the first
international competition on plagiarism detection within the 3rd Workshop on Uncovering Plagiarism,
Authorship and Social Software Misuse (PAN-09). The competition involves the detection of two kinds of
plagiarism within a reasonably large text corpus, and some 47,000 (constructed) plagiarized articles. At the
nd
time of writing, the 2 competition is underway (and involves a second task on detecting vandalism in
Wikipedia). Despite the number of detection systems, there seems to be relatively little apparent participation
by vendors who we would expect to benefit from such benchmarking by proving the efficacy of their systems,
and using such competitions to address the weaknesses.
5. ACKNOWLEDGEMENTS
We are grateful to Neil Cooke for the pointer to [3], and for reviewing a draft of this paper.
6. REFERENCES
[1] Bull, J., Collins, C., Coughlin, C., Sharp, D., Technical Review of Free Text Plagiarism Detection
Software, JISC, (2000).
[2] Culwin, F., The Efficacy of Turnitin and Google, Proc. 9th HEA-ICS Conference, Kent, U.K.(2009)
[3] Dubin, D., The most influential paper Gerard Salton never wrote. Library Trends, 52-4, (2004)
[4] Lancaster, S. and Clarke, R., Automated Essay Spinning, Proc. 9th HEA-ICS Conference, Kent, U.K.,
(2009)
[5] Wenyin, L., Huang, G., Xiaoyue, L., Min, Z., and Deng, X., Detection of Phishing Webpages Based on
Visual Similarity, Proc. World Wide Web Conference, (2005).
88
[6] WIPO Arbitration and Mediation Center, Reuters Limited v Global Net 2000, Inc., Case No. D2000-0441,
WIPO (2000)
89
TRANSITION, ENGAGEMENT AND RETENTION OF FIRST YEAR
COMPUTING STUDENTS
H. M. Sayers M.A. Nicell A. Hinds

University of Ulster University of Ulster University of Ulster
Magee Campus Magee Campus Magee Campus
Londonderry Londonderry Londonderry
hm.sayers@ulster.ac.uk ma.nicell@ulster.ac.uk a.hinds@ulster.ac.uk
ABSTRACT
Student retention, engagement, the first year experience, and transition from secondary to higher education
are global issues on which much research has been undertaken in many countries. This paper presents the
results from a detailed analysis of these issues with 106 first year (including foundation year) computing
students in semester 1 of the 2009-10 academic year. In an effort to fully understand the first year student
psyche, an “inside” approach was adopted where a graduate Research Assistant (RA) was employed to
integrate with the first year cohort throughout semester 1 both inside and outside class time. Data on student
backgrounds (social and educational), entry qualifications, attendance and progression (in terms of semester
1 results), and student opinions on all aspects of the teaching and learning environment was collected and
analysed. Some surprising results, which challenged our initial assumptions about first year computing
students and retention issues, were found which highlight a clear need for a dynamic approach to initiatives to
ensure that they address the specific needs of any new student cohort based on social, educational, and
subject-specific factors.
Keywords
Transition, Retention, Engagement.
1. INTRODUCTION
The purpose of this study was to accurately identify the transition and engagement experiences of first year
computing students within the School of Computing and Intelligent Systems (SCIS), and to consider retention
activities within the School based on the findings when analysed along with semester 1 attendance and
performance statistics. Student transition and retention are major priorities within the University of Ulster
(Ulster) as a whole, and over the last number of years SCIS has made significant efforts to improve its
performance in these areas through a number of initiatives including small group tutorials and enhanced and
extended inductions. Progress has been made, feedback has been positive and retention figures have
improved. Indeed, SCIS has outperformed all other schools within the Faculty of Computing and Engineering
in its retention statistics for semester 1 (2009-10). However, there is little room for complacency and much still
to be done both in the identification of the root of the problems and in strategies for tackling them. This paper
presents the initial results from an “inside” perspective of current Computing students.
2. BACKGROUND
Over the last two decades, as widening participation has become promoted within Higher Education, student
cohorts have undergone fundamental changes and this has presented tertiary level institutions with new and
often exciting challenges. Students come from a diverse range of social and educational backgrounds with
differing expectations and skills. This fundamental change in the nature of student cohorts has been
accompanied by close scrutiny from governments on continuation figures. Indeed, “Levels of student retention
are an important indicator of institutional health” (Bourn, 2007, pg.15). As a result of these changes, there
have been many initiatives, nationally and internationally, to enhance the first year experience, all aimed at
ensuring successful student progression and success. Naturally, some level of attrition is unavoidable.
Students often, for example, withdraw from a course after careful consideration, or due to influences beyond
the control of either the institution or the students themselves.
90
Nutt (2008) summarises the key challenges involved in student retention and engagement under four main
criteria: supporting diverse students; engaging and motivating students; socially integrating students; and
changing and enhancing institutions to provide more supportive and engaging environments.
Initiatives to address transition and retention, evidenced in pedagogic literature, can easily be slotted into one
of these categories. It should be noted, however, that different subject areas often have different retention
rates and different issues which need to be addressed. Strategies that retain one group of students may be
detrimental to the integration of another group (Squire, 2005). Squire (2005), in her review of strategies for
student retention in the UK, states that Higher Education (HE) institutions must choose strategies that match
their “mission” and that attrition is caused by multiple factors which no single strategy can solve on its own.
The transition from secondary to tertiary education is a move from a controlled learning environment to a
more autonomous environment. Students from a more academic background may be more comfortable with
the transition to independent learning than those from a vocational background. Bentley (2006) in her PhD
thesis at the University of Wolverhampton, in which a longitudinal study of computing students was carried
out, found that students from a vocational background performed better in modules with a high level of
continuous assessment and in modules which involved smaller groups and more tutor contact (Bentley,
2006). Assessing the impact of individual initiatives reported in the literature, however, is difficult when one
considers the number of variables involved (educational, social, motivational, subject –related) which may
affect the result.
Combined subject degrees and Mathematical and Computer Sciences are reported by the National Audit
Office to have the lowest continuation rates among all undergraduates (Bourn, 2007). Computing courses are
constantly changing as they strive to stay up to date with technological advances and they generally include a
broad range of topics. Yet it is due to this very diversity and the broad range of uses and applications for
computing, that opportunities must be provided for students from a wide variety of backgrounds and with
vastly different skills to undertake computing courses and take advantage of the many attractive career
prospects within the industry (McGettrick et al., 2004). Students enrolled on a computing degree can be
motivated by different factors but, however motivated, they will inevitably encounter modules in which they are
expected to learn and to apply problem-solving and analytical skills to the resolution of practical problems and
not every student finds this process easy to master (Bentley, 2006). Boyle et al. (2002) claim that a
qualification in Information Technology may not give the student the correct information about what to expect
from a computing degree course.
2.1 The University of Ulster: Transition and Retention in SCIS

SCIS is one of three Computing schools within the Faculty of Computing and Engineering at the University of
Ulster. The School offers courses, BSc Hons and BEng Hons, in both full-time and part-time modes in
Computing and welcomes 120-150 new entrants each year. Specialist routes are offered in multimedia,
electronics, and computer games, in addition to a variety of combined courses with one-third of the course in
another academic discipline. The School also offers an Integrated Foundation Year (IFY) for those students
who do not reach the required entry points for year 1. 260 UCAS tariff points at A Level or equivalent, and
GCSE Maths and English at Grade C are required for entry to all year 1 courses, with only the BEng degrees
specifying one science-based A Level. IFY entry requires 120 UCAS tariff points at A Level for entry.
In recent years SCIS has taken significant steps to address poor retention rates and has managed to improve
figures. Attendance monitoring and follow-up have been closely pursued with the establishment of a First
Year Teaching team meeting weekly to consider students at risk and to take speedy action when necessary.
Other initiatives within the School have included small group weekly tutorials and enhanced, extended
inductions. Induction activities were reviewed and improved to extend over the full year in an effort to avoid
information overload at the start of the year, and a social outing was organised for the first time this academic
year in an effort to assist with social integration. Feedback from these initiatives from both students and staff
has been encouraging and the initiatives have now become established practice.
This research study (IN-TUNE: Investigating Transition to Undergraduate Education) was funded by the
Centre for Higher Education Practice (CHEP) within Ulster.
3. THE EXPERIMENT
The experiment was carried out over the 12 weeks of semester 1, 2009-10. The RA accompanied students to
lectures and laboratory classes and provided a variety of opportunities for both informal and formal feedback,
including interviews, focus groups and questionnaires. There were 106 participants (79 male and 27 female)
enrolled on the courses outlined in Figure 1.
91
3.1 Data Analysis
The student profile statistics were combined with statistics on student attendance and performance. Results
are discussed in the following sections.
Single Honours Courses
BSc Hons Computer Science 32
BEng Electronics and Computer Systems 5
BSc Hons Multimedia Computer Games 6
BEng Hons Computer Games Development 12
Integrated Foundation Year 22
Combined Subjects
BSc Multimedia, Computing and Design 14
BSc Computing with Minor Subject 3
BA Hons Creative Technologies 12

Figure 1: First year course enrolment 2009-10
3.1.1 Attendance
Attendance has consistently been a major issue in addressing student retention. Overall attendance for all
modules for first year and Integrated Foundation Year (IFY) students in semester 1 of 2009/10 is shown in
Figure 2.
92
Despite rigorous attendance monitoring and follow-up, only 29 of the 106 students attended between 81-
100% of all classes with much better attendance results for the IFY. For the individual modules, the highest
attendance rates in year 1 (81% - 100%) ranged between 23% and 39% (Mathematics I: 39%, Introduction to
Computer Games: 39%; Algorithmic Programming I: 33%; Systems Analysis and Data Management: 23%).
The Mathematics and Programming modules had a high level of continuous assessment that could account
for higher attendance recorded, but attendance was similar on the Games module without this level of
continuous assessment. The Games module is the only course-specific module in year 1 with consequently
smaller numbers, but feedback from student focus groups and interviews suggests that these students were
also strongly motivated in their choice of course. The total class size for the IFY was again small (22) for all
modules and the highest attendance rate (81-100%) was much higher than for year 1 modules (Introduction to
Mathematics for Computing: 45%; Introduction to the Internet and Multimedia: 59%; Introduction to
Programming: 63%).
In line with published research and common opinion, we hypothesised that poor attendance correlates with
poor performance, and analyses of variance (ANOVAS) followed by post hoc procedures were carried out on
the data to ascertain if the attendance pattern had a significant effect on first semester results with overall
attendance as the independent variable and overall average result as the dependent variable. As expected,
attendance was found to have a statistically significant main effect on overall performance with poor
attendance relating to poor performance (p=0.000, F[3,61] = 12.208). The post hoc analyses to compare
groups within the data confirmed this result with strong correlations between attendance and results
groupings. The timetable, the change towards more independent learning and more freedom, fear of difficult
subjects and lack of motivation are just some of the factors reported in the literature in relation to poor
attendance, yet it is difficult to pinpoint which factor requires the most attention. Students’ comments from
informal interviews and focus groups were very varied in relation to attendance. For example, one student
considered what he described as “a five hour gap” between classes on a Monday a valid reason for not
attending the later class, while another “liked to be able to choose when to come in”. SCIS has implemented a
policy of weekly monitoring and follow-up which has assisted in the identification of students at risk of failure,
but the first year team remains conscious of the danger noted by Huxham (2005) that “that zealous policing of
93
attendance can be counter-productive”. Various reasons for non-attendance were revealed at the focus
groups and interviews but the RA was convinced that motivation and support were key factors in the debate.
3.1.2 Employment
It was surprising to find that only 40% of the total cohort (34 in year 1, and 8 in IFY) was in paid employment
in addition to their studies. 30% of these working students work between 5-10 hours, per week, with the
majority working between 10 and 20 hours per week. ANOVAs and post hoc analyses were used to test the
hypothesis that the more hours worked, the lower the students’ attendance and performance would be. The
results disproved the hypothesis in that there was no significant effect of work on either performance (p =
0.512, F[1, 65] = 0.434) or attendance.
3.1.3 Educational Background
54% of participants entered with A Level qualifications, with other entrants having a range of BTEC National
Diploma, Irish Leaving Certificate, HND and Access course qualifications. In addition, some year 1 students
had progressed from the Integrated Foundation Year in the previous academic year. The average UCAS tariff
point entry profile was higher than the tariff required for entry for both year 1 and IFY entrants (277 and 151
respectively).
The general perception of secondary level education within Northern Ireland is that Grammar Schools contain
the most academically capable pupils. Therefore, we put this to the test by carrying out an ANOVA with type
of school as the independent variable and overall result as the dependent variable once again. This
hypothesis was again disproved (p=0.724, F[2,65] = 0.325). The secondary education sector of entrants did
not significantly affect overall first semester performance. The same analysis against results was carried out
for A Level entrants since previous research has shown that such entrants perform better (Bentley, 2006).
Again, however, no significant difference was found in A Level entrants’ results (p=0.185, F[1,68] =1.792).
3.1.4 STEM Subjects and Combined Degrees

SCIS course provision consists entirely of Computing degrees and Combined Subject degrees, noted in the
National Audit Office report (Bourn, 2007) for the highest non-continuation figures. An analysis was carried
out in this study to compare performance for these groups in year 1 (55 single honours students and 29
combined degree students). Interestingly, no Combined Subject students reached either the highest (>80%
achieved by 12% of the single honours students) or lowest levels of performance (<20% achieved by 7% of
the single honours students), but a higher percentage (18% combined vs 12% single honours) failed the
semester with an overall average below 40%.
Mathematics and Programming have traditionally been the two subjects with which students struggle, and this
has been evidenced in performance statistics within SCIS over the years. Student feedback, however, clearly
indicates that Mathematics in particular continues to be a central concern despite much effort and improved
statistics. One student commented that he considered a C in GCSE Maths as an entry requirement
insufficient for coping with the module, although he did admit that he was managing well.
3.1.5 Teaching Delivery

In relation to module delivery, there were many comments on the differences from secondary level in terms of
lectures and the move to more “independent learning”. Even 2-hour lectures were considered too long, and
the general consensus was for more practical/tutorial classes instead. One student summed it up, saying, “I
don’t really learn that well in the lectures, just being talked to, rather than doing something like in a practical”.
Many students felt that lectures were particularly inappropriate for teaching Maths with comments such as “we
need to do more questions” and “people who haven’t done it before can’t grasp it (in lectures)”. Comments
included liking “the freedom”, being “treated like an adult”, and “the more relaxed atmosphere” as opposed to
being “constantly told what to do next” (at school), although one student described the change in the following
way: “Here, you’re given quite a lot of cord to hang yourself”!
3.1.6 Socialisation
Mackie (2001) states that, for students who were having doubts about courses, a lack of social integration
was often found to be a major contributing factor. 62% of participants in this study have friends attending the
Magee Campus and 31% have friends enrolled on the same course. Despite this, however, there were many
comments on the lack of social opportunities in which to mix with other students and staff. Almost two-thirds of
the participants live at home and travel to the university daily (63%) which makes it more challenging to
address the issue of socialisation.
94
4. CONCLUSION
Transition and retention are clearly complex and evolving challenges for Higher Education. There are multiple
factors to consider when addressing the issues, both generic and subject-specific. The research carried out
for this study has considered the issues from an “inside” perspective and the findings have challenged many
of our assumptions and identified the need for continued initiatives with constant review and adaptation of
activities and practice to address the needs of each individual cohort.
5. REFERENCES
Bentley, H. (2006) Improving the achievements of non-traditional students on computing courses at one wide
access university, PhD thesis, Wolverhampton University,
http://wlv.openrepository.com/wlv/handle/2436/14640?mode=simple (accessed 18/05/2010).
Bourn, J. (2007) Staying the course: The retention of students in higher education, Report by the Comptroller
and Auditor General, National Audit Office, July 2007
http://www.nao.org.uk/publications/0607/student_retention_in_higher_ed.aspx (accessed 18/05/2010)
Boyle, R., Carter, J. and Clark, M. (2002) What Makes Them Succeed? Entry, Progression and Graduation in
Computer Science. Journal of Further and Higher Education. 26(1), pp. 3 – 18.
Huxham, M. (2005) Extended Induction Tutorials for ‘At-Risk’ students, Napier University, STAR Project,
http://www.ulster.ac.uk/star/induction/extended_induction.htm (accessed 18/05/2010).
Mackie, S. (2001) Jumping the Hurdles – Undergraduate Student Withdrawal Behaviour, Innovations in
Education and Teaching International. 38(3), pp. 265 – 276.
McGettrick, A., Boyle, R., Ibbett, P., Lloyd, J., Lovegrove, G., and Mander, K. (2004) Pre-University Issues, in
Proceedings of Grand Challenges in Computing, Newcastle, March 2004.
Nutt, D. (2008) Getting it Right from the Start: Enhancing the First Year Learning Experience, in Proceedings
of TIC, web3.tarc.edu.my/v1/tic2008/abstract/No%20228FA.pdf (accessed 18/05/2010).
Squire, M. (2005) Successful Strategies of Student Retention and Success: What we can learn from the UK,
Sabbatical Project, School of Advanced Technology, Algonquin College, 2005,
www.algonquincollege.com/student-retention/current_projects.htm (accessed 18/05/2010).
95
Re-thinking Student Induction
Mark Lee Gale Dawson
School of Computer Science Learning Development
University of Birmingham University of Birmingham
B15 2TT B15 2TT
m.g.lee@cs.bham.ac.uk g.dawson@bham.ac.uk
ABSTRACT
The paper will report on a two year project to radically re-design our approach to student induction.While we
recognise that student induction is a process that occurs throughout the student life-cycle, our original
approach to the early induction of new udergraduate students was largely lecture-based and involved key
staff giving presentations on our expectations of the student cohort in terms of dilligence, work ethic and time
keeping etc. Student feedback on this approach tended to be poor. Worse we found it to be ineffective in
preparing our students for the work ahead and we believe contributed to poor retention figures in the first
year.
We believe, a key short coming was that we wasted the high levels of enthusiasm and motivation most
students have directly prior to arriving at university and this lowered their expectations not only of the content
of the degree but also of what we expected them to achieve in their first year. Therefore we redesigned our
early induction to consist of a full week prior to the start of term (week 0) which we called "Welcome Week."
This week consisted of a series of intensive activities centred around teamwork and enquiry-based learning.
Student feedback after the first year of "Welcome Week" was extremely positive.
Our approach to Welcome Week has been widely disseminated across campus and is now the "standard
model" for facilitating the early academic and social engagement of students at our institution. In particular it
has been adopted by other science and engineering disciplines. Therefore we will also discuss some issues
concerning adapting our programme of computer science-based activities to other disciplines.
Keywords
Induction, Student engagement, First year experience, Retention.
1. INTRODUCTION
The transition from school/college to university life is one of the most challenging events in a student's life
and can often mean the difference between successful completion of their studies and early withdrawal from
studies. Therefore we must take student induction seriously and where possible improve on current practice.
Many students withdraw from higher education during the first few weeks of study. This suggests the
importance of starting to build institutional and disciplinary engagement and commitment as early as
possible. This can be achieved by helping students to understand the programme they are able to participate
in, developing communities of learners and encouraging the formation of friendships and social networks’
(Crosling, 2007:173). This paper will consequently report on work we have done over two years to redesign
how we approach the early induction process for Computer Science students at Birmingham.
Induction is a time and resource consuming activity. However, the advantages of good student induction
should not be underestimated. If we can scaffold and facilitate the transition from school/college to university
approaches to learning (e.g. independent and enquiry based) at the very start of the degree then not only will
the first year experience be improved but learning and teaching will be facilitated throughout the degree. We
argue therefore that time and effort spent during early induction is very worthwhile.
2. BACKGROUND
Much of the research on improving student retention suggests that students are more likely to continue and
succeed if they are engaged in their studies and have been encouraged to develop networks and
relationships with their fellow students (eg Crosling et al, 2008). It suggests that this is more likely to happen
where the curriculum is innovative, responsive to the diversity of students’ needs, and promotes academic
and social engagement as well as active learning.
This is done by:

 Considering alternative and innovative approaches to learning, teaching and assessment
96
 Integrating and embedding ‘study skills’ into core learning and teaching activities
 Personalising the First Year Experience
Wingate and others have argued that discipline-based pre-induction and early induction activities need to
prepare students for the transition to learning in HE. ‘A holistic, subject-specific approach is needed to
support all students in the complex process of learning to learn in higher education’ (Wingate 2007:391). This
is further confirmed by Shaw et al (2007) who argue that ‘embedding academic development in mainstream
teaching and learning’ is likely to ‘enhance student retention, progress and achievement’ (Shaw 2007:50).We
therefore need to:
 make our expectations explicit
 fill the gaps that many students have about studying in HE
 encourage students to ‘start engaging with discipline specific material at an earlier date’
 help students to understand the programme they will be participating in
 begin to develop a community of learners
 encourage the formation of friendships and social networks (peer support and personalisation)
QAA advice (QAA, 2008a) on the first year experience is to take a longitudinal and multi-layered approach to
enhancing the academic and social transition of students through facilitating the following key features:
 early understanding of HE-level learning
 formative assessment and feedback
 engaging and challenging learning experience
 enquiry-based, collaborative, small-group work, project-based learning model
 embedding progressive skills development
 reflective skills within a subject context (PDP)
 social networking and peer support
 early staff/peer interaction – e.g. personal tutoring
 introducing employability skills from the beginning of first year
 students understand the value of their degree programme in terms of their potential future careers
 discipline-based support for transition
 co-ordinated, responsive, and resourced
 involves collaboration between academic and administrative staff, and central services
Three key recommendations of the QAA report on enhancing the first year experience through the
personalisation of the first year experience are (QAA, 2008b pp83 – 86):
 Staff should give proactive support to first year students at an early stage and acknowledge that,
regardless of their backgrounds, they need time to adjust to the university learning environment.
Steps should be taken to treat students as individuals and to personalise their experience in small
but significant ways, for example by getting to know their names, taking attendance in large classes
or encouraging small group discussions. The importance of socialisation within this adjustment
process should not be underestimated.
 Students need time to learn how to learn, particularly if they are first generation entrants into HE and
have little social and cultural capital resource. Staff should make space in the curriculum to teach
students how to learn, for example to address issues of meta-learning, even though this is perceived
to take time away from discipline-based tuition. The beneficial consequences are seen in the
empowerment of students, the creation of autonomous and independent learners and the facilitation
of learning communities and systems of peer support.
 HEIs should develop and implement policies on, for example, attendance monitoring, PDP, buddy
systems and personal tutoring, in order to help students personalise their first year experience.
3. OUR INTAKE
Our first year cohort is typically around one hundred students over four different degree programmes and a
small number of half degree programmes (typically shared with Maths). 80% of our intake will take either our
standard Computer Science degree with the rest evenly split between more specialised degrees in Artificial
Intelligence and a major/minor degree involving Business Management. Like most computer science
degrees, we have relatively few female students compared to other subjects and approximately 25% of our
students will be from the local area.
Our previous approach to the early Induction process was accomplished through the use of several lectures
which focussed on deliverying the essential information required for study at Birmingham such as degree
programme information, progression, mitigations and plagiarism. We also invited a talk from the careers and
library services.
At the time we did not evaluate our induction procedure but there was unease as to its effectiveness. While
we cannot attribute a single cause, our drop-out rate for the first year (and in particular the autumn term) was
97
high compared to other computer science departments nationally. In addition, our experience was that there
was a significant minority of our first year cohort who were unmotivated and often relatively disconnected
from the School's culture.
At the same time, we received funding for a "Learner Independence" Project from the University of
Birmingham's Learning Development Unit. The focus of the project was the "first year experience" and
therefore we made student induction a focus. Our first step was analysing how our current first year viewed
their induction experience. We discuss the student voice in the next section.
4. THE STUDENT VOICE

Student feedback on the ‘Welcome’ they received at discipline level suggested that the following factors were
useful and others were a concern
 Students would like their induction session to include a social element
 Student would like the teaching/learning style of their programme, contact hours and the required
workload to be explained.
 Student would like the required standard of their assignments to be explained with specific examples
where possible. Consistent guidelines on how to format work for submission would be useful.
 Student mentors would like to be more involved in the planning of Welcome activities for new UGs
 Students would like the role of the personal tutor (and other staff) roles to be explained and to have
standard regular meetings with them.
 Complicated timetables can be a source of stress for students.
 Students like their welcome to include some motivational/ enthusiastic presentations about their
chosen subject, give an outline of their course
 Students would like to hear from successful alumni during their induction.
 Some induction activities are not helpful for students at the beginning of their course – library talks
when they have not been to the library, plagiarism talks when they don’t have any assignments to
relate it to.
4.1 Welcome Issues
From our evaluation, we identified several key areas and challenges to successful induction.
Enthusiasm and raising expectations
We believe that most students have a high degree of motivation and enthusiasm at the start of their studies
which can be wasted by a poor approach to induction. We wanted an induction which would maintain and
increase the enthusiasm the students have for the rest of their studies.
In addition, the induction should set a standard of work which challanges and engages all students and sets
a standard for the rest of the first year. Accomplishing this is difficult due to the differences in ability,
experience, motivation and learning styles among the student cohort upon arrival.
Pre-arrival
Our questionnaires reveal that students often spend an anxious summer vacation waiting for the beginning of
their university career. Moreover students often have questions about university life which are not necessarily
covered by the usual information we send. Despite this students also report that they feel swamped by the
amount of generic information sent to them over the vacation from central services and increasingly take
summer jobs prior to University. Therefore we felt the need to facilitate the opportunity for students to both
social network and find out information but in a manner which was not intrusive or time consuming.
Student Inclusion
All induction activities should be as inclusive as possible. However, students have a diverse range of
individual needs. For example, the following groups may have specific needs that should be addressed:
 International students (especially students without strong English language skills)
 Students with learning difficulties or disabilities
 Local students
 Mature students (and parents/carers)
 Black and Minority Ethnic (BME) groups
 Religious groups
98
 Other widening participation groups (eg those with no previous family involvement in HE)
Often it is difficult to predict some students’ specific requirements/needs until the student has arrived, but
‘what is evident is that changes stimulated or necessitated by a particular group of students benefit all
students, and thus have a positive impact on the achievement and retention of the whole cohort’
(Crosling,2008: 182)
A particular area of concern is local or "stay at home" students. Approximately 25% of Birmingham students
are from the local area and decide to stay at their family home or live locally during at least their first year.
We believe the current financial situation will result in increasing numbers of students staying at home for at
least part of their studies.Our internal data and experience confirms that students who live locally are at
higher risk of withdrawal compared to those students who live in halls. Our evidence also suggests that
some of this cohort are less likely to engage with the School. Holdsworth and Patiniotis (2004) ‘highlight the
difference in experience between halls and home students. Home students see university as an extension of
college – just a change of bus route – and have little interest in university life beyond attending lectures’
(Gorard,2007:90) Moreoever our previous induction activities overlook the needs of these students and
indeed campus wide activities often unintentionally exclude local students (for instance organised by the
Guild of Students during the evening).
Inclusion is also an issue for students of different cohorts within the same School. Ideally, we want to limit the
duplication of information but if there are differences in the curriculum this runs the danger of making some
content redundant for different degree cohorts. In the School, this has become less of an issue with the
implementation of a "common first year" for all single honours students in the School. However, this still
leaves a small group of students (those taking Business Management as a minor and students on the half
degrees) with a different timetable and programme requirements than the main cohort.
5. REDESIGNING OUR INDUCTION

We designed our Induction Week within the following framework:
Academic contact is restricted to half days throughout the Week
This allows for activities from the rest of campus - notably the Guild of Students and the Careers Service to
schedule events.
Each day to focus on the actual doing of a particular learning activity
For example, the intention was to schedule an actual computer programming laboratory session rather than
just give a lecture-based talk on the rights and wrongs of computer use.
Extensive use of student led activities via the use of student mentors
Our intention is that every formal session is followed by a student-led activity where the freshers are
encouraged to reflect on the session.
5.1 Scheduled Sessions
We scheduled the following activities during the week:
Local Students Session
We arranged a Local Students event on the first day of Welcome Week which we publicised by sending
information about the event only to students living in the West Midlands area. This event involved an
introduction to the School and key staff and some group acitivies (based on writing pseudocode for a
recursive solution to the Towers of Hanoi problem).
Treasure Hunt
Welcome Week is started off for all students with a treasure hunt across campus. We arranged the students
into groups and then gave them a series of riddles which described various parts of the campus. The
intention was that students would then discover key locations such as the library and main lecture theatres.
At the same time we were able to process student registrations.
Computer Lab session
We arranged a session in the first year labs. This was a challange since we cannot assume that all students
have done any prior programming and we did not want to introduce our key programming language (Java)
since that would be repeated in Week 1. In the end we used Logo & turtle graphics. In future we might try
another programming language or environment such as Greenfoot.
Industry Talk
In the first year of running Welcome Week, we invited an industry speaker to talk to our students taking a
minor degree in Business Management since our straight honours students need to attend a module fair on
the same day. However, this is only a small number of students and we encouraged our straight honours to
attend if they wished. To our surprise the entire year attended. This year we invited all students.
Introductory lecture on Computer Science
This lecture was designed to be both challenging yet not part of the programme curriculum. The intention
was to stretch the students but also allow for discussion of any difficulties the students faced. This session
was followed by a social reception in the School where students were encouraged to meet the rest of the
99
teaching staff and assistants.
5.2 Other Activities
In addition to the activities above, we arranged the following as student-led activities.
Web-based social networking
We invited all freshers to join a "CS Freshers" facebook group during the summer vacation which was run by
our student mentors (current second years). 95% of our first year joined this group and the group proved
useful in providing both an initial point of information and an oppotunity for social networking.
Timetabling
Rather than give our students a timetable for the Autumn Term we give them a blank timetable and with the
help of student mentors allow them to construct their own timetable with the explicit instructions that they
should block out spare time for coursework. This approach has shown a marked improvement on our
previous practices.
5.3 Evaluation
During the two iterations of Welcome Week we collected feedback from the students. In all aspects the
students rated our Welcome Week extremely highly. In particular we achived 90% agreement that the
Welcome Week both motivated students for further study and prepared students for lecture-based and
laboratory-based teaching. The students were also extremely positive about the role of the student mentors.
Our approach to Welcome Week has been widely disseminated across campus and is now recognised as
best practice. In addition, it has been adopted by at least four other academic units on campus.
6. CONCLUSIONS
Students have moved from being passive ‘recipients’ of our design for early induction to being partners in
planning and organising the process. By being inclusive and building on student’s strengths and motivation
the changes have facilitated early student academic and social engagement. Setting high standards from the
start of the course and challenging our students is not incompatible with scaffolding the transition to learning
in a research-intensive university.
7. REFERENCES
Bourn J. (2007) Staying the Course: The Retention of Students in Higher Education National Audit Office
http://www.nao.org.uk/publications/nao_reports/06-07/0607616.pdf
Crosling G., L.Thomas and M. Heagney (2008) Improving Student Retention in Higher Education: The Role
of Teaching and Learning, Routledge
Gorard S., N. Adnett, H. May, K. Slack, E. Smith, L. Thomas (2007) Overcoming the Barriers to Higher
Education Trentham Books
Harvey L. (2006) The First Year Experience Literature Review Higher Education Academy
http://www.heacademy.ac.uk/events/media/Lee_Harvey_handout.doc
HEFCE (2006) Widening Partcipation: A Review HEFCE http://www.hefce.ac.uk/widen/WPfinalreview.doc
Nutt D. (2006) Enhancing the First year Experience through the Curriculum Higher Education Academy
http://www.heacademy.ac.uk/events/media/Diane_Nutt_presentation.ppt
Shaw J., K. Brain, K. Badger, J. Foreman and I. Reid (2007) Embedding Widening participation and
promoting Student Diversity: What can be Learned from the Business case Approach?
Higher Education Academy http://www.heacademy.ac.uk/events/detail/towards_a_business_case_oct07
Thomas L., M. Cooper and J. Quinn (2003) Improving Completion Rates Among Disadvantaged Students
Trentham Books
Wingate U. (2007) A framework for Supporting Transition: Supporting ‘Learning to Learn’ in Higher Education
Quarterly Vol. 3 pp 391-405
Quality Enhancement Themes: (2008a) The First Year Experience – transition to and during the first year
QAA http://www.enhancementthemes.ac.uk/documents/firstyear/Transition%20-%20Final.pdf
Quality Enhancement Themes (2008b) The First Year Experience – Personalisation of the first year QAA
http://www.enhancementthemes.ac.uk/documents/firstyear/Personalisation%20-%20Final.pdf
100
WIDENING ACCESS, CPD AND WORK-BASED
LEARNING USING ASYNCHRONOUS DIALOGIC
LEARNING: ABERYSTWYTH, A CASE STUDY
Dr Judith Broady-Preston
Department of Information Studies
Aberystwyth University
Aberystwyth SY23 3AS
jbp@aber.a.uk
http://www.dis.aber.ac.uk/en/staff_in
fo.asp?id=jbp
ABSTRACT
In this paper, I describe and evaluate developments in distance learning education at the Department of
Information Studies (DIS), Aberystwyth University over a twenty-four year period set within the context of the
rapidly evolving, volatile qualifications landscape and an examination of trends within the information
profession as a whole. Delivered via asynchronous dialogic learning, distance education degrees offered by
DIS are designed to facilitate employability in the information professions, in addition to widening access to
higher education as a whole. The success of the suite of schemes in addressing these aims is evaluated,
together with their role in providing CPD opportunities for practising professionals.
Keywords
Distance learning; CPD; work-based learning; widening access; asynchronous dialogic learning..
1. INTRODUCTION
In 1986, the then College of Librarianship Wales(CLW) began offering a Continuing Professional
Development (CPD) Masters degree via distance learning, enabling senior managers within the information
profession to acquire formal recognition of work-based experiential learning, and to equip them for future
career development. Subsequently, the successor university department to CLW, the Department of
Information Studies (DIS), has expanded and developed its portfolio of degree schemes, being recognised
currently as the largest provider of distance education for the profession in Europe (Broady-Preston, 2010).
Since the inception of distance education, DIS staff have pioneered a range of teaching and learning
innovations, including being early adopters of asynchronous dialogic learning within the discipline;
experimentation with both technology and learning approaches is actively encouraged (Hill & Taylor, 2009).
The Aberystwyth experience is evaluated below, set within the context of an overview of relevant
developments in the wider information profession, together with those in the current fast-moving and volatile
professional/qualifications landscape.
2. CHANGING EDUCATIONAL AND PROFESSIONAL LANDSCAPE

The information profession has undergone profound and significant changes in relation to its qualifications
base, which are ongoing (Broady-Preston & Preston, 2007; Broady-Preston, 2007; 2009a; 2009b; 2010).
Barriers relating to qualification levels, to professional practice and to career development are being
circumvented or demolished; the challenges these present to a leading educational provider are identified and
examined here.
2.1 Skills, competencies and professional boundaries

There is widespread evidence that technological developments such as social networking tools are creating
not only the necessary acquisition of differing skills within the information profession, but also causing
professional boundaries to become more diffuse or to “blur” (Broady-Preston, 2009a; 2009c). Furthermore, it
is not only subject or professional domain boundaries which are becoming more diffuse; the previously hard
or impenetrable boundaries between professional and paraprofessional skills are softening. Furthermore,
challenges derive not only from technological developments, but also in the adoption and development of
more generic competency based frameworks for vocational education and training.
101
As Conway remarked in his survey of public library services “competency based recruitment is the
norm…formal qualifications…are regarded as only desirable” (2008, 20). This resonates with the findings of a
2007 survey conducted by the Chartered Institute of Personnel and Development (CIPD), which concluded
that “with the exception of a proportion of small private sector firms, competencies seem to be an accepted
feature of a modern organisation”(CIPD, 2010). Organisations develop competency frameworks in a number
of differing ways, and may derive these from the competency lists produced to support occupational
standards and the National and Scottish Vocational Qualifications frameworks.
Lifelong Learning UK (LLUK) is the body currently responsible for generating and reviewing National
Occupational Standards (NOS) and a review of the work of the NOS for Information and Library Services,
Archive Services and Records Management was announced 5 July 2010 (LLUK, 2010b).
2.2 Professionalism and professional identity

Originally a wholly graduate profession, one of the key professional associations for information work, the
Chartered Institute of Library and Information Professionals (CILIP) now allows individuals to progress from
paraprofessional to professional membership grades, recognising and rewarding work-based learning, in
addition to retaining the more formal, graduate routes into the profession (Broady-Preston 2007; 2009b). This
coincides with developments at a national level. The role and status of professional work in general came
under scrutiny during 2009 with the establishment of the Governmental Panel on Fair Access to the
Professions, chaired by the Alan Milburn (HM Government, The Cabinet Office, The Panel on Fair Access to
the Professions, 2009) A key finding of the Panel was that despite 1 in 3 jobs in the UK being in the
professions, many professions were becoming more socially exclusive over time not less, with a widening
social gap in entry to the top professions, and many professions being unrepresentative of modern society. It
remains to be seen whether or not the new UK Government, elected in May 2010 will act upon the findings
and recommendations of this Panel.
Moreover, as cited in an earlier paper, more radical thinkers are predicting the demise of both the concepts of
professionalism and indeed of qualifications per se, with individuals building their own portfolios of e-learning,
largely derived from their access to learning “in an open way on the Internet” (Edwards (2005) cited in Broady-
Preston, 2010, 73-74).
Looking beyond the narrower information landscape, thus far this century, the UK has seen a fundamental re-
evaluation of the relationship between formal education, employers and workforce development. Government
initiatives such as the establishment of Foundation degrees, the Treasury-led Leitch Review of Skills, the
establishment of the UK Commission for Employment and Skills (UKCES) amongst others are arguably
resulting in a shift from supply to demand led education and training (Broady-Preston & Preston, 2007;
Broady-Preston, 2007; 2009a; 2009b; 2010). Clearly there is a necessity for universities to offer a range of
educational opportunities as “the skills landscape is rapidly evolving, job roles are changing and ‘jobs for life’
have disappeared. Now, more than ever, people need to upskill, or increasingly, to reskill” (LLUK, 2010a, 2).
Furthermore “the speed and extent of the recession has caused significant challenges for countries and their
economies the world over. Many of the UK’s industries have been directly hit, meaning for the first time in
fifteen years young people are struggling to get work, and even experienced workers are facing uncertain
futures” (LLUK 2010a, 2).
3. DISTANCE LEARNING AT ABERYSTWYTH

3.1 Background to the case study
The Department was founded as the College of Librarianship, Wales in 1964, merging with the then University
College of Wales, Aberystwyth in 1989, becoming an academic department of the university. DIS is the sole
university department offering library, information and archival studies within the Principality. Distance
learning (DL) degree schemes began in 19861, with the establishment of the MSc Econ Management of
Library and Information Services, designed as a CPD scheme for practising library and information
professionals with a minimum of five years in professional-level employment and who normally were
Chartered members of a relevant professional body. A non-standard entry DL undergraduate programme,
accrediting prior experiential learning, began recruiting in 1994, offering 120 credits at entry to library
assistants (or equivalent) with a minimum of 4 years work experience. The MSc Econ Information and Library
1
Details of all distance learning courses offered currently by DIS, Aberystwyth may be found at
http://www.dis.aber.ac.uk/en/courses/courses.asp (accessed 29 July 2010).
102
Studies professional preparation degree via DL began in 1999, and the Department now has a suite of
differing Masters’ schemes available through DL study, including research preparation schemes, together with
those in the archival and records management fields.
Currently, there are 1044 students registered on the suite of DL schemes (July 2010) based in over 50
countries, with c.800 enrolled on Masters’ programmes. As part of our philosophy is to treat students as
individuals (see below) the enrolment statistics cited here refer to actual student numbers and not full time
equivalents (FTEs).
3.2 Course design and delivery

All DIS distance education degrees share certain key features, whether offered at undergraduate (UG) or
postgraduate (PG) level. Delivered using blended, asynchronous dialogic learning, they are:
 flexible;
 work-based;
 recognise and reward experiential learning;
 facilitate access to higher education from disadvantaged groups.
3.3 Flexibility and learning support

Principles of flexibility are embedded within the design and delivery of all degree schemes at Aberystwyth.
Attendance is mandatory at 2 or 3 study schools (depending on the scheme), where students are introduced
to the modules and mode of study, including study skills. Thereafter, students work at their own pace within
an open learning framework, using both paper-based and electronic materials. Further support is offered via
the dedicated Virtual Learning Environment (VLE), designed in-house using moodle, which includes areas for
each module, office areas for administrative matters and queries, together with study skills, exam preparation,
and café areas. Both the materials and delivery mechanisms are designed to facilitate independent and self-
reflective learning. Progression and completion of degree schemes is flexible within the constraints of
university registration and submission regulations. Each student devises an individual study plan in
consultation with their Personal Tutor or Learning Advisor, based upon an annual series of set dates for
assignment submission.
Retention and completion rates across all DL schemes demonstrate that students value and respond
positively to the flexibility and relative freedom they have to plan their learning; the most recent statistics,
compiled by Aberystwyth University Careers Office give rates of 95% for PG and 93.5% for UG DL schemes
respectively. Such flexibility and freedom creates challenges for the learning provider in developing
group/cohort identity, and may result in some individuals experiencing difficulties in prioritising study time
within the constraints of their professional and personal responsibilities (Hill & Taylor, 2009). Group/cohort
identity is encouraged and created at the first study school, and reinforced on subsequent occasions.
Students are encouraged to form study circles and to maintain contact with each other and with relevant staff
via the appropriate areas on moodle in between their periods of attendance at study schools.
Whilst delivery modes have changed and evolved over the 24 years of DL, nonetheless there remains a
persuasive case for the continued use of both blended and asynchronous dialogic learning. On a practical
level, facilitating synchronous social and collaborative learning across differing time zones is problematic, as
is ensuring that all participants have equal access to the requisite technology. An everyday difficulty such as
intermittent energy supply in many countries is yet another factor to be considered. Moreover, as the majority
of students on degrees such as these are employed in the public sector where average incomes are relatively
low, and with many being wholly or partly self-financing, affordability in relation to cutting edge technology is a
key factor. To insist on individuals having access to expensive software and hardware, including mobile
technologies, would militate against the widening access philosophy which has underpinned all developments
to date. The rate of adoption of newer forms of e-learning such as 3D worlds, simulations and games is at
least the subject of debate, with Shepherd arguing that “there is still a long way to go and, despite the
unbelievable optimism of most expert predictions, we’ll see very modest progress in the short, and even
medium, term” (2010, 11). Research conducted at Aberystwyth acknowledges the potential for geographically
remote and isolated distance education students to become demotivated and drop-out, but cohort studies and
overall completion rates suggests the mix works (Broady-Preston & Bell, 2001; Hill & Taylor, 2009).
3.4 Work based experiential learning and widening access

All students are required to be in relevant work-related employment for the duration of their studies, and
assignments are directly related to the work environment. Thus, the design and delivery of degree schemes in
this discipline builds upon the skills and knowledge already acquired through work-based experiential
103
learning, enabling the individual to set their practical skills and knowledge within a more formal, theoretical
context, facilitating both lifelong learning and evidence-based professional practice.
In terms of addressing the widening participation agenda, the undergraduate degree offered via distance
learning warrants especial mention in this context. It was the first such scheme developed in the UK,2
designed to attract non-standard entry applicants, and formally accrediting experiential or workplace learning
as outlined in 3.1 above. The primary market for this scheme is library assistants or paraprofessional staff,
many of who are mature women or from ethnic minority groups, unable to study full time for a variety of
reasons, and additionally may lack formal academic qualifications, but with extensive practical experience.
The challenge for DIS has been to widen participation in higher education and offer a route into the
profession, without in any way compromising quality or standards. Obviously schemes such as this rely not
only on professional association recognition, but also employer engagement and buy-in. Significant numbers
of students on this scheme, both past and present, have received full or partial support from employers, and
DIS maintains a database of anecdotal evidence in relation to career statistics for these student in relation to
promotions obtained and other positive career outcomes. Again, the latest figures from the Aberystwyth
University Careers Office statistics show that 98.2% of these graduates are in relevant professional
employment on graduation.
3.5 Employability and CPD

Arguably in a severe economic recession, there is even more pressing need for universities to provide degree
schemes which equip students for the labour market. As Yorke (2006) notes, there are many differing
interpretations of the term “employability”, adding the somewhat trenchant rider that it “is not the same as
employment” (6) although the two are often treated synonymously. His preferred definition is that it is “a set of
achievements – skills, understandings and personal attributes – that makes graduates more likely to gain
employment and be successful in their chosen occupations, which benefits themselves, the workforce, the
community and the economy” (Yorke, 2006, 8).This definition and his subsequent elaboration of its meaning
underpin the design and progression of the Aberystwyth schemes. Of note in this context, are the Masters’
schemes which have been developed, including the first UK Masters’ degree in Archive Administration offered
via distance learning, and the first Research Training Masters’ degrees offered in distance mode, which
attract government funding and Research Council recognition as constituting suitable preparation for Doctoral
degrees.
Moreover, there is self-evidently a relationship between this richer understanding of employability and CPD.
Both concepts are worthy of more detailed than space allows here. However, if employers are to recognise
and value qualifications, be they competency –based or otherwise, they need to be assured that not only is
the initial qualification fit for purpose, but that the qualification is updated continuously. Given the proposed
introduction of a mandatory light touch CPD scheme for Chartered Members of CILIP from January 2011, the
need for opportunities to update and acquire new skills and knowledge will clearly be of even greater
importance (Broady-Preston, 2009b). There is a pleasing circularity to this development; DIS began its DL
provision with a CPD scheme and will celebrate 25 years of distance education in the year that CPD becomes
mandatory for the information profession.
4. CONCLUSIONS
Distance learning education at Aberystwyth has grown exponentially, as a result of our continually reviewing
the design and delivery of our courses, offering flexibility and CPD opportunities, in addition to widening
access to higher education. The current volatile environment presents numerous challenges, and there are no
obvious pathways through a complex and challenging educational landscape. Asynchronous dialogic learning
works for us and our students currently, but as our history shows, we have grown and survived because we
are willing to embrace new ideas, methods and markets.
5. REFERENCES
Broady-Preston, J. (2010) The information professional of the future: polymath or dinosaur? Library
Management, 31 (1/2), pp. 66-78.
Broady-Preston, J. (2009a) Professional education, development and training in a web 2.0 environment, a
case study of the UK. New Library World, 110 (5/6), pp. 265-279.
2
There were equivalent part time degrees offered to a similar market at Leeds Polytechnic and Manchester Polytechnic in
the 1970s and 1980s, but these did not accredit work-based learning formally, nor were they available via distance
learning.
104
Broady-Preston, J. (2009b) Continuing professional development: its role in the changing education and
qualification landscape of the information profession, a case study of the UK. Strategies for regenerating the
library and information professions. Eighth World Conference on Continuing Professional Development and
Workplace Learning for the Library and Information Professions, 18-20 August 2009, Bologna, Italy. Ed by. J.
Varlejs and G. Walton. K.G. Saur, Munich. pp.260-278. [ISBN 978-3-598-22044-9]
Broady-Preston, J. (2009c) Structuration and social identity theories: qualitative methodologies for
determining skills and competencies for the information profession in the 21st century. Performance
Measurement & Metrics: The International Journal for Library and Information Services, 10 (3), pp. 172-179.
Broady-Preston, J. (2007) Changing information behaviour: education, research and relationships. The
internet, libraries and the changing information behaviour: perspectives for research and education, Division
of Education and Research, World Library and Information Congress: 73rd IFLA General Conference and
Council, 19-23 August 2007, Durban, South Africa. http://archive.ifla.org/IV/ifla73/papers/158-Broady-Preston-
en.pdf (26 July 2010)
Broady-Preston, J., & Preston, H. (2007) Blurring the boundaries? Information studies education and
professional development in England and Wales. Globalization, Digitization, Access and Preservation of
Cultural Heritage: papers from the 6th International Conference, held in Sofia, Bulgaria, 8-10 November 2006
Eds. H. K. Achleitner and A. Dimchev. Sofia: St Kliment Ohridski, University of Sofia. pp. 289-300.
Broady-Preston, J., and Bell, S. (2001) Motivating mid-career LIS professionals: the Aberystwyth experience
New Library World, 102 (1169), pp.372-381.
Chartered Institute of Personnel and Development (CIPD) (2010) Competency and competency frameworks.
Revised June 2010. http://www.cipd.co.uk/subjects/perfmangmt/competnces/comptfrmwk.htm (26 July 2010)
Conway, P. (2008) Professional standards of service. Chartered Institute of Library and Information
Professionals (CILIP), London.
http://www.cilip.org.uk/SiteCollectionDocuments/pdfs/policyadvocacy/conway_mainreport.pdf (26 July 2010)
Edwards, G. (2005), Connecting PDP to Employer Needs and the World of Work, The Higher Education
Academy, London, November 29, p. 8,
http://www.heacademy.ac.uk/assets/York/documents/resources/resourcedatabase/id71_connecting_pdp_to_
employer_needs.pdf (3 February, 2009).
HM Government, The Cabinet Office, The Panel on Fair Access to the Professions (2009) Unleashing
Aspiration: Summary and Recommendations of the Full Report, A. Milburn, Chair.
http://www.cabinetoffice.gov.uk/media/227105/fair-access-summary.pdf (26 July 2010)
Hill, J. and Taylor, M. (2009): Taking off: encouraging deeper learning and cohort identity through Dialogic
Learning. Collected Conference Papers and Abstracts September 2009, The Cambridge International
Conference on Open and Distance Learning 2009, The Von Hügel Institute, St Edmund’s College, Cambridge
in association with The Open University and The Commonwealth of Learning, Sept 09. Ed. by A. Gaskell and
R. Mills. pp.222-229. http://www2.open.ac.uk/r06/documents/CambridgeConferenceMainPaper2009.pdf (19
March 2010)
Lifelong Learning UK (LLUK) (2010a) Innovation during a recession: How skills for learning professionals will
drive economic recovery. Annual Review 2008-2009. http://www.lluk.org/documents/lifelong-learning-uk-
annual-review-2008-2009.pdf (26 July 2010)
Lifelong Learning UK (LLUK) (2010b) National Occupational Standards reviews 05 July 2010.
http://lluk.org/national-occupational-standards-reviews.htm (26 July 2010)
Shepherd, C. (2010) Slowly, slowly, catchee monkey. e.learning age, February, 11.
Yorke, M. (2006) Employability in higher education: what it is- what it is not. The Higher Education
Academy,York.
http://www.heacademy.ac.uk/assets/York/documents/ourwork/employability/id116_employability_in_higher_e
ducation_336.pdf (26 July 2010)
105
ENGAGING SCHOOLS WITH COMPUTER PROGRAMMING
Laura Crane Phillip Benachour
Lancaster University Lancaster University
Communication System, Infolab21 Communication System, Infolab21
South Drive, Lancaster. South Drive, Lancaster.
LA1 4WA LA1 4WA
l.crane@lancaster.ac.uk p.benachour@lancaster.ac.uk
ABSTRACT
In this paper, we describe activities with schools to visit the University and attend one or two day workshops
on learning to programme through visualization and animation. The work aims to study the link between
visualization as a teaching tool to enhance the student’s learning of programming principles and engage
young people with ICT and computing subjects. Four schools visited the University on two separate occasions
and worked on Adobe Flash and Alice 3D environments. Feedback has shown that students are likely to gain
confidence in their programming ability by learning programming concepts then by developing new ways to
handle knowledge through application and independent learning.
Keywords
Programming Visualization in Education, Media in Education, Hands-on learning, Schools Engagement and
Outreach Activities
1. INTRODUCTION
One of the major motives for this work is to introduce and aid the understanding of basic programming
concepts to school pupils and potentially future Computer Science and Information Technology students.
Many educational papers have been published which recognize the immense value of visualisation as a
solution to the ongoing problem of declining numbers of students both taking computer science courses, or
ultimately completing the course. Therefore there is a significant correlation between early stages computing
experiences and success in computer science, or computer-based subjects in the future. Computer Science
course drop-out rates fluctuate amongst different universities and institutions, but with 10% of enrolled
students dropping out within the first year. This perceived lack of preparation for learning programming has
been attributed to the fact pupils are not prepared in algorithmic thinking, with weak mathematics and numeric
computation being a major cause of the problem. Howe et al stated in [1] that pre-university students who
performed well during high school mathematics and science had a superior probability of being successful
computer science students. No other student feature or attribute was established as having either a positive
or negative correlation with success in computer science courses. The overall challenge in delivering material
on programming should be of the development of insight into the methodology and principles, not just an
intimidating overview to the content. Baldwin et al identified in [2] the sources of the issues which need to be
resolved in order to improve the situation in computer science sources. The three main points they contend
are that of ‘too much material’, an ‘inappropriate over-emphasis on design aspects of the discipline’ and the
‘lack of insight-building activities’. Although there are criticisms and questions which surround the validity of
using narrative as a vehicle for delivery programming principles; studies in [3,4,5] have perceived a positive
effect. Such trials suggest the narrative approach ‘motivated the students’ and pleased the teachers in relation
to their ‘learning outcomes’. Although, there are many issues with computer science, and the way it is being
delivered, the most important and notable point for many is that of visualization [6]. In [7] it is argued that a
new strategy for teaching introductory computer programming will attract a group of students not excited by
the invisible, abstract, and text world. Visualisation achieved through animation and interactive programming
[8] provides the students with an actual tangible perspective to help them understand exactly what their
actions are producing with the system. Although it has to be noted, visualization technology no matter how
well it is integrated is of little educational value unless it engages learners in an active learning activity. As like
the programming languages themselves, the actual 'animation of algorithms' require standards and disciplines
[9].
2. PROGRAMMING LANGUAGES IN RELATION TO EDUCATION

In teaching students computer programming, various models have been designed which aim to convey what
106
exactly should be included in delivering programming languages. Figure 1 depicts the four major points that
must be considered when teaching programming [10]. In this figure, the vertical axis denotes the ‘transfer of
competence’; whereas the horizontal axis represents the ‘learning of content’. It is believed it is only horizontal
axis which concerns the educational environment; yet the contemporary perspective is to include each of the
four points. For modern students, it is essential to meet each of the requirements; in order to fully expose and
exploit the time a student spends practicing programming.
Although many different solutions have been
developed which aim to engage students in
computer programming, one of the major
discussion points has been whether the
objective is actually, ‘learning to program’, or
simply ‘programming to learn’? in the debate
as to whether it is more important for the
student to simply learn programming, or that
learning programming should be used in
conjunction with other disciplines. Different
programming cultures have emphasized
different balances between the two opposite
positions and again; everyone has an
individual opinion on exactly what is the
F purpose of learning programming or scripting.
Figure 1: Objectives for programming activities
3. TUTORIAL DESIGN APPROACH

We approach the design of the tutorials with a similar view where the learning of computing concepts is
initially focussed on by introducing the software packages to the participants and giving examples to
demonstrate simple concepts such us increment/decrement operators, conditional statements and looping in
Actionscript. In Alice, this would ultimately be a simpler introduction into the ‘grouping’ and ‘sequence’ options
available when creating a constructed animation. Demonstrating the effect of the ‘Do Together’ and ’Do in
Order’ functions on the code itself, to promote the idea of order and sequence in commands. The construction
of a moonwalk across the screen as shown in Figure 2 was introduced as an intermediate example. The use
of related music and the opportunity to add extra dance movements or actions to the world was also added as
an extension to the exercise.
This exercise was aimed at including more
animation than the introduction tutorial. The
exercise used the space environment and the
addition of an astronaut character to execute the
movement of a moon walk across the screen. This
revises the basic principles which were taught but
adds more difficulty and accuracy in the animation.
The “object tree” is explored further as the
astronaut can control each finger of each hand if
necessary. The “object tree” of the astronaut is
top-right in Figure 2. The refinement of the
animation was increased by using the “custom
number” function which is available in the
software; this allows for a number of variable to be
stated to two decimal places. In creating this
animation itself, a large amount of trial and error
was necessary in order to create the right look for
the movement of the character itself. The “method”
is shown at the bottom of Figure 2.
The benefit of this exercise is to encourage
students to explore new ways to handle existing
knowledge (independently) based on earlier
concepts learned. The Majority of students are
familiar with the moon walk therefore recognized
what they were trying to achieve.
107
Figure 2: Moon Walking Astronaut
The following exercise required the students to create a
simple story using conditional statements. For this
exercise, objects (the road, the barn, the chicken, the
crossing sign, the bend in the road and car) are added to
the scene. The scene was setup using the quad view to
ensure they were correctly aligned on each of the co-
ordinates. Most objects had to be resized and rotated so
they appeared proportionally correct. The chicken had to
think about its motivation for crossing the road, so it has to
make a decision. The chicken’s “method” were selected;
and needed it to ‘think’, subsequently the thought was
typed in. A ‘Do Together’ statement is created and
animation added to the car; moving it forward down the
road towards the chicken whilst it is thinking.
Figure 3: Chicken and Road
The Method for Chickencrosses

The chicken looks both ways before walking across the
road. The car moves forward, past the chicken and the
animation ends.
The Method for Chickenstays
For this method, the chicken decides not to cross the
road, therefore the chicken turns to face the barn, the
barn door opens and closes as the car drives past; and
the animation ends.
Once the two possible endings had been created; the
If/Else statement was created to offer the audience the
opportunity to decide the ending for the animation.
Here, the ‘if/else’ function was dragged into the world’s
method and a ‘ask user for yes or no’ conditional
statement was added. In the textbox the question is
added, subsequently the two possible worlds were
dragged and inserted into the spaces provided.
Figure 4: Methods for Chickencrosses and Chickenstays.
This exercise demonstrates the use of

arrays in AdobeFlash. The overall concept
is based on creating an array which
contains a series of different fortunes or
replies which would ouput the responses
upon user interaction. Here the code is
used to initialize, fill and call the array. The
use of the random function was then
necessary to select unsystematically
responses which could be outputted into
the fortune variable. The output needed to
be displayed on the stage itself. This was
completed by using a dynamic text box
disguised in a cartoon speech bubble to
make it look more interesting, and fit in
with the rest of the design. To include a
multimedia dimension, the addition of a
sound was also added so once the crystal
ball was clicked, it both produced a
prediction, and a thinking sound.
108
Figure 5: working with Arrays using the crystal ball
The final example of this paper looks at game design and creation in Adobe Flash. The game itself would
have to be relatively uncomplicated based on an existing game such ‘rock, paper, scissors’, ‘noughts and
crosses’ or ‘space invaders’. This exercise demonstrates the use of conditional statements, collision detection
and its use within gaming, and user interaction with a mouse.
Both the fish and shark were imported onto a layer and
converted to movie clips, Actionscript code could then be
added to provide animation.
This was controlled using the X and Y positions or co-
ordinates of the shark character and making them move
across the screen. Once the movement was complete, a hit
test was necessary, in combination with an if statement.
Here, if the shark hits the ‘player’; in this case the fish; the
movie will move to frame 2 of the timeline.
The fish needed to be connected to the movement of the
mouse input. This code was placed on the fish object itself,
and hid the normal cursor; so it appeared as if the fish was
the cursor, making it easier to understand in the context of
the game.
Figure 6: Flash game: avoid the sharks!
4. STUDENT ENGAGEMENT AND EXPERIENCE

A mixed selection of students aged between 12-14 years from four schools in the north-west visited Lancaster
University on two separate occasions to participate in the study, once for Flash and again for Alice. The
students were selected externally and possessed a range of abilities. Their experience was structured as one
introductory tutorial in the morning session of an hour and a half, with a short accompanying lecture on using
the features of the software. In the afternoon session, the more advanced tutorials were presented with the
duration around two hours inclusive of a small break to avoid over working the students. Table 1 below
demonstrates the student’s responses to the study:
Gender of Participants
Male 57.9%
Female 42.1%
Previous Experience of Programming Languages:
None 89%
Very Little 11%
Some Experience None
Considerable Experience None
Question Easy Ok Difficult
How did you find Flash as an animation tool? 16% 84% 0
How did you find Alice as an animation tool? 92% 8% 0
How did you find using Actionscript? 21% 68% 11%
How did you find constructing animations? 84.2% 15.8% 0
Question Posed / Tutorial Introduction Crystal Ball Fish Game All
Activity
109
Which tutorial did you find 37% 11% 47% 5%
the most engaging?
Which tutorial did you find 37% 32% 5% 26%
the least engaging?
Question Posed / Tutorial Chicken & Road Monster Mash Cow Array Narrative
Activity
Which tutorial did you find 5.3% 0 0 94.7%
the most engaging?
Which tutorial did you find 10.5% 52.6% 36.87% 0%
the least engaging?
Table 1: Student responses and feedback
From this question posed to the students, their most engaging tutorial was that of the crystal ball, which
delivered the subject of arrays. A close second was the initiation tutorial which demonstrated the basic tools
and functions which are present in Flash and Actionscript. Finally, an influential piece of data extracted from
the study was that 94.7% of the students found the narrative activity the most engaging; and this was obvious
by the length of time they spent carrying out the activity.
5. CONCLUSIONS
Both researching and experimenting with different animation tools has been interesting and informative, as it
has provided different methods of thinking, procedural understanding and completion of tasks. Statistical data
analysis was useful in understanding the success of the project. Each of the applications had various benefits
and disadvantages. Flash was more familiar and structured; and has a larger range of applications and
options when creating tutorials and prototypes. Alice’s three dimensional nature made it an obvious choice
and forward thinking difference which encompasses the concept of a truly three-dimensional virtual learning
environment. This point was highlighted by most of the teachers who attended the workshop. The student’s
responses from the visit were useful in both establishing and reiterating the points which have been raised in
the research portion of this report. And again; the positive nature of responses surrounding the engagement
and interest of the students provides a positive outcome of this project. Furthermore, the outcomes of this
project offer both optimism and confidence that the problems faced by computer science educational
establishments could have a solution in animation and interactive programming.
6. REFERENCES
[1]. Howe, T., Petersen, C. Predicting Academic Success in Introduction to Computers, AEDS Journal (1979)
[2]. Baldwin, H., Koomen, H., Scragg, G. Computer Science Needs an Insight Based Curriculum, ACM
SIGCSE Bulletin 26 , p.p 150-154 (1994)
[3]. Bruckman, A., Piper, A., Zagal, J. Kids Telling Fables Through 3D Animation, GVU Tech Report: GIT-
GVU-04-23 (2004)
[4]. Brindley, G., Eccleston, G., McDermott, R. More Than a Good Story – Can You Really Teach
Programming Through Storytelling? HE Academy for Information and Computer Sciences (2007)
[5]. Kelleher, C., Pausch, R, Kiesler, S. Storytelling Alice Motivates Middle School Girls to Learn Computer
Programming, CHI Proceedings pp. 1455 -1464 (2007)
[6]. Cooper, S., Dann, W., Pausch, R. Developing Algorithmic Thinking with Alice, Proceedings of ISECON,
17, (2000) pp 506—539
[7]. Guzdial, M., Soloway, E. Teaching the Nintendo Generation to Program, Communications of the ACM
45, p.p 17-21 (2002)
[8]. Angel Velazquez-Iturbide, J., Almstrum, V., Dann, W., Fleischer, R., Hundhausen, C., Korhonen, A,.
Malmi, L., McNally, Naps, T., M., Robling, G., Rodger, S. Exploring the Role of Visualization and Engagement
in Computer Science Education, ACM SIGCSE Bulletin 35 p.p 131-152 ( 2003)
[9]. Benachour, P., Edwards, R. Animation and Interactive Programming: A Practical Approach, Electronic
Notes in Theoretical Computer Science 224 (2009)
110
[10]. Mendelsohn, P., Green, T.R.G. & Brna, P. (1990) Programming languages in education: The search for
an easy start., In J-M. Hoc, T.R.G. Green, R. Samurcay & D.J. Gilmore (Eds.), The Psychology of
Programming (pp. 175-200). London: Academic Press.
111
LISTENING STEP BY STEP
Janet Lavery Liz Burd

Technology Enhanced Learning Research Group Technology Enhanced Learning Research Group
School of Engineering and Computing Sciences School of Engineering and Computing Sciences
Durham University Durham University
janet.lavery@durham.ac.uk liz.bur@durham.ac.uk
www.durham.ac.uk/janet.lavery www.durham.ac.uk/ecs/computing.science.tel/
ABSTRACT
This paper will present the results from an investigation in to the use of audio files to support the learning of
algorithms in the level-one Computer Systems module. Specifically algorithms associate with the Machine
Architecture sub-module of Computer Systems. As an example the algorithm for converting a decimal
number to a number in any other base has been recorded and used. Recent research suggests that the
cognitive load for learning can be reduced, if when working through problems, not all the learning-objects
(instructions, problems, examples, and methods or algorithms) are text. It is proposed that providing some of
the support as audio rather than text will allow the students to learn more for the same effort. This suggestion
presupposes that students will use, effectively, audio only learning-objects. Each audio file contains a
precisely worded description of a single algorithm or a section of a single algorithm. The terminology used is
the same as that which is used in lectures. The algorithm will have been worked through by the lecturer as a
worked example in a lecture. Some algorithms will have been worked through by students in small exercises
during lectures. It is recognised that not all students attend lectures. The paper will present the results of a
short survey about whether the students liked the audio files and thought they were useful when working
through exercises. In addition, it will present the results of a number of interviews with students that were
held to discover how students went about using the audio files to solve set problems in laboratory practicals
and in revision for the benchtest.
Keywords
Podcasts, Audio Learning-objects, Algorithms, Cognitive Load, Lab Work.
1. BACKGROUND
The Computer Systems level-one module is concerned with the historical, technical, and social issues
surrounding complex distributed computer systems and is composed of four discrete sub-modules Machine
Architecture, Operating Systems, Introduction to Networks, and Databases. The sub-modules are taught
consecutively and assessed separately. The Machine Architecture sub-module is comprised of 10 lectures, 4
laboratory practicals in which students are given formative feedback, and is assessed via a one-hour
benchtest. The focus of the Machine Architecture sub-module is how computers perform work, data
representations, and Boolean logic.
Computer Science is considered a ‘knowledge-rich’ (Van Lehn, 1989) domain in that it takes a significant
amount of learning to understand the domain and meaningfully work on the problems in the domain. The
ability to correctly select and apply algorithms for problem solving is recognised as necessary in computer
science and demonstrates a meaningful understanding of computer science concepts (Kordaki et al., 2008).
Cognitive load is comprised of intrinsic cognitive load, caused by the intellectual complexity of what is to be
learned; extraneous cognitive load, caused by the format of the learning opportunity; and germane cognitive
load caused by the learning-related activities employed by the learner in order to learn (Kalyuga, 2006).
Intrinsic cognitive, extraneous cognitive and germane cognitive loads combine to form the cognitive load that
112
the learner must cope with using working memory. If excessive, the learner’s ability to adapt their knowledge
to include the new knowledge will be incomplete, disordered or incorrect. Requiring learners to use the same
part of the working memory processing system (visual or auditory) in all activities intended to support the
learner will increase extrinsic cognitive load. Designing learning opportunities that allow a learner to distribute
the learning over different processing systems in working memory will reduce the extraneous cognitive load
(Kalyuga, 2006). For Machine Architecture laboratory practicals students usually have access to lecture
slides, lecture handouts, and their own notes and the lab exercise-sheets all of which are text based and
require reading.
Podcasts are digital media files, both audio only and audio-visual, that are automatically downloaded to
computers or mobile devices via Really Simple Subscription (RSS) technologies when a learner has
subscripted to a particular source of digital media (Frydenberg, 2006, Chan et al., 2006, Tynan and Colbran,
2006). However, it is not uncommon in the literature for the term podcast to be used to describe any digital
media file that can be downloaded by students from a lecturer’s repository, for example a VLE, (Salmon et al.,
2008).
In this paper we will discuss learning-objects that are audio only digital media that could potentially be
podcasts but are at present discussed as audio files because they are .wav files developed using the
freeware Audacity and suitable for storage and subsequent use on the University provided computers and
within the University’s virtual learning environment (VLE). Thereby ensuring all the students in the cohort had
access to the audio files. A number of algorithms for the manipulation of binary representation are taught as
part of Machine Architecture sub-module and it is these algorithms that have been recorded for students to
access in lab practicals. It should also be noted that the use of the audio files was, at the start, not optional
for students. They were instructed to try the audio files in the first lab practical. Use of the audio files after the
first lab practical was controlled by the students.
2. PROVIDING THE AUDIO FILES

The Machine Architecture lectures and their associated worked examples, student in-lecture handouts, and
related lab practical exercises, were developed concurrently. An exercise-sheet for a single lab practical
normally contains exercises relating to two or more lectures. The audio file transcripts were written after the
lecture and its associated content was complete thereby ensuring that the audio files and the lectures used
the same terminology and provided algorithms with the same steps and in the same order.
The audio files were intended as scaffolding (Puntambekar and Hübscher, 2005, Pea, 2004) to support the
students’ initial learning effort while working through the exercise-sheets. The audio files contained the steps
of the algorithm without reference to a specific example allowing students to review worked examples in their
lectures notes as well as assist their efforts in completing the exercise-sheets. It has been observed that
students often review the worked examples given in lectures while working through lab practical exercise
sheets. To ensure that students had control of all the learning-objects within their learning environment the
audio files were made available to students the week prior to the lab practicals and were left available until the
end of the exam period.
2.1 Development of the Audio Files

The audio files contained precisely worded algorithms reflecting their mathematical content and application.
In the first recording session it took 90 minutes to record 7 audio files none of which were over a minute in
length. In the second recording session it took 90 minutes to record 7 audio files none of which were over two
minutes in length. The transcripts of the audio files were adjusted during the recording session because after
listening to a recording it would become apparent that some of the phrasing that works well in a written
description on a slide did not ‘sound right’. The changes did not alter the correctness or completeness of the
algorithms and more correctly reflect how they were stated when in lectures. The required precision in use of
terminology and order of the steps meant that all the audio recording took several takes.
The audio files were recorded to support work on the exercise-sheets for the first and second Machine
Architecture laboratory practicals. The first practical was concerned with the simpler concepts of converting
between various number systems including binary, decimal, and hexadecimal. These algorithms were broken
into two distinct sections, an initial series of steps, and the subsequent iterative series of steps. For example
to convert a decimal number to a representation of another base initially the decimal number is divided by the
base providing a quotient and a remainder. If the quotient is not zero then the there needs to be repeated
divisions using the previous quotient repeating until the quotient is equal to zero. Therefore two audio files
were recorded the first “Initial steps for converting decimal to any base” and second “Repeating steps for
converting decimal to any base”.
113
The second laboratory practical was concerned with concepts that built on the previous practical and
contained simple arithmetic exercises for example addition and multiplication in range of data representations
including binary, two’s complement, and floating point. For the second practical the decision was made to
extract the smaller algorithms from floating point addition and multiplication algorithms and make individual
audio files for those smaller algorithms. For example addition of floating point requires an understanding of
mathematical normalisation, deriving a bias exponent, and aligning exponents. The audio file detailing how to
add to floating point numbers mention the steps for example “normalise the mantissa” but does not state how
to normalise which is contained in a separate audio file dedicated to the steps of how to normalise the
mantissa. Separating out the smaller algorithms meant that the steps did not have to be recorded more than
once thereby ensuring consistency. It was thought that students having difficult with only one small area of
the problem could focus on that area.
2.2 Using the Audio Files

The Computer Systems cohort of 42 students is divided in to three groups for laboratory practical work. Each
group is schedule to do the work on a different day and time. Practical exercise-sheets and supporting audio
files were available to the students a week prior to their schedules labs. Students were free to review, start
and even finish the exercise-sheets prior to the start of their scheduled lab practicals. Students were warned
in advance that they would need headphones to use the audio files associated with the practical exercises.
Some students opted to use their iPod headphones which they normally have with them. Inexpensive
Logitech headphones were available to the 19 students who did not bring headphones in the lab. For health
and safety reasons these were distributed but not required to be returned.
Attendance at the lab practicals is mandatory but illness and lethargy mean that not all practicals are fully
attended. 38 of the 42 students attended their first lab practical and 39 out of the same 42 students attended
their second practical lab. At the time of the second practical one student was at home due to sever illness.
However, that student did the practical exercise-sheet and used the audio-files at home. This student was
interviewed later and stated that the audio-files proved useful for both her and her father who was trying to
help with the work.
There were no major technical ‘glitches’ and the only one broken link to a single audio file which was found in
the first practical by a student that normally works ahead of his peers. The broken link was fixed quickly in the
practical and only one student was inconvenienced for five minutes.
The students were instructed to use the audio files in the first lab practical but were allowed to choose
whether or not to use the audio files associated with the second lab practical.
3. EVALUATION
An anonymous feedback survey was given to the 39/42 students at the end of the start of the third Machine
Architecture lab practical 25 were returned completed. The survey contained six questions and a request for
comments ‘As a student what is the best / worst thing(s) about audio learning-objects’. The University’s
virtual learning environment provides a tracking facility on all learning-objects by cohort and individual student.
The tracking data when students were accessing the audio files. The tracking data together with the students
summative test marks was used to provide a starting profile for the nine students who volunteered to be
interviewed after the completion of the Machine Architecture. The interviews were held at the start of the next
term when the students’ workload was light enough for them to find the time to be interviewed.
3.1 Anonymous Feedback Survey Results

3.1.1 Question 1 - What is your preferred way of learning new information? Select as appropriate
The choices were Pictures, Audio, Text, Active (doing), and Don’t Know. The replies were: Picture Only (1),
Audio Only (0), Text Only (1), Active Only (5), Combinations excluding audio (11), Combinations including
audio (5), and Don’t Know (2). Only five students selected audio as a way they preferred to learn and only in
with a group of other ways. Active, learning by doing, was selected singly or as part of a group by 18
students, pictures 12 and text 10.
3.1.2 Question 2 - How many of the audio files for Practical 1 did you listen to?
The results were All (7), Nearly All (3), About Half (4), A Few (7), and None (4).
Despite being given instructions both written and verbal to use the audio files four students replied that they
had not used any audio files in the first of the two practicals. It should be noted that the tracking reports do
not concur with the survey. The tracking only tracks students who have accessed the files. Perhaps these
students made the distinction between accessing them to listen to and using them to support their completion
114
of the exercise-sheet. None of the four students listed audio as a preferred way to learn. One of the four
students stated that he/she also did not use the audio-files associated with in the second practical and was
not planning to use them during revision. One of the four students did use a few of the audio files in the
second practical but did find them useful and was not planning to use them in revision. His/her comment was
“I just cannot take in learning etc. via audio. Text + practice w/examples (with examples) for the win.” This
student stated that they preferred to learn via text and doing. The remaining two of the four students went on
to use about half of the audio files in the second lab practical. Both of these students found the files useful
and were planning to use them in revision. This was not an unexpected as the concepts for the second
practical are more complex.
Of the seven students who stated that they used all the audio files only one found them very useful. Three
found them sometimes useful and three did not find them useful. Six of the seven students tried nearly all or
all of the audio files associated with the seconded practical. The remaining student tried a few of the audio
files in the second practical and found them sometimes useful as well.
3.1.3 Question 3 - Overall the audio files I listened to in Practical 1 were:

Very Useful (1), Useful (8), Sometimes Useful (7), Not Useful (5), and N/A (4) which matches the students
who did not use them. It is encouraging that so many students found the audio files useful
3.1.4 Question 4 - How may of the audio files for Practical 2 did you listen to?
All (3), Nearly All (7), About Half (7), A Few (5) and None (3) which is consistent with the tracking data.
3.1.5 Question 5 - Overall the audio files in Practical 2 were

Very Useful (0), Useful (8), Sometimes Useful (8), Not Useful (6), and N/A (3). The more complex algorithms
and exercises were associated with this lab practical.
3.1.6 Question 6 - How likely are you to use the audio files in our revision for the Machine
Architecture Benchtest?
Very Likely (4), Likely (7), Unlikely (9), Definitely not using them (5). The tracking data support that the audio
files were reasonably will used in the benchtest revision period.
3.1.7 Student Comments

Students were asked “What is the best/worst thing(s) about audio learning-objects.” Though not all the
students provided comments the majority did. The comments have provided some insight into how student
view not only the audio files but the lab practical work. It is interesting the number of time concerns were
expressed about the amount of time that audio files required and how ‘scanning’ text was faster and therefore
better. Ideally, I would not want students ‘scanning’ anything and if the audio files are slowing them down
then for me that is an improvement to their working practices in the labs. One student state that the worst
thing was that “they need repeating several times” which is of course the point. He/she went on further to
advise that the “attached text file would be excellent”. This student was not the only one who commented that
the addition of the written transcript would be an asset. It is worrying that some students did not realise the
written transcript was on the lecture slides and should have already been part of their notes. One student
comments that he had difficulty finding the relevant section of an audio file. These files are all less than 2
minutes long and I can see no way to make them smaller and more focused. A few students mentioned that
they had difficulty with the terminology used in the audio files. For example one comment was “Some of the
jargon used was slightly unclear, and not greatly explained.” In fact none of the terminology was explained in
the audio files. The terminology was explained in the lectures and in the lecture slides.
Like any survey the majority of the comments are going to be negative. There were, however, some positive
comments from students who had begun to understand how to the use the audio files to achieve success.
For example one student commented that the best thing was “Ability to examine question simultaneously with
the listening.” And another wrote “They (the audio files) provide a means to listening to an explanation whilst
still being able to look at your work, which is very useful as my concentration on the problem is not broken”.
Others comments point out that one lecture with a worked example is not enough if students are to succeed in
the working through the exercise-sheets. “Provide some extra guidance” was how one student put it.
3.2 Interviews
Nine students volunteered to be interviewed. All of the interviewed students said that they like the audio files.
The audio files provided extra support and all but three of the students would have reasonably happy with any
further support audio, textual, or one-on-one instruction with the lecturer. Three of the students have a
preference for audio learning-objects. All nine students thought the step-by-step approach to the algorithms
115
in the audio file supported the step-by-step way they worked through the exercises. Eight of the nine students
thought that is was excellent that smaller algorithms needed a more complex larger algorithm were recorded
separately. When learning something reasonably new the separation allowed them to focus on one thing at a
time. When they had some a general understanding larger algorithm the separation allowed them to focus on
the part they did not completely understand. Only one student wanted everything in the one learning-object
as a long file. This student also wanted the terminology explained in the audio files as well. I believe he
wanted recordings of the lectures not support for the exercises. This particular student struggles with all
mathematical components of the level-one programme and is using the audio files as support and not
scaffolding that will fade
Six of the students felt that the audio files were for support during the lab practical and to be consulted if they
required help with an exercise. These students intended to work from their written notes for exam revision
and not consult any learning-object currently held on the VLE. The tracking data shows that most of the
students are using some files more than other and a few files not at all.
Only one of the students thought the audio files should be used every time they worked through an associated
exercise and for revision. This particular student struggles with all mathematical components of the level-one
programme and is using the audio files as support and not scaffolding that will fade. However, a cursory
glance of the tracking data has shown that this particular student has not used the audio files or any of the
new audio-visual files that were put in place specifically for revision.
All of the students felt quiet strongly that the students should have control over the learning-objects within the
VLE. One student who has transcribed the audio files for inclusion in their notes would still not want them
removed.
4. FURTHER WORK
Building on what was learned from the work with audio only learning-objects. Audio-visual learning-objects
were developed and used in a revision practical for Machine Architecture. These learning-objects
emphasised the step-by-step approach preferred by the students and provided worked examples visually with
audio used to provide ‘filler’ information for example why a step occurs in a particular place in an algorithm.
Feedback from the students has been requested and tracking data was collected.
5. REFERENCES
CHAN, A., LEE, M. J. W. & MCLOUGHLIN, C. (2006) Everyone's learning with podcasting: A Charles Sturt University experience. The 23
rd Annual ASCILITE Conference: Who's learning? Whose technology? Sydney, Australia, Sydney University Press
FRYDENBERG, M. (2006) Principles and Pedagogy: The Two P's of Podcasting in the Information Technology Classroom. ISECON
2006. Dallas, EDSIG.
KALYUGA, S. (2006) Instructing and Testing Advanced Learners: A Cognitive Load Approach, New York, Nova Science Publishers, Inc.
1-59454-868-4.
KORDAKI, M., MIATIDIS, M. & KAPSAMPELIS, G. (2008) A computer environment for beginners' learning of sorting algorithms: Design
and pilot evaluation. Computers and Education, 51, 2, 708-723.
PEA, R. D. (2004) The Social and Technological Dimensions of Scaffolding and Related Theoretical Concepts for Learning, Education
and Human Activity. The Journal of the Learning Sciences, 13, 3, 423-451.
PUNTAMBEKAR, S. & HÜBSCHER, R. (2005) Tools for Scaffolding Students in a Complex Learning Environment: What Have We
Gained and What Have We Missed? Educational Psychologist, 40, 1, 1-12.
SALMON, G., MOBBS, R., EDIRISINGHA, P. & DENNET, C. (2008) Podcasting technology. IN SALMON, G. & EDIRISINGHA, P. (Eds.)
Podcasting for Learning in Universities. Maidenhead, Open University Press 9780335234295.
TYNAN, B. & COLBRAN, S. (2006) Podcasting, students learning and expectations. IN MARKAUSKAITE, L., GOODYEAR, P. &
REIMANN, P. (Eds.) The 23 rd Annual ASCILITE Conference: Who's learning? Whose technology? Sydney, Australia, Sydney University
Press.
VAN LEHN, K. (1989) Problem Solving and Cognitive Skill Acquisition. IN POSNER, M. I. (Ed.) Foundations of cognitive science.
Cambridge, Mass, MIT Press, 0-262-16112-5.
116
SUSTAINING SUSTAINABILITY:
DEVELOPING MATERIAL FOR HIGHER EDUCATION
Neil Andrew Gordon

Department of Computer Science
University of Hull
Cottingham Road, Hull, HU6 7RX
n.a.gordon@hull.ac.uk
www.hull.ac.uk/php/cssnag/
ABSTRACT
In 2010, just over half way through the United Nation’s decade of sustainable development (S.D.), it seems
timely to consider the impact and progress made in including sustainable development within Higher
Education (H.E.) curricula. This paper will review some of this in the context of numerous U.K. strategies, and
in particular how the profile of this is developing within the H.E. computing community in the U.K.
Further, this paper will consider some of the experiences of the author in developing teaching materials and
practices on teaching S.D. within H.E. The main focus of this part of the paper will be the author’s own
teaching in Computer Science. However, the paper also considers how material can be made accessible to
other colleagues within and outside of the discipline of Computing, as well as how to encourage and support
those colleagues interested in this topic in other fields.
As well as the curriculum issues that the above impinges upon, there are some technical aspects that can
support and enhance such material and that illustrate the role of computing in supporting such learning
generally: namely the use of reusable learning objects as a mechanism and framework for developing
material that can be used widely and long term, namely in a sustainable way.
Keywords
Sustainable Development; Green IT; Reusable Learning Objects
1. INTRODUCTION
Sustainable Development has been on the agenda for education for several years. As a general topic, it can
be encapsulated in the following quote from the World Commission on Environment and Development (1987)
as “development which meets the needs of the present without compromising the ability of future generations
to meet their own needs”
Education was identified early on as a key mechanism to promote this agenda, and thus led to the Education
for Sustainable Development approach (UK National Commission for UNESCO, 2010).
Moreover, 2005 saw the launch of the United Nations decade of Education for Sustainable Development
(GDRC, 2010). There appears to be quite a lot of progress through estates strategies on campuses
throughout the U.K. with initiatives such as the EcoCampus (EcoCampus 2010) and the Higher Education
Council for England strategy on carbon reduction (HEFCE, 2010). However, progress with respect to
curriculum change seems more limited from a practitioner perspective. Recent initiatives by the Information
and Computer Sciences subject centre – such as a special journal on SD in the computing curriculum and a
workshop on this subject – whilst garnering some support from a group of enthusiasts, indicates that there is
still a lack of engagement and action across the sector, at least within the ICS discipline. Further, experience
of the author in similar campus based initiatives shows that there is a lack of awareness or desire to engage
with this agenda across other subjects.
117
2. INITIATIVES TO ENCOURAGE S.D. PROVISION
2.1 The U.K. context
A number of U.K. government bodies have been promulgating S.D. and encouraging its coverage, both within
education and more widely, for example via regional development agencies. As far as education goes, S.D. is
encouraged through the Sustainable Schools programme (Huckle, 2008). The implementing of this can be
considered within the national curriculum areas of “self, relationships, society and the environment”
(teachernet, 2010). Within Further Education, policy from the Learning and Skills Council has specified the
inclusion of S.S. within this sector. Whilst there is evidence of some progress here (DCSF, 2010), actual
student data indicates there is some way to go at the pre-university level. From a survey of 150 first year
computing students in 2008, only 55% had heard of the term before (see Figure 1). Note that figures from a
smaller survey in 2009 showed a higher level of awareness.
Question Positive response
Have you heard the term ‘Sustainable Development’ before? 55%
Were you aware that as a professional practitioner you will be expected to 58%
take account of the social and environmental impact of your work?
Do you consider it appropriate to include content about sustainable 45%
development within modules you attend on your specialist subject?
Figure 1: student perceptions of SD
2.2 U.K. Higher Education (the tertiary sector)

Within Higher Education in the U.K. the funding councils (such as HEFCE) have been encouraging institutions
to engage with the agenda through a range of activities. Some of these have been linked directly to funding,
others through medium term strategic aims. Within the U.K., the Higher Education Academy has instigated a
project on Education for Sustainable Development (HEA 2010) to help “institutions and subject communities
develop curricula and pedagogy that will give students the skills and knowledge to live and work sustainably”.
This project has funded mini-projects and activities to encourage the development and shared practice
regarding sustainability in H.E.
Another initiative to further the sustainability agenda within U.K. H.E. is the EAUC (Environmental Association
for Universities and Colleges). This organisation supports member institutions in developing sustainability
within the management and curriculum, and as such can be considered along the lines of the Sustainable
Schools initiative within primary and secondary education.
The HEA Sustainability project and the EAUC launched the Sustainability in Higher Education Developers
(SHED) network, which aims to support curriculum development of sustainability within H.E.
2.3 Resources and initiatives within the H.E. ICS sector

With the HEA embedding sustainability within its own medium term strategy, the HEA subject centres have
also implemented sustainable development related activities. These have included workshops and
newsletters, as well as other initiatives on sustainability generally. The HEA subject centre for Information and
Computer Sciences initially ran a joint focus group with the Maths, Statistics and Operation Research (MSOR)
subject centre, on sustainability within these disciplines as part of a wider information gathering exercise. The
outcomes of this contributed to a report on the state of sustainability in the HE curriculum at that time (Dawe
et al, 2005). As noted in that report “Subject Centres that have an interest in ESD, but have found it much
more difficult to embed ESD widely or deeply into their curricula. Examples include Information and Computer
Sciences and Mathematics, Statistics and Operational Research”.
The ICS subject centre’s own site on Sustainable Development (ICS, 2006) provides some starting points for
colleagues interested in SD within computing, including case studies on integrating content into the
curriculum.
Another useful resource within computing is provided through the British Computer Society (BCS). The BCS
have focussed so far on green computing – especially power usage, carbon footprints and the manufacturing
processes - with professional qualifications in related areas, and a Green IT specialist group (BCS, 2009).
However, the wider set of topics that fall under sustainable development, and which can be considered as
part of sustainable computing, include environmental; social; economic; scientific and governance issues.
118
Details of wider implementation of SD within the ICS sector are harder to unearth. The 2008 UK National
Commission for UNESCO report on ESD in the UK only identifies one example within ICS - a “ESD Case
Study for HEA ICS Subject Centre” at the University of the West of England.
3. SUSTAINABLE DEVELOPMENT IN THE CURRICULUM

Desk based research on Sustainable Development within U.K. undergraduate computing curricula finds little
evidence of progress. Whilst, a number of MSc programmes explicitly include SD, and some focus on it.
anecdotal evidence regarding undergraduate teaching - from discussions with colleagues across the sector -
indicates that where SD is included currently, it is included under the general areas of ethical and professional
behavior and awareness.
As part of a HEA mini-project, the approach identified above was used by the author and colleagues, with
evaluation in terms of its effectiveness in developing awareness of these topics for students (Gordon, 2009a
and 2009b). Whilst the results were positive, the experience also demonstrated the difficulties in developing
effective material that can be used within the context of a computing curriculum. The positive impact on
students’ perceptions of embedding SD material within a CS course can be seen from the data in Figure 2.
Question
Yes (%) No (%)
(increase on
first survey)
Are you aware of any requirements to demonstrate an awareness of social and 67 33
ethical issues within your programme of study? (+35)
Were you aware that as a professional practitioner you will be expected to take 83 17
account of the social and environmental impact of your work? (+25)
Given your subject of study, do you believe that your subject has any potential 60 40
impact to any area of sustainable development – environmental, social or other? (+14)
Figure 2 Student perceptions post teaching of SD material
4. CREATING SHAREABLE RESOURCES: A CASE STUDY

Encouraging engagement by colleagues may be accomplished through taking advantage of technical and
policy changes. Of particular relevance to this agenda is Reusable Learning Objects (RLO) which offers one
way to support colleagues in integrating new material – potentially SD – within their own teaching.
4.1 What are learning objects and what makes them reusable
A Learning Object can be “defined as the smallest independent structural experience that contains an
objective, a learning activity and an assessment". Reusable learning objects can be considered as “by being
a cohesive learning resource focused on one clear learning goal” (ICS, 2009a) and further will typically be
electronic in nature “reusable learning objects as web-based interactive chunks of e-learning designed to
explain a stand-alone learning objective” (ICS, 2009b). Whilst not necessarily related, such learning objects
would naturally fit into the ambit of Open Educational Resources.
4.2 Developing RLOs for SD

4.2.1 An institutional project
At the author’s own institution, a small amount of funding is supporting a project to encourage staff across
campus to develop small learning resources on sustainable development. These range from careers and staff
development, through to specific disciplines such as health and Computer Science. Colleagues are
developing materials that include guidance for staff, learning materials (PowerPoint slides, electronic learning
objects), and assessment materials. The learning objects are being developed as simple collections (e.g.
Word files and PowerPoint slides), as well as flash objects developed using Articulate (Articulate, 2010) or
Xerte (Xerte, 2010). Assessments are either suggestions for activities such as reports or essays, to be tutor
marker, as well as some developed in Sakai (the local VLE) and exported for importing into other compatible
VLEs and other computer based assessment tools.
119
4.2.2 Defining meta-data and good practice for RLOs
To ensure that the learning materials have as wide an impact as possible, developers have been encouraged
to follow good practice, in particular
 Ensuring good practice in referencing and acknowledging materials
 Considering and obeying copyright
 Allowing for the context and transferability of ideas/information (especially considering subject specific
language)
 Considering the provenance/context for external links
 Defining appropriate Meta-Data for the learning objects: the chosen set of data for this project is given
in Figure 3
Name
Description
Subject
Topic
Level (3, 4, 5, 6 using U.K. level descriptors
Credit value (Hull credits)
Sub-level: e.g. introductory; advanced
Specialised (subject dependent) or service
Focus : background; motivation; introduction; examples; theory;
applications; advanced; implications
Aims and objectives
Pre-requisites
Learning outcomes: intellectual; transferable
Mode of study: taught; led; supervised; supported; self; independent
Type of study: individual; group
Assessment: automated; manual; tutor; self; peer; group; mixed
Figure 3 Sample Meta data descriptors for SD RLOs
4.2.3 Successes and failures

Successes include the wide number of faculties involved – including computer science, education, social
health, business, careers and staff development. However, there has been no engagement by arts subjects
on campus as yet. Whilst geography and biology do include relevant content, there are currently no
representatives from those areas involved with this project. Discussion with colleagues from across campus
has shown that the awareness of sustainability, and what it encompasses, varies widely. Many colleagues
were initially unsure how it could be relevant to their discipline – although having explored options, realised
that it was already present implicitly, and could be developed explicitly with some learning material. This has
been the key success of this project to date.
4.2.4 Sustaining sustainability

One commonly perceived problem for innovations is how to maintain momentum – especially in an area
where funding is limited in value and time. Taking onboard ideas of community sites, the long term plan for
this project is to use a community website – in this case hosted on Sakai – to try to
i) Share resources and best practice
ii) Encourage colleagues to consider how to embed sustainability into their own teaching
iii) Allow those developing resources to maintain and evolve them
iv) Encourage a community of practice – through a Wiki, discussions and other shared resources
v) Encourage colleagues to reflect, analyse and record the impact with their students.
At this stage, the intention is to share these within the institution, via various dissemination routes to support
the community site. In the medium term, the possibility of sharing them further will be explored, but as with
OER materials generally, this requires agreement within the institution on issues of copyright and publication
of such material.
120
5. CONCLUSIONS
This paper has summarised some of the issues pertaining to Sustainable Development within Higher
Education. Empirical evidence demonstrates that a positive impact can be made on students’ awareness of
this subject through embedding content on this, and that one effective way to produce such content is through
reusable learning objects. Such learning objects have the potential to allow colleagues across the sector to
include this material efficiently and effectively. However, in order to encourage such activity, which requires
changes to teaching material and some reorganisation, it seems that more proactive action is required
elsewhere – particularly with respect to subject benchmark descriptions and accreditation criteria for
computing and related degrees. One suggestion of the author would be for such descriptions and
accreditation to require that computing graduates will be able to demonstrate an awareness and
understanding of green IT and of the more general topic of Sustainable Computing, and which could
potentially span across from the topics of ethics and professional behaviour to the areas of system design and
effective resource management. If such learning resources could be made available as Open Educational
Resources it would offer further scope for a wider take up of this important topic within Computer Science and
other disciplines.
6. REFERENCES
 Articulate, (2010), E-Learning Software and Authoring Tools - Articulate Rapid eLearning,
http://www.articulate.com/, (19 May 2010)BCS, (2009), Green IT Specialist Group,
http://www.bcs.org/server.php?show=nav.10547, 01/12/2009
 Dawe, G., Jucker, R. and Martin, S., (2005), Sustainable Development in Higher Education: Current
Practice and Future Developments, A report for The Higher Education Academy
 DCSF: Department for Children, Schools and Families, (2010), Evidence of impact of sustainable
schools, http://publications.teachernet.gov.uk/default.aspx?PageFunction=productdetails&PageMode
=publications&ProductId=DCSF-00344-2010&, 18/05/2010
 EcoCampus, (2010),Environmental Management System http://www.ecocampus.co.uk/, 18/05/2010
 GDRC: Global Development Research Center, (2010), United Nations Decade of Education for
Sustainable Development, http://www.gdrc.org/sustdev/un-desd/, 18/05/2010
 Gordon, N., (2009a), Sustainable Development and Social Responsibility – Making it Professional,
proceedings of the 2009 Higher Education Academy Subject Centre for Information and Computer
Sciences, University of Kent, August 2009, pp21 – 24
 Gordon, N., (2009b), Improving student awareness of sustainable development and related
employability issues through embedded course content”, on EvidenceNET,
http://www.heacademy.ac.uk/assets/York/documents/events/annualconference/2009/papers/Neil_Gor
don_Paper.doc, 02/07/2009
 HEA, (2010), Sustainability, http://www.heacademy.ac.uk/ourwork/teachingandlearning/sustainability,
25/04/2010
 HEFCE, (2010), Report on Carbon reduction target and strategy for higher education in England
 Huckle, J, (2008), An analysis of New Labour's policy on education for sustainable development with
particular reference to socially critical approaches, Environmental Education Research, Volume 14
(1), pp 65 – 75
 ICS, (2006), HEA ICS Education for Sustainable Development (ESD),
http://www.ics.heacademy.ac.uk/resources/supp_learning/esd/index.shtml, 17/05/2010
 ICS, (2009a), HEA ICS Learning Objects for Introductory Programming,
http://www.ics.heacademy.ac.uk/resources/rlos/introprog/intro.php, 18/05/2010
 ICS, (2009b), HEA ICS Reusable Learning Objects, http://www.ics.heacademy.ac.uk/resources/
rlos/index.php, 18/05/2010
 L'Allier, J., (1997), Frame of reference: NETg's map to the products, their structure and core beliefs,
NetG
 Teachernet, (2010), Teaching sustainable development, http://www.teachernet.gov.uk/
sustainableschools/about/about.cfm?levelselected=4&id=4, 18/05/2010
121
 UK National Commission for UNESCO, (2008), ESD in the UK in 2008: A Survey of Action. The UN
Decade of Education for Sustainable Development, 2005-2014
 UK National Commission for UNESCO, (2010), Education for Sustainable Development
http://www.unesco.org.uk/education_for_sustainable_development, 15/05/2010
 World Commission on Environment and Development, (1987), Our Common Future: from one earth
to one world, http://www.un-documents.net/ocf-ov.htm , 23/01/2007
 Xerte, (2010), Open Source E-Learning Developer Tools, http://www.nottingham.ac.uk/xerte/,
19/05/10
122
DEVELOPING SUSTAINABLE INTERNATIONAL PARTNERSHIPS:
A CASE STUDY OF THE INDIAN SUB CONTINENT
Dharmendra Shadija Richard Hill Neil Richardson

Sheffield Hallam University University of Derby Sheffield Hallam University
LATtICE Research Group LATtICE Research Group LATtICE Research Group
Sheffield, UK Derby, UK Sheffield, UK
d.shadija@shu.ac.uk r.hill@derby.ac.uk n.richardson@shu.ac.uk
ABSTRACT
This paper explores some of the issues often attributed towards working with international students, and
describes a 4 year long case study of how an educational partnership has progressed between a UK HE
institution and an institution affiliated with the University of Mumbai, India. The adoption of a values-based
approach has thus far facilitated a more effective and forward-looking relationship than has been experienced
before. This closer relationship has yielded several key contributions: a) A better understanding of the
learning/teaching context within each institution; b) a specific set of guidelines for shaping the curricula at both
institutions, including a specific ‘bridging studies’ module to help prepare Advanced Diploma students for a
one year ‘top-up’ programme at the UK institution.
Keywords
Sustainability, internationalisation, educational partnerships, computing.
1. INTRODUCTION
The internationalisation agenda is becoming increasingly important for UK Higher Education institutions.
Recent political events in the UK have underlined the need to include overseas international recruitment as a
significant component within an overall university business model.
Sheffield Hallam University (SHU), like many other institutions, has been part of a vibrant and competitive
market that has brought many overseas students to study in the UK. Over the years, the approach towards
international recruitment has become more sophisticated, including the creation of in-country offices and staff
to manage the growing number of applicants.
The reality of these changes in student recruitment has had an inevitable impact upon the teaching provision.
In the case of SHU, successful recruitment activities in the Indian Sub-Continent and South East Asia has
created new cohorts that are predominantly from either demographic; not quite the multi-faceted, rich and
diverse set of cultures of the classical international cohort, with students from a wide variety of backgrounds
studying together. Such recruitment patterns are not without difficulties; it is likely that the influx of a
predominant ‘other’ culture will be a fresh challenge for many academic staff, and when combined with the
relative upheaval faced by the new students, presents a teaching environment that is rich with learning
opportunities and potentially, fraught with misunderstanding and ignorance.
This article explores one approach towards building a successful educational partnership with an institution in
Mumbai, India. Section 2 briefly describes some of the common perceptions of international students by
academic and administrative staff, and also considers some of the feedback received from international
students studying in the UK. Section 3 considers how a values-based approach has informed the fostering of
a mutually beneficial partnership between the two institutions, before examining the specific changes to the
curricula in Section 4. Section 5 concludes and offers suggestions for other HE institutions to explore.
2. OBSERVATIONS
A traditional approach to overseas recruitment has been to enlist the services of an in-country agent, acting on
behalf of a UK institution. Such agents invariably work for a variety of institutions and therefore relatively basic
forces come into play; agents will consider commission rates, service levels (application processing) and
recruitment volume to strategically utilise their own efforts. Competitive practices from institutions has resulted
in an array of refinements that have been built upon this basic economic model. Institutions may have
123
preferred agents, or establish their own office in-country. Some institutions employ ‘consultants’, who have a
more focused approach to recruitment, without making a significant long-term increase in the UK institutions’s
overheads.
Essentially, each stakeholder is attempting to balance costs against commitment. In a fast-moving,
competitive market, there is a tendency to take the view that risks need to be mitigated by reducing the
amount of commitment wherever possible.
2.1 Staff perceptions of international students
The mention of international students often is a catalyst amongst university staff for discussions around
stereotypical behaviours. Some of the traits that are deemed to be regularly exhibited by international students
are:
 not turning up to lectures for  arriving late to a session (there is also the situation
weeks on end whereby they can't cope with changes to timetables, so
they turn up to the wrong venue and wait there).
 not being familiar with the  pass taught diploma stage, then fail the dissertation (to be
module materials addressed in part by ALSS, Ind Prac and RPP).
 unable to log on to the Virtual  no interaction of any value (smiling, nodding heads)
Learning Environment
 not studying between classes  excellent recall in multiple choice exams
 arriving late to a course  arrive at sessions with nothing to write with and on
 handing in work late  poor at extended essay writing
 handing in the wrong work  reluctance to reflect and write
 carrying several referrals  in many cases, working at least one part-time job
 group working often  arriving to class tired and unattentive
disintegrates
 generally poor at organising  unwilling to debate, just want an answer
themselves and others
 plagiarised work  won't be openly critical of peers, everything is good work!
Of course these behaviours are not the exclusive preserve of international students; many UK/European
Union students also demonstrate such traits. However, the experience at SHU has tended to indicate that
where one demographic dominates the population of a cohort, then such behaviours appear to exacerbate the
‘issues’ attributed to a class of international students.
For instance, it may be that a particular class has zero attendees during a week where an assessment is due
for another subject. Such strategic behaviour is much more powerful when performed as a collective, and
becomes much more disruptive for teaching and administration staff who have to manage the emergent
situation. The loss of one tutorial is one week gone from the whole schedule. Such impositions create added
anxiety for all concerned and serve to detract from the whole learning experience. Whilst Carroll and Ryan
(2005) offer some teaching strategies for increased engagement, it is more likely that a holistic approach is
necessary.
2.2 Student perceptions of studying in the UK
The issue of attempting to understand what the underlying challenges faced by international students is
problematic; whilst it is correct to consult the wide body of literature on internationalisation (Knight 2008), it is
also important to recognise the specific local contexts that are pertinent to a particular student studying at a
particular institution. A recent set of focus group interviews has illustrated some of the broad issues that
international students report when studying at SHU. These have been collated into three categories: induction,
support and integration.
2.2.1 Induction
Students often feel there is a lack of effective communication between themselves and the University prior to
their arrival. Although there is an official line of communication present via the local agents, opinions were
voiced that the agents were not equipped with sufficient knowledge to provide the answers and information
required. Instead the students rely upon their peers and seniors (students who have previously completed the
course) to impart the necessary information, though this may be erroneous at times. From an administrative
124
perspective they would see a lot of the information if they were given access to the University VLE. However,
University rules are such that only enrolled students have access to the VLE. One approach that has been
successful is the use of social networking sites such as Orkut, to propagate supporting information and ‘good
advice’, such as the importance of arriving to start a course in good time.
2.2.2 Support
The University offers a number of schemes to help with student development outside of their courses and
modules. These include the University English scheme, including sessions on; planning, referencing, library
searching, and how to read and critique academic articles. The students say that they feel these schemes are
very useful, yet attendance data indicates the contrary. When pursued, the students suggest that their lack of
engagement stems from the fact that there are no immediate assessment rewards for attending these
schemes. Once their assignments are released, they feel that they should focus their efforts towards marks-
related activities, as this is key to obtaining their degree. The long term benefits of continuing with the support
schemes are therefore lost.
2.2.3 Integration
International students often bemoan the fact that they find it difficult to integrate with UK students. The low
enrolment numbers of UK students on the courses was one of the first things the students encountered, a
reality very different to their expectations. They feel that closer integration with UK students would be
beneficial to their studies and personal development. International students often choose to live with other
international students, preferably from the same countries and localities. There are cases of students
choosing from the beginning of their stay in the UK to move into a house with a mixture of UK and
International students. These students generally report that this helped to enrich their experiences and assists
integration into UK culture. International students feel that more should be done to try to encourage this
integration outside the University through the choice of accommodation available.
2.3 A question of culture
Discussions around cultural background of students and staff are challenging, particularly since a lot of the
behaviours that are misunderstood are often attributed to ‘cultural differences’ with no further investigation or
explanation. There is no doubt that culture has a significant effect upon all concerned in a learning and
teaching environment, but rather than leave pertinent questions unanswered we adopted an approach
whereby the differences would be explored, with a view to educating all partners.
3. A PARTNERSHIP OF EQUALS
Bearing in mind the differences that exist between two cultures it would seem prudent to consider how the
relationship between two geographically diverse institutions might be affected. Often, the ‘partnership’ word is
over-used, to the point where it becomes a statement for the purposes of contractual documentation. An
educational partnership suggests something more than a contract where the provision of education is the
business.
During early discussions with a potential Indian partner, it was apparent that something more than a business
arrangement was to be sought. Under the guise of a ‘2+1’ arrangement, both parties understood that
successful management of the transition from the Indian educational system to SHU was vital if the
partnership was to work.
The outcome should be a learning experience that is informed by international issues. The incoming students
will be better adjusted to a different way of teaching. The UK institution will be better adjusted to a new way of
learning. Much more than a verbal or written statement, a values-based approach should underwrite the
operation and maintenance of the relationship.
4. CASE STUDY IN MUMBAI

4.1 Background
Vidyalankar School of Information Technology (VSIT) is affiliated to the University of Mumbai, and delivers a
BSc Information Technology course. The University of Mumbai prescribes certain characteristics of the
courses delivered on their behalf, including: content, contact time with students, assessments and delivery
methods. As a consequence it is difficult for VSIT to exercise any autonomy over how the course is delivered.
SHU has a formal agreement with VSIT whereby students spend two years of a university course at VSIT and
then progress onto the final year of the BSc Software Engineering degree in the UK. This arrangement
exposes the contrast in provision offered by each partner.
125
4.2 The VSIT learning and teaching environment
There are some apparent differences between the education received by the students in their home country
opposed to that delivered in the UK. In brief some of the key observations are:
- Teaching is class based;
- Students remain in class, tutors move from room to room;
- Large quantity of contact hours in comparison with those provided in the UK;
- Students given time to work on assessments in class, rather than the UK model of learner autonomy;
- The Indian curriculum tends to favour content over process;
- Assessment is predominantly by examination, with some small assessed exercises. However the
majority of the marks are allocated to the end of semester exams.
VSIT students generally are taught using the ‘chalk and talk’ model of delivery, and typically follow core texts
prescribed by the University of Mumbai. Similarly assessments have a prescribed, almost template-like
format, the result of which is a set of uniform submissions. This lack of autonomy severely constrains
opportunities to demonstrate creative and innovative interpretations as solutions. At UG level there is no use
of a VLE to share lecture and tutorial information. In our experience students tend to copy each other’s work
before they submit their assignments, inevitably leading to plagiarism. Bearing in mind the UK HE emphasis
upon critical thinking, this singular issue is potentially preparing students to fail.
4.3 The Sheffield Hallam experience
Whilst the SHU Quality Assurance procedures had ensured that learners from VSIT had suitable credit for the
relevant learning outcomes, a number of issues emerged during teaching the final year of the degree at SHU:
1. Final year learners who had progressed through the SHU system for the first two years had
experience of building substantial software applications. Final year learners from VSIT had learned
the same theoretical concepts but had not applied these concepts practically to the same extent.
2. There was a marked difference between the 'autonomy' of SHU learners in comparison with those
from VSIT. It became apparent that VSIT learners did not have the same opportunities to grow in the
same way, in part due to the way they were taught.
3. SHU learners had already been exposed to software development tools used in industry prior to
commencement of the final year, whereas VSIT learners had very little exposure to such tools.
4. Both sets of learners had studied topics related to professional development. However, whilst many
SHU learners had part-time employment, and therefore some work experience to draw from, this was
in sharp contrast to VSIT learners who rarely have any work experience.
These issues are relatively common with overseas partners and one approach is to employ 'flying faculty' staff
to deliver SHU-style teaching in the partnership institution. This was rejected primarily since it is costly to
provide and difficult to staff, but also since it can serve to dictate a one-way relationship where the overseas
'partner' is clearly subservient and hardly a ‘partnership of equals’. This was deemed inappropriate for a
potentially long-term partnership and thus it was decided to develop a mutually beneficial strategy that
comprises three distinct threads:
1. Curriculum development – recognising the rigidity of the existing curriculum that is prescribed, a
bridging curriculum has been developed to directly address transition issues between the two
institutions.
2. Staff development – as a consequence of the adoption of a bridging curriculum, it was necessary to
undertake staff development with both partners. Using the approach that something was to be gained
for each partner in terms of a truly internationalised learning experience, the ‘partnership of equals’
value was being actively upheld.
3. Quality enhancement – the adoption of practices informed by values has been an education for both
partners. The emphasis here is the shift from prescribed regulation towards a more process focused,
values-based culture, that is critically appreciative and enhancement-driven.
4.4 Building bridges
The content-led curriculum prescribed by the University of Mumbai is predominantly based upon theory, with
relatively little opportunity to engage in practical exercises. In particular, the conceptual experience of VSIT
students starkly contrasts with the conceptual and applied application development experience that SHU
students engage with. This was one area where it was felt by both parties that the bridging curriculum could
have an immediate positive effect.
126
Apart from the difference in content provision, a major factor is the mere exposure of the VSIT students to
more autonomous modes of learning, and fundamentally, to experience a taste of applied critical thinking. This
approach educates the VSIT students’ expectations, reducing the shock somewhat when they arrive in the UK
for the final year of study.
4.4.1 Curriculum content
Using the application of knowledge as a key thrust, VSIT students design and develop a web application using
UML, Java and JSP. This encapsulates the Object Oriented concepts and design approaches, application of
the software design goals, and exposes them to the issues surrounding implementation and deployment.
4.5 Impact and reflection
The bridging module has had a positive effect upon recruitment. There are now a number of students who
have expressed a wish to study in the UK in a year’s time and they have started to think about their life and
studies, and how they can adjust to the inevitable changes.
During the first year of delivery, the bridging studies module was delivered as an intensive block, which
prevented the students from synthesizing their thoughts and ideas. In its most recent incarnation, the delivery
approach has settled down and is supplemented by regular video conference sessions. The video
conferencing enables all stakeholders to engage, fostering a collegiate environment where staff and students
learn together. Corrective measures are part of the enhancement process and daily conversation, rather than
being a formal action of quality assurance.
5. CONCLUSION
Whilst this work might appear to be a standard HE enhancement approach, the innovation here is to develop
internationalised variants of each educational partner’s contribution. By internationalised we refer not to
partnership-specific detail, but more to globally-inclusive policies and procedures that will enrich and develop
the existing policies at both SHU and VSIT.
The second innovation is the approach adopted to enable these developmental activities to take place.
Recognising that different institutions and educational systems had disparate local requirements to comply
with, it was decided not to engage in a never-ending process of mapping attributes between the institutions.
Rather, a values-based approach was taken that concentrates upon the key course outcomes for the learners
and the attributes that we want them to exhibit. This has required a common understanding of academic
leadership amongst the stakeholders, with subsequent benefits to staff, learners and both institutions
collectively. Such work has served to create a legacy of change that underpins our model of sustainability.
6. REFERENCES
Carroll, J. & Ryan, J. (eds) (2005) Teaching International Students. Improving learning for all. Oxon:
Routledge.
Caruana, V. & Spurling, N. (2007) The internationalisation of UK Higher Education: a review of selected
material: project report. York England: Higher Education Academy, 147 pages. Available at:
http://www.heacademy.ac.uk/ourwork/learning/international (accessed June 2010).
Knight, J. (2008) Internationalization: A decade of changes and challenges. International Higher Education,
50, 6-7. Available at: http://www.bc.edu/bc_org/avp/soe/cihe/newsletter/Number50/p6_Knight.htm (accessed
June 2010).
127
CUTTING WEDGE OR CUTTING EDGE? - EMBEDDING E-LEARNING
FOR ENGAGEMENT
Paul Carden
London South Bank University
Department of Informatics
London South Bank University
cardenp@lsbu.ac.uk
ABSTRACT
Although there has been a movement away from the classic didactic lecture, with the student as passive
witness, to more active learning, there are still times when the nature of material plays on human nature and
leads the academic down the path of the former for an easier life. This paper will take a case study approach
and examine how the lectures of a first year undergraduate unit were taken from the one dimensional to a
richer blended learning experience, in-class and beyond.
Over the course of a 13 week semester, a class of up to 100 students across the computing and business
information technology programmes, were exposed to a new form of delivery in their timetabled lecture slot for
a business information systems unit. Various technologies were introduced to the students and they were
encouraged to engage with the materials in and beyond the class. The teaching team conducted mini-Kolb
cycle exercises in reflective practice for professional development as e-learning practitioners.
This paper is a report on the development of the project, including an evaluation of its effectiveness in
learning enhancement, support of assessment and stimulation of student engagement. Finally, a set of
recommendations for future injections of blended learning are presented.
Keywords
Active learning, social interaction, zone of proximal development, more knowledgeable other, reflection.
1. INTRODUCTION
When delivering a general module with a wide range of topics to a large cohort of students there is a
temptation to fall into the role of didactic lecturer - working through the material without the perceived
overhead of active student participation and management of the tasks that accompany it. This paper
describes the work undertaken on such a module and the steps taken to fix it as suggested by Richardson [1].
2. RESEARCH GENESIS
This action research was born from the desire to develop and sustain the curriculum design practice that
emerged from a University funded innovation project - 'Embedding e-learning in the curriculum'. Members of
the project team determined that closure of the project should not lead to the end of valuable teaching team
collaboration undertaken during its lifecycle.
A decision was taken to apply the knowledge gained to the delivery of a core unit, Business Systems and the
Organisation (BSO). The module chosen runs across the whole of the first year undergraduate programme.
This ensured the activity would gain maximum exposure to the student body.
3. RATIONALE
BSO has been running for many years. It is a general information systems unit covering a wide range of
topics. The weekly seminars had previously been delivered as a two hour lecture with a discussion at the end.
Engagement had been weak and many students had commented on the difficulty of maintaining an interest
for the whole of the session. Each lecture mapped to a chapter of the core book which gave the sessions a
clear focus but this one-to-one relationship lent itself to islands of delivery with little attention applied to their
interdependencies.
Furthermore, assessment preparation and formative assessment delivery were mostly located in the
timetabled tutorial slots with potentially negative consequences. The tutorials were conducted by a single tutor
from a pool of six. These individuals were reading from the same unit guide, but there was evidence from
seminar discussion, that tutors were sometimes applying personal interpretations to the assessment remits.
This often led to inconsistent student expectations of unit workload across the cohort.
Subject Centre for Information and Computer Sciences 128
The team felt there was a strong case for changing the delivery. A move towards a blended learning approach
could:
 combine the necessary didactic theory presentation with a leveraging of e-learning technologies;
 enrich teaching, for both the lecturers and students, through more varied delivery options;
 maintain the freshness of curriculum design as nurtured by the innovation project;
 facilitate greater cohesion between sessions - the missing ‘glue’ of the unit seminars;
 lead to a better understanding of coursework requirements
 immerse the teaching team in new technologies
4. THE APPROACH
The move to a blended learning approach would be a fresh development for the teaching of the BSO unit. It
would evolve on a week by week basis depending on the successes and failures of the previous weeks. The
newly applied teaching methods needed a supporting process to ensure quality standards were maintained
and that lessons learned were encapsulated in future teaching and learning delivery.
Kolb [2] developed a model for reflective practice - The Kolb Cycle. Reflective practice ‘is important to the
development of lecturers as professionals as it enables us to learn from our experiences of teaching and
facilitating student learning’ [3]. A positive benefit of this approach is that it can lead to the embedding of
reflexivity as routine for continuous development [3]. The Kolb model was chosen to underpin the
development of new practices over the weeks of the semester.
At the chalk face and at the point of any e-reification of teaching and learning, greater engagement between
students and lecturers was to be encouraged. Similarly, the more knowledgeable students were to be coaxed
into helping the other students learn. These aims were supported by Vygotsky [4]. His Social Development
Theory argued that social interaction plays a fundamental role in cognitive development. He proposed the
idea of ‘The more Knowledgeable Other’ (MKO), a teacher or peer with greater understanding of a subject or
ability to carry out a task. The Zone of Proximal Development (ZPD) is the difference in learning potential
between the student working alone and the student working with one or more MKOs. The aspiration was to
lead students to the latter reaches of the ZPD.
5. ACTIVITIES FOR ENGAGEMENT

At the beginning of the semester it was decided to preserve the chapter-to-seminar mappings as there was
nothing fundamentally flawed about this curriculum structure. However, the didactic lecture would be
shortened in length and activities embedded into the session end. This section outlines each of the activities.
The basic essence for each activity was chosen from the end of chapter exercises and discussion points of
the core book. During the running of the activities the tutors present would feed back and forward to other
sessions to facilitate a holistic view of the material.
Weekly meetings were held to promote Kolb-like reflective practice. These meetings were not always face to
face. Telephone conferencing and video conferencing technologies were also utilised. Sometimes this was
out of necessity but it was mainly to develop new approaches to curriculum design.
5.1 Business Sans IT

In this activity students had to come up with a business that did not need IT to function. This was a useful
exercise to start off the series with. The topic was easy to buy into and allowed the social interaction
necessary for better learning to develop (see section 4).
A live blog was used to record comments at the lectern. Photos were taken and also uploaded. Students were
encouraged to use their laptops, iPhones and smartphones to contribute. The live blog was kept open until
the next week and embedded in the VLE for the duration of the semester.
The students engaged well with the activity and there was some interesting and lively discussion. However, a
challenge often faced by the lecturer during open social interaction in class is dealing with discourse that
borders on the offensive or encapsulates prejudices. However, this is not necessarily a negative occurrence.
There were a few comments that were made by class ‘comedians’ for gain in the popularity stakes that might
have been offensive to female students in the class. But the deconstruction of the statements led to a
meaningful exchange on diversity that might not take place often during the students’ engagement with the
curriculum.
129
Also, the activity allowed the teaching team to bring strategic and operational issues into debate and play the
role of MKO. Overall, the collective addressing of the activity increased the chances of the class learning at
the higher level of the ZPD.
5.2 Post-IT Gallery

For this task, a brief case study was presented and students were grouped and chose a name for themselves.
Each group was assigned a perspective to take and expand on with other groups.
Post-Its were used to record contributions. A gallery of these was created along the lecture room wall.
Contributions were grouped by position taken. The students were asked to view the gallery and were
encouraged to comment on the postings. The POST-ITs were scanned by grouping after class and attached
to Forum topics for further discussion.
5.3 MCQ E-tivity

The e-tivity is an approach to delivering dynamic interactive online tasks using a framework developed by
Salmon [5]. The designed MCQ e-tivity (see Figure 2) included watching a fun YouTube video on different test
strategies, taking a reflexive MCQ test, reading some MCQ strategy guidance and watching an episode of the
show Eggheads on BBC’s iPlayer.
Figure 2 – The MCQ e-tivity

The activity allowed the teaching team to clarify assessment requirements and feed over to the Personal
Development Plan (PDP) of another core unit.
5.4 Crims Versus Contras

This activity focussed on web security. The class was divided in to three groups. The first group, The Crims,
had to come up with ideas for exploiting the Internet for illegal and immoral gain. The Contras, the second
group, had to come up with countermeasures for fighting cyber crime. The last group comprised of a panel of
four whose role was to award points to the contributions of the other two groups.
For this task another live blog was used. However, this time the scoreboard facility of the blog was used to
create urgency for social interaction. The blog and scoreboard were displayed on the projector screen. Once
again, the blog content was embedded in the VLE and kept alive for the remainder of the week. During the in-
class discussion the teaching team fed back to the previous tasks were appropriate and fed forward to the
summative assignments particularly the essay.
5.5 Google Maps and Snapshot Graphs

This activity was a demonstration of different ways of presenting data with particular focus on two
technologies, Google Maps and Php.
The University has a diverse student body and this cohort was similarly rich in their backgrounds and journeys
to UK HE. This was used as a driver for leveraging Google maps to create a simple GIS to represent the class
student body. Its creation and the associated data gathering facilitated some engaging social interaction as
students were keen to voice their way onto the projector screen and into the GIS.
130
Next, the tutors introduced dynamic charting of data. An online essay workflow application was used for this.
The students have to engage with it as part of a summative essay coursework. Developed during the
innovation project (see section 2), it allows the students to upload an essay plan and have it reviewed by both
an anonymous peer and their tutor. From a unit management perspective, it is useful to monitor the progress
of the workflow and ensure students are engaging with the assessment.
The class were given a tutorial on using the system and given a timeline for the workflow, from registration to
the eventual writing of the essay. They were then asked to consider how the teaching team could use reports
from the essay application database to monitor engagement and workflow. A tutor then created a web page to
produce a report with charts using the previous year’s data. This part of the activity was not simply a design
and build task but was used to re-enforce the baseline coursework guidance of the unit guide - deadlines,
deliverables and tutor support - a previously identified goal (see section 2).
A forum was created for students to discuss the activity and suggest applications for the data representation
techniques presented.
5.6 Ethics Audit

An ethics case study was the basis for this activity. PowerPoint and the Turnkey student response system
were used to prompt discussion of key points and canvas positions taken in the debate. Stimulation of social
interaction, encouraging and facilitating the involvement of the MKOs and the navigation of the ZPD were
again very much in mind when developing this activity. There were also staff development aspects to it - the
improvement of practice and utilisation of technology for curriculum design.
Figure 2 – Visual Curriculum Design with Compendium

Figure 2 shows an example of an innovative approach (for the team) to planning curriculum design.
Compendium is a tool that provides a rich and flexible visual interface for managing the connections between
information and ideas [6]. It was used to design the ethics topic activity. The design was shared and
discussed over the Internet in a video conference hosted using Adobe Connect.
5.7 Essay E-tivity

This activity revisited the essay plan peer review system as it was going live at this point. Students are
assessed on their engagement with the plan and the reviewing process so previous submissions and
engagement were assessed live and this was used to develop among the students an understanding of the
assessment requirements and standards expected. For some students this is their first essay, so prescriptive
help is provided online. This was also discussed with the help content on-screen.
An essay e-Tivity was also released on the VLE. This e-tivity included; listening to a podcast on creating an
essay plan, accessing mind-mapping software, some related slides to browse and some mind-map examples.
Students were asked to engage with the e-tivity in the week ahead.
5.8 P2P Review System

At the last moment before this session, the author had to return home to an emergency, unable to attend the
weekly seminar. Online communication is an area of interest, so this unexpected event was used as an
opportunity to adapt through video conferencing technology and deliver a presentation remotely live - a first
for the author. Unexpected absence was quite topical with the Eyjafjallajökull volcano affecting some of the
students’ lecturers in this way during this particular week. So, at home, the author donned a Panama hat,
made a steaming coffee, put some Mambo music on, pretended to be stranded in Venezuela and with the
131
help of a tutor in the lecture theatre gave a short presentation - a useful and enjoyable simulation from a
teaching perspective.
There was another unexpected occurrence that day and it concerned a software demonstration. It had been
noted after the first MCQ test that not all students were aware of the core book CD including some practice
tests that would help. So the intention had been, whether face to face in the Elephant and Castle or remotely
in Caracas, to demonstrate the CD resources. This did not go according to plan as anti-virus software rejected
the CD as infected. This part of the presentation was therefore terminated.
6. STUDENT FEEDBACK
The student feedback was generally positive. There were initial misgivings expressed about the length of the
sessions, as in previous runs of the unit. However, it was felt that once the output of the first session
materialised and was posted to the VLE, the students could see objectives of the teaching team and gain a
sense of ownership of the material. In the author’s experience, the course board is the litmus test of student
opinion and the course student representatives reported that their peers had ‘really enjoyed’ the end of
session activities.
7. CONCLUSIONS
Overall, the team felt this first attempt at a blended learning approach was worthwhile with the general
feedback from students being positive. Attendance at sessions and their retention until the end of these
sessions improved on previous years. The levels of social interaction in class were much greater and the
stronger students were allowed to offer leadership and followership to their less capable peers. However,
although, students contributed to the live blog, the main area of disappointment was the poor engagement
with the activity forum.
With each activity the team used Kolb’s model to help improve delivery for future runs of the unit. The
reflections of tutors and suggestions for trying out what has been learned will inform next year’s blended
learning:
 Activities will be published in advance including links to resources including live blogs and video
conferences;
 Ways of improving forum engagement will be examined, such as assessment of contributions;
 Mid-seminar student feedback will be taken from students, the premise for that specific inquiry being that
students are more likely to give useful feedback if there is some benefit perceived. This had originally
been planned but unexpected time pressures resulted in the survey being posted at the end of term;
 Live online engagement will be delegated to another tutor as duality of activity host and monitor impacted
on the quality of activity delivery;
 MKOs will be identified to carry out key roles such as panel members to reduce passive involvement;
 Many enterprise ideas emerged, these will be re-channelled and recorded as a dragon’s den-like activity;
 A help sheet for online client configuration will be produced and added to the unit guide as an appendix;
 New or innovative technologies will be more rigorously tested before being used in class to avoid
embarrassment and delivery failure.
8. REFERENCES
[1] Richardson, D., (2008), Don't dump the didactic lecture; fix it, Advances in Physiological Education,
32, 23-24.
[2] Kolb D.A., (1984), Experiential Learning experience as a source of learning and development, Prentice
Hall, New Jersey
[3] Davies C. and Lowe T., Kolb Learning Cycle,

http://www.ldu.leeds.ac.uk/ldu/sddu_multimedia/kolb/kolb_flash.htm, [accessed 22 May 2010]
[4] Learning Theories, Social Development Theory (Vygotsky), http://www.learning-

theories.com/vygotskys-social-learning-theory.html, [accessed 22 May 2010]
[5] Salmon G. (2002), E-tivities: The Key to Active Online Learning, Routledge, London
132
[6] Compendium, http://compendium.open.ac.uk/institute/about.htm, [accessed 22 May 2010]
133
INTRODUCING A.L.L. TO COMPUTING
Lisa Payne
Department of Computing and the Digital Environment
Faculty of Engineering and Computing
Coventry University
Priory Street
Coventry CV1 5FP
L.Payne@coventry.ac.uk
ABSTRACT
The Faculty of Engineering and Computing at Coventry University is undertaking a radical change, introducing
an activity-led learning (ALL) pedagogy into all its programmes. This is intended to enhance student
motivation and their professional focus. From Sept 2009 every study programme was required to include an
activity learning project for the first 6 weeks of first year students’ study. The requirement was that the project
should challenge these new students and replace some of their formal teaching with student led learning. It
was designed to provide students with an experience of the essence of the career they may have after
graduating and thus motivate students through their course of study. This paper describes the
implementation of this policy in the Computing programmes.
Computing students were asked to develop a website which could be used to collect charity sponsorship
money, similar to JustGiving.com: we called this Sponsors4U. Each week students were given the next part
of their task and a brief introduction to the tools, techniques or concepts which they would need to use. The
project evoked a wide range of student responses and it involved many tensions and difficulties but brought
some clear benefits. As part of a Faculty-wide evaluation students were asked if they would like to see more
use of activity-led learning. For this computing project, 65% said that they would like to see more use made of
ALL and 61% would recommend ALL to others. This paper explores the detailed feedback and some of the
specific lessons learnt from running this project which will need to be carried forwards to future operation.
Keywords
Active learning, ALL, problem based, inquiry, engagement, community of learners
1. INTRODUCTION
The use of active styles of pedagogy is not new. This paper describes the first stages of a radical programme
at Coventry University to deliver much learning using an activity-led approach. The Faculty of Engineering
and Computing is in the process of developing ‘communities of learners’ in which students are actively
involved in professionally focussed learning. The Faculty’s broad strategy is first introduced. The specific
implementation of this for computing students is then detailed. The formal evaluation of the scheme is
described and finally the evidence of ways in which this approach has changed students and their learning is
discussed.
2. THE CONTEXT: ALL IN EC

It has long been recognised that it is important to actively involve students in their learning. This was initially
popularised by Kolb (1984), as Experiential Learning. This work seems to have largely led to students being
given opportunities to undertake practical activities which support material delivered in a more formal manner.
Since then much work has been done in looking at different forms of actively involving students in their
learning. A wide range of approaches have been adopted, under diverse names, such as problem-based
learning, inquiry-based learning, action-learning, activity-based learning and activity-led learning.
There are a number of respects in which such active learning approaches can vary. These have been
identified by Savin-Baden and Major (2004). The most evident differences relate to the nature of the task and
the role of the students. Students may be undertaking Project-Solving Learning: trying to find a solution to a
134
well-defined problem or project. They are members of a project team working on a common problem: the
lecturer provides the information that students require and guides them to a solution. Most early active
learning was probably of this type. Alternatively, the students may not be provided with the bulk of the
information they require but are expected to find it for themselves. Tutors may act largely as advisors or
facilitators. In yet other schemes, the task given to students might be more open-ended where students are
much more independent and develop their own learning experience or goals.
Through much of the Faculty there has long been a desire to improve the students’ learning experience in
order to improve their satisfaction, retention and progression rates. The nature and scale of the issues varies
widely across the Faculty but in Computing there is a desire to improve all of these. Many local initiatives
have been tried but have led only to minor improvements. (For example, the National Student Survey results
remain lower than desired.) The Faculty senior managers decided that radical change was needed to effect
significant improvements: they decided that we would change so that the prime mode of delivery will use
activity-led learning (ALL). The Faculty anticipates that the move towards ALL will bring a number of
important benefits. These include improved student retention and achievement but also engendering a
collective ‘community of practice’ with enhanced student engagement and satisfaction. These benefits are
discussed in Wilson-Medhurst (2008).
The working definition of ALL adopted by the Faculty is that:
“ALL is a pedagogic approach in which the activity is the focal point of the learning experience
and the tutor acts as a facilitator. ... Activity-led learning requires a self-directed inquiry ... [a]
process in which the individual learner, or team of learners, seek and apply relevant knowledge,
skilful practices, understanding and resources (personal and physical) relevant to the activity
domain to achieve appropriate learning outcome(s) or intention(s).” (Wilson-Medhurst et al, 2008)
This description is deliberately very broad and permits a very wide range of types of work to be set and of any
scope. However it makes very clear that the core requirement is that the work set should require students to
be involved in self-directed inquiry or discovery. Of course, active pedagogies were already being used in the
Faculty, to varying extents and in various ways, according to the interests and predilections of staff and the
nature of the particular topic.
At around the time that the introduction of a new pedagogy was being considered it was decided to develop a
new building as the home for the Faculty. The design of the building has come to be intimately linked with the
pedagogic changes. The building has been designed taking account of the needs of ALL. It will, of course,
take some time before this building is available and the intervening period is being used as a period of
development and experimentation: trying out, developing and refining various ALL approaches.
The first phase of ALL implementation, in autumn 2008, consisted of a small pilot in which first-year
Mechanical Engineering students undertook an intensive series of practical labs during the first six-weeks of
their course. In autumn 2009, the requirement was for all first year students on all courses, to be given one or
more projects or activities as a ‘6 week experience’. The projects were to form a large part (if not all) of the
students’ work for this period and to integrate between a number of modules. They were to provide students
with a sense of the nature of their discipline, so that they could see the purpose of the rest of their course.
The projects were to challenge students, so that they would be motivated to apply themselves. Students were
to be provided with a range of support and guidance and, although lectures were not forbidden, students were
expected to need to use their own discovery skills to identify most of the relevant information. Students were
allocated into groups to conduct this work, thereby providing a form of support for these neophyte students.
3. THE COMPUTING ALL PROJECT

The Faculty of Engineering and Computing at Coventry University consists of five Departments, one of which
is Computing and the Digital Environment. There are a large number of subjects and degree programmes
within this department, from Electronics to Aeronautical Engineering, from Games Technology to Forensic
Computing, from Computer Science to Network and Mobile Computing. The Faculty requirement for all first
year students to be given a ‘6 week experience’ led to the department creating 5 separate sets of work. This
paper describes one of those, the project undertaken by students on Computing and related programmes.
3.1 Student Activity, Structure and Timescales

Computing students were given staged activities which together lead to the development of a web-based
system to collect charity sponsorship money, similar to JustGiving.com: we called this Sponsors4U. It was
stipulated that we must include some core programming content in their project, so students also created the
kernel of a back-end, offline, administrative system. Each week students were given the next part of their
135
project and a brief introduction to the relevant tools, techniques or concepts which they would need to use.
Each week was coordinated by a subject lecturer, an expert in that week’s topic. The weekly stages were:
1. Group formation and Team Working; Requirements Gathering

2. Web page development using HTML and JavaScript
3. Usability Testing
4. Linking to a web page, using PHP and MySQL
5. UML data modelling using class diagrams
6. Java programming
During this six week period, alongside undertaking the Sponsors4U project these students were studying
maths, Java programming and a professional skills module. Also, they were completing practical computer
architecture labs. In addition, all Coventry students take an Add+Vantage module, a module selected from a
wide-range of career-enhancing options. The pattern of student activity was broadly the same each week.
There were 161 students, necessitating 26 groups of roughly six students each. It was necessary to divide
students into two cohorts and replicate all activities. Figure 1 shows a typical weekly timetable for one cohort.
09-10 10-11 11-12 12-13 13-14 14-15 15-16 16-17
ALL Professional ALL support

Mon Architecture labs
briefing skills lecture lecture
ALL lab Maths Maths Personal
Tues lecturer
supported
tutorial lecture tutorial
Weds Add+Vantage Java programming lab space available
ALL support
ALL lab
Thurs intern
session
supported lecturer
supported
student debrief e.g.
Fri lab space available
presentations
Figure 1: The typical weekly timetable
Each week started with a Monday morning briefing setting out that week’s challenge. Later that day, most
weeks, they had a lecture introducing relevant technical material. From that point on students were expected
to investigate and learn for themselves, using resources that they either found for themselves or that they
were directed towards. During the week there was some supported lab time during which guidance could be
obtained. Personal tutors were encouraged to discuss overall progress with students at their Tuesday group
meeting. A review session was scheduled for Thursday afternoon, during which subject lecturers had
informal progress discussions with each group. Each Friday students were required to present their
achievement to subject staff and the rest of their cohort. In most weeks this was by a 10 minute, formal,
PowerPoint presentation which provided a vehicle for peer (and staff) formative feedback.
Some of the sessions were not ideally located during the week. Timetabling constraints, of many types, were
very limiting. Of course other resources were also tight too. The staff directly involved with this work had
other timetable commitments. Intern staff also had restricted availability. But a big constraint was the scarcity
of rooms and bookable lab time: other modules were being taught in this period and all other first year
students were being given a similar practical ‘six-week experience’. Thus the weekly timetable was the best
that could be provided and is not presented as being ideal.
Ideally we would have liked to have more booked labs, with an increased level of staff support; we would have
liked to conduct the informal progress reviews a little earlier in the week; we’d have liked to provide the
technical lecture earlier on Monday. The library at Coventry University provides large areas of space for
collaborative work and, whilst located quite close to the department, we would have preferred to provide
suitable student work space in the students’ ‘home’ building. We would have liked a closer relationship
between group tutors and students’ project work, ideally such that tutors were more involved in the group’s
monitoring and guidance. We would have liked those tutor meetings to be earlier in the week so that group-
specific support could be provided in a more timely fashion. We would have liked fewer other classes to be
scheduled during these weeks but, for a range of reasons, this was not possible. The Faculty’s new building
is being designed to address some of the physical constraints.
136
3.2 Student Assessment
The Sponsors4U project itself was not explicitly assessed. However some elements were used for
assessment in relevant modules. Shortly after the end of this project, groups made a presentation and wrote
an academic paper, which were assessed on communication criteria, as part of their Professional Skills
module. Also some small elements of their UML design and their Java programming were used for
assessment in relevant modules. The quality of the website itself was not assessed.
3.3 Student Achievement

Overall, the level of group achievement was impressive though groups succeeded to varying extents. Some
groups contained students who had substantial relevant prior experience but some groups did not contain
anyone with even rudimentary HTML experience. Hence some groups produced systems which, superficially
at least, would pass for professional sites and most groups managed to produce sites with some functionality.
Figure 2: Sample student websites
4. PROJECT EVALUATION
The Faculty ALL initiative is being subjected to systematic evaluation. This involved all students being asked
to complete a comprehensive questionnaire (Wilson-Medhurst, 2010a) and some students contributing further
through focus groups. Staff views were also elicited. The overall results of this Faculty-wide survey are
documented in Wilson-Medhurst (2010b).
There are other sources of information available relating to the students’ view of ALL in the Sponsors4U
project. As has been mentioned, student groups were asked to write an academic paper covering their
achievements. These often contained relevant comments. Also, separately, students were asked to write a
reflective Personal Development Plan (PDP). These also contained many relevant comments.
The systematic Faculty survey showed 65% of respondents ‘would like to see more use of activity-led
learning’ and curiously a lower number, 61%, ‘would recommend activity-led learning to others’. (The
equivalent figures for the engineering projects were somewhat higher, and the Maths project somewhat
lower.) Students were given 42 statements: for each they were asked to rate whether they agreed with the
statement and whether they thought the issue to be important. Each statement was rated on a 1-5 Likert
scale, where 5 represented ‘strong agreement’ and 3 was an expression of a neutral stance. Statements
covered many issues ranging from “I feel part of a learning community where I can learn from others” to
“availability of academic staff for personal support”. Students were also asked to provide summative
comments.
137
Overall students agreed that all 42 points are important to some degree, with averages ranging from 3.57 to
4.38. Views about satisfaction were much more varied from 2.77 to 4.25. The average response for three
statements was under 3, (i.e. overall slightly ‘unsatisfied’). Students were slightly unsatisfied about the
‘balance of workload between the activities’ (though responses varied widely, and were quite uniformly
distributed). On average, students were also unsatisfied that ‘activities were challenging without being too
difficult’, though more students were satisfied than were not. The third aspect about which they were
‘unsatisfied’ was ‘how you are being taught’. Here the pattern of responses was very balanced with almost as
many students expressing agreement as disagreement but with most students giving a neutral response.
Students were satisfied (with ratings over 3.8) with ‘the teaching environment’, that they had ‘an opportunity to
develop ideas in conversations with others’, ‘had a chance to work in diverse groups’ and that they had
‘developed team work’ and ‘developed communication skills’: all issues which they felt were important. Very
few students disagreed to any extent about any of these points.
Students made many positive comments about their experience:
“Working in groups to solve problems and seeing other groups’ work helped solve ours.”
“Got to meet new people and learn from each other.”
“Get to see things from someone else’s point of view.”
“How this course is useful for real life.”
“I learned very much new information and skills so I think this should continue.”
“The helpers were nice and friendly.”
“The work was easy and fun to do with a group”
“Meet lecturers on a less formal level.”
For some, the only positive comment they could think of was “The End” or, more reflectively, “Even though we
were bad at everything we weren’t assessed.”
Students were also asked for comments for improvements too:
“Don’t put so much pressure on new students. At least teach them how to do it before expecting them
to produce something.”
“Easier tasks; was far too much to learn each week for people new to the subject.”
“More practical sessions.”
“Provide more help.”
“Reduce workload as we had less time for other modules.”
“Vary tasks from strongly programming related tasks to something more practical.”
“More relevant to course.”
Some students had very strong negative views: “Don’t do it.” “Disband the idea and never use it again. It was
a waste of my time and money and I’ve suffered numerous breakdowns.” “Cancel ALL” “First teach us
something then make us do this s*** ”. Some of these student comments are being critical of the fundamental
principles of the ALL scheme, as we were being asked to implement it.
5. DISCUSSION
The ALL experience met with mixed student reactions. All groups engaged with the technologies required for
the project and every student now at least understands the way database-enabled websites are created. In
our recruitment we have no requirement for students to have studied computing or ICT. Thus some new
students have few relevant technical skills. However many students, through prior study or personal interest,
have already explored some of these technologies. Many can create a basic webpage using a tool such as
Dreamweaver but few had used HTML directly. A few students had very significant previous experience.
Students were allocated to groups randomly, with the only constraint being that they were in course groups,
for example all Computing Technology students were grouped together. The effect of this was that some
groups contained a student with significant prior knowledge whilst other groups had no such advantage.
Some of these ‘advanced’ students resented being asked to guide their fellow group members: “being put into
a group of skill-less people does not improve my skills.” Whilst there was no expectation that groups would
have any such strength on which to draw the difference in prior experience was evident in the amount each
group achieved. It seemed that some groups without such strength were demotivated by seeing the product
of stronger groups. By contrast, some other weaker groups felt supported by this exposure: “seeing other
groups’ work helped solve ours”.
At the time of writing it is not clear if students are technically more skilled or knowledgeable than in previous
years: first year results generally are not yet available. An early indication though is that the average mark for
the Java Programming module seems to have increased from 43% to 52%. Even if this improvement is
confirmed and replicated in other modules, this improvement must, at least in part, be a consequence of the
recent increase in course entry requirements and the increased proportion of international students. As
138
counter evidence, subsequent activity has made it very apparent that learning was far from universal: skills do
not seem to have transferred well between students. A few students still could not create a very basic
website.
Some students made comments such as: “almost all tasks could be completed by the end of Tuesday”. The
project was chosen since it gave scope for a wide range of implementations: producing a professional quality
site provides a very significant challenge. We did witness some macho, competitive, ego-loaded rhetoric.
There have been some clear, positive impacts in other ways. Undertaking their project in groups has given
students early and significant exposure to team work. This was new to many students so they were given
some guidance (such as Escalate (Moon, 2009) ) and on-going support through their tutor groups. All the
usual groupwork problems arose and needed managing. Not all students enjoyed the experience or were
good at collaboration, but it is recognised that this is an important graduate skill.
“Working in groups to solve problems”
“Do something about slacking group members.”
“We were asked to ‘teach each other’ ”
“Make the students more serious and competent.”
From a very early stage of the year all students knew a group of others: “made some friends and they could
help with things I didn’t understand”. The shy loner, who might try to get through the course without peer
support, seems to no longer exist: shy students were helped into relationships. Some students much prefer to
work alone but at least all students now know some others.
Students were put into a pressurised situation from the start: they needed to present work by the end of the
first week. They immediately found that they needed to spend significant time on their studies: “I did not like
it. Each week I had to learn a new program and do some practical work”. There was no chance for them to
develop lazier, more relaxed study patterns. This good work ethic has carried forwards: there have been
many fewer comments about the overall workload this year.
There is little substantive evidence, but first year students do seem to be more independent this year. Being
encouraged to undertake a project which was supported and structured, but required self-reliance, does seem
to have made it obvious to new students that studying in HE is ‘different’.
6. THE FUTURE: NEXT YEAR

ALL is part of an on-going Faculty strategy to transform the student experience. The Faculty acknowledged
that this approach to learning would not suit all students and that over time our student body would change
somewhat. There is a requirement for us to extend the ALL scheme next year, though the requirement is
more flexible than previously. The form of our delivery next year is not yet decided.
Even if the Faculty requirement were unchanged then there are significant changes we would make for next
year. It is clear we expected too much of students. We would try to pull-back on the scope of the project and
remove the UML and Java programming elements. This would permit more time to be spent on the web-
based system and allow each new technology to be introduced rather more slowly.
Other concerns also merit attention. More student support would be beneficial, preferably through extra
supported lab time. If scheduling permits it would also be helpful to try to balance the groups in terms of pre-
existing skills and other profiling though this is problematic for an activity which starts so early in the course.
However the use of student groups which remain fixed throughout the academic year (as tutor groups) is a
feature which has brought such significant benefits that it ought to be retained.
7. REFERENCES
Kolb, D. A., (1984), Experiential learning: experience as the source of learning and development, Prentice
Hall, Englewood Cliffs, New Jersey
Moon, J., (2009), Making Groups work, HEA Education subject centre, http://escalate.ac.uk (30 Sept 2009)
Savin-Baden, M. and Major, C.H., (2004), Foundations of Problem-Based Learning, SRHE/OUP, Maidenhead
Wilson-Medhurst, S., (2008), Towards Sustainable ‘Activity Led Learning’ Innovations in Teaching, Learning
and Assessment, In Proceedings of the 2008 Engineering Education (EE2008) Conference, ‘Innovation, Good
Practice and Research in Engineering Education’, July 2008, Loughborough University.
http://www.engsc.ac.uk/downloads/scholarart/ee2008/p008-wilson-medhurst.pdf (accessed 7 July 2008)
Wilson-Medhurst, S., (2010a), Knowledge Production within the Innovation System: A UK case study, Higher
Education Management and Policy, Vol. 22 (forthcoming)
139
Wilson-Medhurst, S., (2010b), Activity Led Learning cross faculty evaluation: preliminary findings, presented
to Engineering and Computing Faculty Board, February 2010, Coventry University (Also submitted for external
publication)
Wilson-Medhurst, S. Dunn, I. White, P. Farmer, R. and Lawson, D., (2008), Developing Activity Led Learning
in the Faculty of Engineering and computing at Coventry University through a continuous improvement
change process, In Proceedings of Research Symposium on Problem Based Learning in Engineering and
Science Education, June 30 - July 1, Aalborg University, Denmark.
140
USING EVS AND RESPONSEWARE TO ENHANCE STUDENT
LEARNING AND LEARNING EXPERIENCE
Emma Cliffe James H. Davenport Marina De Vos Nitin R. Parmar Alan Hayes
University of Bath
Bath, BA2 7AY, UK
{E.H.Cliffe, J.H.Davenport, M.D.Vos, N.R.Parmar, A.Hayes}@bath.ac.uk
ABSTRACT
A significant body of literature has made the case that Electronic Voting Systems (EVS) have a positive effect
upon students' learning and their learning experiences. After a successful pilot study in a final year unit at the
University of Bath, in which we used EVS systems to promote deep learning, we expanded upon the use of
such systems and applied this technology to other units within our undergraduate Computer Science degree
programme. This expansion enabled us to adopt a range of techniques and approaches to our teaching with
a view to enhancing our students' educational experience. Examples of this development include the
application of EVS systems to revision lectures for a first year unit and asking final year students to reflect
upon their learning and use this reflection to develop EVS questions as a component of their coursework. We
also used such systems as an alternative mechanism to obtain unit student evaluation. It was in this context,
in the absence of support for free form answers, that we considered the use of web-based ResponseWare
systems with a view to enhancing and complementing the work that we had undertaken in adopting EVS
systems. This paper reports on our and our students' experiences using both of these technologies.
Keywords
EVS, ResponseWare, Improving student learning and experience
1. INTRODUCTION
In any learning environment that guides students to take control of their own learning, motivation and
feedback play crucial roles. A successful learning environment is one that provides both encouragement and
motivation complemented with feed-forward and feedback formative assessment strategies that enable the
students to evaluate their performance, reflect upon, monitor and take responsibility for their own learning.
Although students might seem to be mainly interested in their summative marks, numbers on their own
provide little on which to reflect and even less to improve upon. On the other hand, providing feedback that
students are willing to take on board is essential for helping students to actively take control of their learning
(Race 1994)
Electronic Voting Systems (EVS) are increasingly being adopted in learning and teaching strategies with a
view to facilitating the students' active engagement in their learning. Such systems also provide the lecturer
with a means through which students can be given instant anonymised feedback within a classroom context.
Students are given a device, often called a 'clicker', with a number of buttons and are asked a series of
multiple-choice questions (MCQs). For each question, the students select an answer and press the
associated button on the 'clicker'. The system collects all the student responses which can be displayed to
the class thus a student can ascertain the holistic response of the class to each question but no student can
identify the response given by another student. Consequently, in a safe and anonymised environment, a
student not only receives feedback on where he/she is in engaging with the material, he/she can also
benchmark their performance against the class cohort. A significant body of literature (Crouch & Mazur,
2001; Russell, 2008; Draper, 2009) is present detailing the benefits of such a technology to the student
learning experience. Studies both in programming (Cutts and Kennedy, 2005; d'Inverno et al., 2003) and non-
programming CS units (Stuart, et al., 2004; Davenport et al., 2009) indicate that EVS (also referred to as ARS
- audience response systems) increases student interactivity, enriches the lecture environment and can be
used to facilitate learning in small group teaching sessions and the students' self-study environments.
141
This paper draws upon our experience gained in the use of such systems within the Department of Computer
Science. Initially, EVS systems were used to promote student engagement with the taught material and we
developed revision exercises as a way of providing interactive formative feedback beyond the simply factual
(Davenport, et al., 2009). We did so with a view to providing instant formative feedback, facilitating peer
interaction, enhancing student motivation and triggering deep learning. We have extended this and now also
use it to deliver summative assessment (of EVS questions the students themselves set). EVS sessions then
allow students feedback from their peers on the questions that they submitted. We also use sessions to
obtain feedback to us from students who have studied the unit. Apart from reporting on the use on EVS in a
variety of classroom situations we also report on the use of ResponseWare, a web-based voting system with
the possibility of asking open-ended question. In particular, feedback can be given by tutors on students’
code snippets supplied via an anonymous free text facility, as a pedagogic adaptation of agile coders dojos (a
flexible mechanism to engage a large group in to writing a piece of software interactively - Sato et al. 2008).
We report on a questionnaire used to determine the students' voice in the effectiveness of this development.
2. EVS AND RESPONSEWARE

At the beginning of the 2008/2009 academic year, the University of Bath,
purchased 200 TurningPoint RF ResponseCards (or 'clickers'), with 200 added
later. Operating on a radio frequency, thereby not requiring line of sight between
the clicker and USB RF receiver, and with the relevant software installed by default
on lecture room computers, the technology was relatively straightforward to set-up
in a teaching space prior to a lecture. Interactive question slides can be created
within Microsoft PowerPoint using the TurningPoint 2008 toolbar and easily
dropped into an existing presentation. The day to day administration of the
Figure 1: A clicker
TurningPoint EVS is managed by the Audio Visual team, supplemented by
pedagogical and technical support by the e-Learning team (one-to-one meetings and group workshops), and
in-class support.
ResponseWare enables students to respond to presented questions in real-time using a Wi-Fi or data
connection on a mobile phone or laptop. Importantly, the software can be used seamlessly in mixed
environments alongside traditional clickers, with results collated by the TurningPoint 2008 software. While
TurningPoint EVS allows students to respond to multiple choice, alphanumeric and multiple response type
questions using a clicker, ResponseWare Web allows students to respond to open questions with free text.
3. EDUCATIONAL CONTEXT
3.1 Networking
This is a final year unit, compulsory on most Computer Science
programmes at the University of Bath. Due to the cohort size (58 in
2009/10) and highly factual nature of the material, the lecturer, who
has taught this course for several years, has always struggled to
engage the students. Problems classes have ended up being
another form of lecture. Hence the suggestion of EVS seemed like a
plausible technique. In 2008/9 we tried fairly conventional use of
EVS: the major distinguishing feature, described in Davenport et al.
(2009) was the use of questions with "no right answer", to promote
engagement with the deeper levels of the material. This proved
relatively successful, so in 2009/10 we also required the students
themselves to set such questions. The process went as follows: two
sessions of questions were set by the lecturer, which the students
answered individually. As well as their direct pedagogic benefit,
these introduced the students to the technology and the style of
question expected. The students were divided into fourteen groups
of approximately four. They were introduced to Bloom's taxonomy
(see Williams (2006) for more details) and a public resource
Figure 2: Individual then group vote
(Appendix C of Carneson et al., 1996) describing its interaction with
142
MCQs. Each group was allocated a topic, typically a part-chapter from the course text, by selecting from a set
of sealed envelopes. The general rubric read as follows: "This question should not have an obvious answer,
and should demand some thinking and discussion". Each group submitted its question and answer slides to
the lecturer, who vetted them for factual correctness and pedagogic appropriateness. At this stage, one
group was told to go and rethink, and two more had the answer slides enhanced. Each question was asked
twice to the entire class in an adaption to the Mazur sequence (as described by Nicol & Boyle (2003)): (1) Ask
the question, students vote individually. (2) When all have answered, but before the answers are shown, ask
the groups to confer. (3) Part-way through this, show the individual answers. (4) Tell the students to press
the group clicker, according to the group consensus. Marks (10% of the unit total) were allocated based on
the correctness and difficulty of the question, and the extent to which it made the group think (assessed by
observation and considering group and individual vote patterns). Figure 2 shows one such question, with first
the individual, then the group, responses.
Unit evaluation responses suggest that some students perceive the unit as merely requiring the memorisation
of large amounts of facts. In addition students enter the compulsory unit with very different levels of interest
and prior experience of the subject. It was hoped that working in groups to discuss and answer questions,
explore an area of the subject in depth and design a question which highlights a misconception of their peers
would challenge assumptions about how to study the subject. It was expected that promoting peer interaction
and introducing a competitive facet would raise levels of motivation and, through use of imposed ("Imposed"
groups is in fact departmental policy for summative assessment, to ensure mixing of ethnic groups and prevent de facto
segregation) rather than peer-selected groups, sharing of prior experience and knowledge.
3.2 Programming
EVS was also used in Programming, a compulsory double unit for all first year computer science students.
The intention was to create a more interactive environment. In previous years, revision quizzes were
organised in the lecture by means of pen and paper and via the e-learning platform Moodle. In this unit two
programming languages are taught: Python and Java. The Python revision lecture was a traditional quiz,
students provided questions from which the lecturer would selected the most appropriate ones to compose a
quiz. For the Java revision lecture, an EVS quiz was offered but without the competition element.
After the first programming exam students were given the opportunity to attend remedial lab classes if they
wished to recap some key ideas at a speed determined by those present. ResponseWare was used by the
tutors to ask a mixture of MCQs and questions inviting a free text response enabling less able or less
confident students to anonymously put forward their own ideas without risk of embarrassment. As these were
remedial classes tutors were asked to ensure they did not move on until all students were ready. We used a
ResponseWare adaptation of a coders dojo (Sato et al., 2008), an agile programming mechanism to allow
1
participants to improve their (coding) skills. In one of its traditional forms, the randori , two people start the
exercise. After five minutes one of them leaves the pair to be replaced by one of the other participants. The
rotation continues until the problem is solved. The approach
improves participants’ skills by doing and by seeing other
people doing. For the remedial classes we adapted this
version by using the software as a sounding board for
suggestions allowing students to participate without having to
step in the spotlight. ResponseWare also allowed students
to let the tutors know whether they were ready to move on.
Figure 3 gives a screenshot of one the sessions.
Figure 3: ResponseWare slide with answers

3.3 Unit Feedback
Feedback is not a one-way mechanism from staff to students. Feedback from students allows staff to reflect
and improve the unit for following years. At Bath, students are surveyed regarding the effectiveness of each
unit via paper-based questionnaires or via the web. Summary results are presented at the Staff Student
1 See http://codingdojo.org/ and http://me.andering.com/2008/10/21/as-a-programmer-i-want-to-go-to-a-coders-

dojo-so-that-i-can-improve-my-skills/ for more information.
143
Liaison Committee. Student representatives can then pass this information on to their peers. Unfortunately
this is a timely process. This year we decided to run in parallel an EVS evaluation session for the
Programming unit. With the EVS questionnaire we wanted to create an environment of immediate feedback
and encourage an environment of discussion between students and between students and the lecturer.
4. EVALUATION
4.1 Networking
A group of 23 students responded to the evaluation questions, of these 83% agreed or strongly agreed it was
good to be more interactive in lectures. Students were asked to consider various aspects of the group work.
65% felt that answering as a group was more educational. 60% of respondents found working in groups to
answer questions to be beneficial or very beneficial, however, as might be expected with imposed group work,
17% felt it was a sad necessity and 4% found the experience disastrous. Respondents were more divided in
their opinions on designing questions as a group, while 43% did find this experience to be beneficial and 17%
were unconcerned by the experience. 29% felt that the group work was a sad necessity and a further 13%
found the experience disastrous! Despite these opinions on group question design, when asked to consider
whether that involvement in designing a question had improved their understanding of the subject 45%
strongly agreed that it had with a further 23% agreeing and 14% neutral. Hence a small majority responded
positively towards all aspects of group work except the experience (rather than the outcome) of designing
questions in groups: the questions were marked and this may have contributed to this response. The lecturer
felt the questions produced by the students were generally good, and could be reused in future deliveries.
Some questions proved difficult for other students to answer which suggests that they adequately highlighted
areas of misconception. The lecturer also noted that this cohort was far more active than previous years in
discussing revision topics among themselves, and trying to "see the wood for the trees".
4.2 Programming
At the conclusion of the second quiz 76% agreed that it was good to be interactive in lectures, 44% liked the
EVS quiz, 28% the paper quiz with 24% liking both. When students from programming 1 were later asked for
their views on the quizzes 59% agreed that EVS quizzes are more effective as they allow comparison with the
rest of the class. Those who in hindsight regarded quizzes positively still favoured the EVS quiz however,
42% agreed that they would prefer not to have quizzes with only a third agreeing that EVS was useful for
receiving feedback from lecturers. This contrast between views in the quiz and respondents to later questions
might suggest students absent themselves from quiz classes if they don’t believe them to be effective or that,
in hindsight, they were unsure of the expected outcomes of the activity or how to use the feedback. Another
interpretation is that stand alone quiz classes seemed less effective as the same group reacted positively to
the suggested model of regular short quizzes used to inform the provision of lab exercises aimed at resolving
uncovered misconceptions. This idea was put forward by the two lab tutors and we note such a model forms
the basis for Cutts and Kennedy's (2005) connected learning environment. There was little consensus
whether student written questions or lecturer written questions motivate them to improve. Some students
found them equally useful while others showed (sometimes strong) preferences for one or the other. From a
lecturer's perspective, the EVS presentation resulted in more discussion than the pen and paper variant
despite the fact that in the latter students wrote their own questions and hence competed. While the lecturer
had to write the EVS questions, this guaranteed the coverage of the entire syllabus and a range of difficulties.
4.3 ResponseWare
Students who attended the three remedial labs agreed that using ResponseWare enabled them to receive
effective feedback (93%, 76%, 100%) and to influence the direction of the class (87%, 39%, 71%). In 2
classes they were asked whether the system should be used in usual exercise labs and this idea was very
positively received (80%, 83%). In a subsequent survey 69% agreed that using ResponseWare to suggest the
next step in "live code" classes would be an effective way to check their understanding and requests for "live
code" classes are a frequent occurrence on unit evaluation forms.
The tutors found ResponseWare allowed them to monitor the understanding of all individuals in the group but
that it was challenging to prepare sessions if responses completely controlled the direction of the class. While
144
open responses allowed students to put forward their ideas the real time display of this input was very
inefficient for the tutor and incomplete for the students. Despite this tutors noted that discussions in terms of
student ideas rather than tutor generated examples were more realistic. The tutors agreed that, in theory,
ResponseWare allowed mining for problem areas rather than relying on serendipitous discovery. However,
they did not feel that ResponseWare could be used in standard exercise classes as students' perceive these
as individual programming time on pre-determined tasks. They suggested that separate labs, like the
remedial classes, bringing together concepts from lectures with "live code" in the manner of a coding dojo
could use ResponseWare if the inefficiencies and overheads of using the system for free responses could be
overcome. A student suggested that a technology such as GoogleWave might be useful.
4.4 Unit Feedback

Students were engaged and felt more involved in the live survey which resulted in a lot of discussion. When
later asked whether they agreed it was more effective 63% of respondents felt that it was with 78% agreeing
that clickers were useful for giving feedback to lecturers. The outcomes of both the EVS and the traditional
unit evaluation surveys correspond. The students participating mentioned that they would prefer the lecturer
to give their expectation of the results in between the voting and the announcement of the results. The
downside of using the EVS as a unit feedback mechanism is the lack of free-form comments and suggestions.
In the web-based version details on the aspects of the unit they liked and disliked and suggestions for
improvement can be entered. For this reason, we are considering using ResponseWare, for this type of
activity. From a lecturer's perspective, asking for unit evaluation in class with immediate and public feedback
can be rather daunting. Students can see the lecturer's reaction and there is no escape from possible
negative votes. It does however open a door to more interaction between staff and students and gives
students some proof that their comments are considered and not just gathered and placed in a drawer.
5. CONCLUSIONS
The final year cohort, despite not appreciating all aspects of the imposed group work seemed to agree that
involvement in writing and answering questions improved their learning. The first year cohort had more mixed
views and may benefit from a clearer educational rationale or a more integrated use of EVS. The first year
students appreciated providing unit evaluation using EVS and the lecturer found this a useful activity.
Programming students frequently ask for “live code” classes and our first uses of ResponseWare suggest that
students view this as an effective way to learn however we feel that a more natural solution and efficient
solution to the display and use of the responses would be required for more serious use.
6. REFERENCES
 Carneson, J. Delpierr, G. and Master, K. (1996) Designing and Managing Multiple Choice Questions,
http://web.uct.ac.za/projects/cbe/mcqman/mcqman01.html.
 Crouch, C.A. and Mazur, E. (2001) Peer Instruction: Ten years of experience and results, American
Association of Physics Teachers, 69:9, pp. 970 – 977
 Cutts, Q.I. and Kennedy, G.E., (2005) Connecting learning environments using electronic voting
systems. In Proceedings of the 7th Australasian conference on Computing Education. pp. 181-186
 Russell, M.B. (2008). Using an electronic voting system to enhance learning and teaching,
Engineering Education Vol 3, Issue 2. 2008
 Draper, S.W. (2009) Catalytic assessment: understanding how MCQs and EVS can foster deep
learning, British Journal of Educational Technology, 40: 2, pp. 285 - 293.
 Davenport, J.H., Hayes, A., Parmar, N.R. (2009). The use of an Electronic Voting System to enhance
student feedback. In the proceedings of: 4th Plymouth e-learning Conference: Boundary Changes:
Redefining Learning Spaces. http://opus.bath.ac.uk/12505
 d'Inverno, R.A., Davis, H.C. & White, S., (2003). Using a personal response system for promoting
student interaction. Available at: http://eprints.ecs.soton.ac.uk/9202/ [Accessed February 8, 2010]
 Nicol, D.J. & Boyle, J.T., (2003). Peer Instruction versus Class-wide Discussion in Large Classes: a
comparison of two interaction methods in the wired classroom. Studies in Higher Education, 28(4),
457-473.
 Race, P. (1994). The Open Learning Handbook: Promoting Quality in Designing and Delivering
Flexible Learning. Kogan Page Ltd.
145
 Sato D., Corbucci H, and Bravo M. (2008) Coding Dojo: An environment for Learning and Sharing
Agile Practice. In the Proceedings of Agile 2008 Conference. Toronto, Canada, August 2008.
 Stuart, S.A.J., Brown, M.I. & Draper, S.W., (2004). Using an electronic voting system in logic lectures:
one practitioner's application. Journal of Computer Assisted Learning, 20(2), 95-102.
 Williams, J.B. (2006) Assertion-reason multiple-choice testing as a tool for deep learning: a qualitative
analysis. Assessment & Evaluation in Higher Education, 31(3), 287-30
146
ENGAGING THE ATTENTION OF TECHNICAL STUDENTS IN
LEARNING BUSINESS SKILLS
Marion Gerson
Sheffield Hallam University
Howard Street
Sheffield S11 8RN
m.gerson@shu.ac.uk
ABSTRACT
Seven changes made to the delivery of a Business Process module resulted in a substantial improvement in
the engagement of technical IT students and in their learning of business skills. It is thought that no single
change was responsible but that it was the combination and complementary nature of the changes that
produced the desired effect.
Keywords
Engagement; Technical students; Business; Skills.
1. THE BUSINESS PROCESS MODULE

The Masters level module in Business Processes had been set up originally for recent graduates working in
local business and industry. The core assignment was a real process improvement exercise carried out by
students within their own companies, together with a reflection on learning. It had proved a popular and
effective format for aiding local companies.
It was decided that this module should become part of a new MSc course in Technical Consulting. One of the
big selling points of this course is that it has extensive SAP training and accreditation as part of the course.
This makes it attractive to technical graduates, especially international students, wanting to substantially
upgrade their academic and professional qualifications and skills.
2. 2008/9 - WHAT WENT WRONG

There are two entry dates each year for the Masters course, in September and in January. However students
attend modules that are taught once a year so students starting the module in October 2008 were at different
stages of the degree and not a coherent group. Some (but not all) had several years of experience in IT jobs.
They were technically strong but many had a very weak grasp of business principles.
As Module Leader, I had decided that it would be best if the students carried out the process improvement
task in groups since the ability to work with and for other people is an essential business/consultancy skill.
Three business contacts had promised projects that the students could undertake. But when the students
were due to begin the main assignment, one of these contacts was away ill, another's company was having
serious financial difficulties and the project suggested by the third turned out to be very stretching at this level.
The first two weeks of lectures and tutorials focussed on academic aspects of team working, including Belbin
roles [1] in order to try to form balanced teams. But several students struggled with understanding what these
roles meant and found the assessment methodology very alien. It was possible to form two groups of 4 each
but the last group then consisted of the three students whose lack of attendance and engagement meant they
had not even attempted the Belbin self-assessment. It also became clear that some of them were either not
accustomed or not willing to engage in team working.
147
That year's delivery was not a success. There was very poor attendance by the end. Furthermore although
everyone passed the module eventually, only one student got a good mark. I felt responsible for the poor
learning and poor performance and resolved that the next year's delivery would be better. I tried to analyse all
the ways in which I had failed to ensure students' attendance and attention. With help from the students'
reflective reports and from colleagues I tried to imagine all the barriers to engagement and ways to overcome
these.
3. CHANGES THAT WERE INTRODUCED FOR 2009/10

The students who started the module in October 2009 were very similar in many ways to those of 2008.
There was again one student from the UK with the other 12 coming from Fiji, India, Malaysia, Pakistan, Sri
Lanka and Zimbabwe. Some were new to the course while others had been studying in Sheffield for 9
months. They were technically able but many were not used to working in teams and some had little
understanding of business.
But this time there were changes. These involved:
1. Having the two one-hour slots timetabled consecutively and in a room which had a computer on every
desk
2. Creating learning materials that involved in-class demonstrations, problems and work, often in
groups, rather than using the standard lecture and tutorial formula
3. Making sure that the first lecture and tutorial were interesting and motivating for the audience I now
knew I would have, then building on its success to keep the students coming back for more
4. Setting early assessment tasks that rewarded continual engagement and team working from early on
5. Creating smaller project teams (3 rather than 4) and having these teams sit and work together in
class
6. Using a method of putting students into teams that deliberately cut across cohorts and yet could be
seen to be a fair method of allocation
7. Expecting the students to find their own business process to investigate and improve
These changes are shown in figure 1 using an Ishikawa Cause and Effect Diagram [3]
Timetabling Mixed teaching Relevance & motivation

method from the start
Success in
engaging
students
Early Teams of 3 Method for Have to find

assessment assigning teams own process
Figure 1. Cause and Effect Diagram for the successful engagement of students
3.1 How the Changes Were Intended to Contribute to Student Engagement

The change in timetabling meant that the students only had to remember this module once a week and it was
worth attending even if it was the only taught element in the day. It also facilitated the mixed teaching
approach that allowed the two-hour session to be divided into digestible chunks of listening and doing.
148
As Module Leader having worked for many years as a consultant in process improvement, it wasn't difficult in
principal to make the module relevant to these students. But the key to doing so in practice was trying to
understand what they might already know and what their ambitions might be. Some of this understanding had
been acquired from the 2008 cohort and also from discussing the course with other teaching colleagues.
In the first session students were set a short individual assignment with very specific instructions (at least 3
real examples and 3 references using Harvard referencing) and a very tight deadline, but carrying only 10% of
the marks. This was almost certainly ahead of any other assignment deadline for these students so they
naturally engaged quickly and carried out the task with gusto and with a high level of research. Quick and
positive feedback on this task engendered a feeling of enthusiasm within the module.
The next assignment for the students was a team presentation about the process they had chosen to
investigate. This made it essential for each team to get on and find a suitable process quickly since they
could see that the date for the presentations was frighteningly close. So every opportunity given them to work
together in class on some aspect of the project was seized upon. There was soon an understanding within
this cohort of students that each task in class was relevant and related to something they were doing.
Consequently they didn't dare miss sessions without good cause or, if ill, would make sure to catch up with
their group as soon as possible.
Because students were at different stages in their course and many had part-time jobs, they had different
timetables and outside commitments. Having most project teams of size 3 encouraged a greater sense of
responsibility but also made it just a little easier to arrange meetings and visits to the client.
To assign teams the students were asked to form three equal groups (only 12 of the 13 were present) with the
people they preferred to work with. Then the students in one group were given blue cards, those in the next
group had yellow cards and the third group had blue cards. Each person was asked to write their name on a
card. Placing these face downwards and after some shuffling of the cards, one red, one blue and one yellow
card were picked and stapled together. Another team was picked in the same way, then the third and fourth.
Since the absent person was in hospital for an operation, he was put into a team with a friend on the
understanding that he could engage as quickly as possible when he had recovered. The students were not
entirely happy about the way in which every group of friends had been broken up, but they recognised it as
fair and understood the explanation that this was to simulate what it would be like as a consultant when they
couldn't choose who they worked with.
Asking students to choose their own process to investigate turned out to be an excellent move. Because
many of them worked part-time to help support themselves, they mostly chose processes that at least one
member of the team knew well. The processes came from a care home for the elderly, a food processing
company, a hospital pharmacy department and one of Sheffield's large taxi firms. In each case the problems
were clearly identified. Sensible cause and effect diagrams could be set up and data could be obtained to
show which causes were the major ones, why this process needed improving and/or the costs and benefits of
the proposed actions.
4. RESULTS FOR 2009/10

You can guess that the outcome was largely a success otherwise I wouldn't be presenting this paper.
The improvement was substantial. Apart from illness, for which students always sent professional email
apologies, there was 100% attendance throughout and the level of engagement was high.
In every case the business report that was prepared by the team was of genuine value to the organisation that
had been host. The reports were also written in an appropriate style; students had clearly spent considerable
effort on making the reports concise and informative. The author of this paper is well known to be a hard
marker, but two of the reports were marked in the 80's and the lowest mark was 70%.
Having put so much effort into the team work, it was disappointing to see that the students' reflective reports
were not of the same standard; no-one scored above 65% and the average was 55%. However this final task
was competing with large assignments on more conventionally taught modules and, for some students, with
their MSc dissertation. In any case, whether they could explained it well or not, the students had engaged
fantastically well with the main part of the module and all had clearly been using and developing business
skills of importance to a technical consultant.
149
5. WORKING WITH A NON-TEAM PLAYER
There was one individual who was unwilling to work with other students who he regarded as less able and
less motivated than himself. He also turned up at least 5 minutes late to almost every class.
J was two hours past the midnight deadline in his submission of the first assignment. He was shocked and
surprised to be told that he was too late and so his work should get zero. An explanation of this didn't just
quote the University regulations but also put it in the context of real life consultancy where on-time delivery is
crucial. As a concession he was told we could award the pass mark (40%) for the task by regarding this as a
referral but that, if he wanted to contest the decision, the authorities would definitely back the Module Leader
in awarding zero. Reluctantly he accepted the penalty. It was in discussion of this with the Course Leader
that it transpired that J had a reputation for being very litigious and had previously clashed with several
members of staff and with the Course Leader.
When students were put into teams, most accepted the way these were formed. The only real exception was
J, who was furious. He had elected to be in a group with the UK student, the only person he identified as
being up to his own standard. Over the first 3 or 4 weeks he fought hard to be allowed to do a project on his
own, especially since he was in the team of four.
In the end his fellow team members, generally peaceable people, came to complain that J had hijacked the
group presentation (I could tell that from its inappropriate content and its delivery and had given it a low mark).
Because the assessment criteria for each assignment had already been published, I could point out to J that
he would lose many marks if he couldn't, for example, reflect on the team's performance in his individual
reflection. He did the sums and decided that it would be better to be in a group that might get a lower mark
than to work on his own and be guaranteed of losing those marks; self-interest meant making sure the team
did well, even though he was far from happy about this. We then agreed that the whole team could make
their presentation the following week; fortunately this did turn out to get a good mark.
It wasn't entirely plain sailing from there but the important battle had been won. Still, J continued to fight right
up to the final Award Board. So it was a surprise and delight to receive the following in an email from him (the
emphasis is mine):
"Thanks for the feedback and I really appreciate each and every class of yours. With regards the final
marks, well, I guess nothing else can be done accept for me to reflect upon it and use it to my benefit
for the future. I would like to take this opportunity to give you my most sincere gratitude for being able
to be your student throughout this module." (Teo, personal communication)
6. COULD FEWER CHANGES HAVE ACHIEVED SIMILAR RESULTS?

Clearly the application of process improvement thinking has caused the same module with the same lecturer
and the same kind of students to be substantially better. But could the same result have been achieved with
fewer changes? Was there a small number of these changes that were more important than the others?
It would in principle be possible to carry out a highly condensed fractional 27 experiment in which changes
were or were not made, using the design shown in Figure 2 below.
assessment
Teams of 3
Motivation
Timetable
formation
Find own
teaching
Cohort
process
Mixed
Team
Early
A Yes Yes Yes Yes Yes Yes Yes

B No Yes No No Yes No Yes
C Yes No No No No Yes Yes
D Yes No Yes No Yes No No
E No No Yes Yes No No Yes
F No Yes Yes No No Yes No
150
G Yes Yes No Yes No No No
H No No No Yes Yes Yes No
Figure 2. Design for an experiment to estimate the effect of individual changes

The advantages of such a design are that 4 cohorts have a particular change 'on' and 4 have it 'off'. In
addition there is orthogonality [2, 4], which means that for each time a particular change is 'on' (and each time
it is 'off') there is also an equal number of 'on' and 'off' for every other change. Provided sensible measures of
success for a cohort could be identified (e.g. attendance, average cohort mark, etc.) a simple analysis of
results would show which changes have the greatest or the least effect.
Cohort A has already been done, so it would take the author 7 more years to carry out this study, always
supposing the MSc course itself (or at least a similar one) continued to run and to recruit in much the same
way. In reality this is unlikely to happen.
An alternative experimental approach would be to ask which changes the Module Leader would be most
prepared to give up if she had to and then to drop these sequentially until the performance of cohorts started
to suffer. Again, several years would probably be required.
But neither of these approaches properly addresses the possibility that a positive outcome may be due to a
complex interaction. So the effect, for example, of choosing small teams (so there can be no passengers)
using a semi-random method combined with early assessment that encourages those teams to start working
together and an urgent need to find a suitable process may be very much greater than the sum of effects of
each of these factors separately. Combining these with providing enablers in the form of timetabling, class
exercises that assist the team in its progress and an understanding from the beginning of how and why this
fits into a technical qualification may similarly have much more than a simple additive effect.
On reflection it seems to me that all seven of the changes were interlinked and important contributors to
student engagement in this module. Taking just one of those factors away might result in an outcome similar
to that of 2008, something we prefer not to risk.
7. REFERENCES
1. Belbin, R Meredith (2010) Management Teams: Why They Succeed or Fail, 3rd edn. Butterworth
Heinemann, Oxford
2. Gerson, M. (1999) Improving Control of Large-scale Continuous Processes by the Use of Designed
Experiments, Proc. 2nd International Conference on the Control of Industrial Processes, Ed. M.
Gerson and P. Barker, University of Newcastle upon Tyne, Newcastle.
3. Hoerl, Roger and Snee, Ronald (2001) Statistical Thinking: Improving Business Performance,
Duxbury, Pacific Grove, California
4. Montgomery, D.C. (2009) Design and Analysis of Experiments (7th Ed) John Wiley & Sons Inc,
Hoboken NJ
151
DEEP LEARNING VIA STUDENT ENGAGEMENT WITH SEMI-
AUTOMATED LECTURE TRANSCRIPTS
Dr Marcus Lynch Philip Phelps
Department of Computer Science & Creative Department of Computer Science & Creative
Technologies Technologies
University of the West of England University of the West of England
Coldharbour Lane, Frenchay, Bristol Coldharbour Lane, Frenchay, Bristol
marcus.lynch@uwe.ac.uk Philip3.Phelps@uwe.ac.uk
http://www.bit.uwe.ac.uk/~mlynch/index.html http://www.zenpho.co.uk
ABSTRACT
We describe the process and interim results of our HEA-ICS Development Fund project. The project is
funded until the end of July 2010, so we will present to the HEA-ICS Conference at Durham final evaluations
of the project, drawing on student feedback and performance in their final examination. The cohort involved is
composed of approximately 75 final year students (drawn from several awards) who study the module
"Multimedia Systems: Contexts and Applications", for which automatically generated lecture audio transcripts
have been corrected.
Final student feedback will cover the period from 29th January 2010 up to their subject examination on 20th
May 2010. We also explain the method used to collect and evaluate student feedback and usage metrics.
Students are asked for feedback on the project at four points throughout this period
We describe the nature and extent of student involvement in the project. Student uptake of the system has
been strong - with 70 active students on the system, submitting 1380 segments, i.e. 7h54m24s (28464.99
seconds) of transcription for approval.
We outline plans for the next phase of project development, including further automation of several stages
and integration with Virtual Learning Environments (VLEs), in particular with Blackboard.
Project work on automating transcription has involved:-
 extending and modifying Open Source projects for mp3 segmentation;

 developing across platforms (Win32, MacOS X, and Linux) for the use of automation tools such as
AutoIT and AppleScript;
 programming of a PHP based front-end in use by students and staff.
(All code is Open Source and available on request)
At the HEA-ICS August Conference, we will present our final evaluation of the project, including reflections
and lessons learned about the usefulness and practicalities of developing and extending our approach to
other subject areas at UWE, as well as through potential collaborations with colleagues in other HE
institutions.
Keywords
Speech recognition, deep learning
152
1. INTRODUCTION
Overview of HEA-ICS funded project

This project evolved out of a desire to provide hearing-impaired students with transcripts of lecture materials.
All users (hearing-impaired or otherwise) can benefit from spoken word becoming text; since it can be
powerfully manipulated with data-mining, searching, and cross-referencing techniques for teaching and
learning purposes.
Early work in this area (Lynch & Phelps, 2009) produced a system to generate first-draft transcripts by
automating the flow of data from mp3 recordings of lectures, through commercial speech recognition software
(Nuance/Dragon Naturally Speaking), and out for editing/correction by a single person in an open-source
time-coded audio transcript editor (Transcriber, obtained from Sourceforge).
This HEA-ICS funded project expanded the above work into a collaborative editing/correction system, where
multiple users simultaneously edit first-draft transcripts, working together quickly to produce usable, indexable
and searchable transcripts.
The major area of investigation for this HEA-ICS project has been to evaluate the learning benefits of direct
student involvement in the transcription process, and subsequent use of the system to support revision.
2. STUDENT INVOLVEMENT AND EDUCATIONAL CONTEXT

The current HEA-ICS project attempts to involve students actively. Students on a final-year Multimedia
Systems module collaboratively edit and correct first-draft transcripts as part of their assessment regime. For
each 42 seconds of edited audio/text, students receive 0.5% of the module total marks, up to a maximum of
5%.
This small incentive has encouraged 70 students in the cohort to participate in transcript editing/correction. In
doing this, students actively engage with the material at least three times, in at least three modes (visual
auditory & kinesthetic)
 during lectures;
 during transcription/correction;
 during examination revision;
 by reading;
153
 by listening;
 by writing/typing (trancript edits, and/or personal notes in lectures).
3. SURFACE LEARNING, DEEP LEARNING AND LEARNING STYLES

One of the main concepts we have focused on is that of deep learning versus surface learning. Marton and
Saljo (1976a & 1976b) based their concept of deep learning on the depth/levels of processing theory.
Craik and Lockhart (1972) explain: "This conception of a series or hierarchy of processing stages is often
referred to as "depth of processing" where greater "depth" implies a greater degree of semantic or cognitive
analysis.
Retention is a function of depth, and various factors, such as the amount of attention devoted to a stimulus, its
compatibility with the analyzing structures, and the processing time available, will determine the depth to
which it is processed.
Memory [can be] viewed as a continuum from the transient products of sensory analyses to the highly durable
products of semantic-associative operations."
The ideas are summarised neatly by Kearsley (2010): "The deeper the processing, the more that will be
remembered."
Craik and Lockhart describe the behaviour of memory in general terms, whilst Marton and Saljo (1976a) focus
on the learning behaviour of students when studying text-based material; describing deep-level and surface-
level processing as "corresponding to the different aspects of the learning material on which the learner
focuses. In the case of surface-level processing the student directs attention towards learning the text itself,
[and is] more or less forced to keep to a rote learning strategy. In the case of deep-level processing, the
student is directed towards the intentional content of the learning material, towards comprehending what the
author wants to say about, for instance, a certain scientific problem or principle".
It is important to note that the deep- and surface- levels of processing concept does not attempt to classify
individuals as "deep" or "surface learners", it does not imply that a particular individual will learn in a "deep" or
"surface" way. McKeachie (1995) explains further by saying "as in most things, there are potential undesirable
side effects from the use of learning style concepts. Probably the most serious is that styles are often taken to
be fixed, inherited characteristics that limit students' ability to learn in ways that do not fit their styles. Thus,
some teachers draw the implication that they must match their teaching to the student's particular style, and
some students who have been labeled as having a particular style feel that they can only learn from a certain
kind of teaching. Some teachers become devotees of one or another learning style system. However, the
"styles" or "types" identified by learning style inventories are not little boxes, neatly separated from one
another; rather, they represent dimensions along which learners may differ".
It is anticipated that students taking part in this project will deepen their learning in two ways:
1) by collaboratively correcting first-draft transcript text when listening to short segments of audio and
the corresponding text produced by speech recognition software;
2) by viewing, searching, and cross-referencing related concepts across multiple lectures after all
transcripts have been student-corrected (and approved by the lecturer).
4. EVALUATING THE IMPACT ON STUDENT LEARNING

We have collected online questionnaire feedback after 1%, 50% and 100% of a students' transcription target,
and have a final short questionnaire section in their examination, which will give their final evaluation of the
process, after revision and sitting the examination.
154
.
Because the deadline to submit transcript sections for approval is May 7th, and the examination for the
module is on 20th, May 2010 we will fully evaluate the impact of the project on student learning later in 2010,
after the submission deadline for this paper. Initial qualitative feedback from students shows that at least
some of them perceive that the approach has been helpful to their learning.
We expect students' use of Search and Viewer functions to increase dramatically in May, as the examination
approaches. This will provide part of the basis for analysis, which we will present at the August Conference.
We intend to examine correlations between use of Search and Viewer functions with examination
performance, as well as comparing the performance of this cohort with last year's cohort for the same subject.
As well as quantitatively-based data, we have collected students' qualitative feedback comments, both
through questionnaire formats and through personal communications.
5. OVERVIEW OF SOFTWARE DEVELOPMENT AND FUNCTIONALITY

As mentioned previously, an earlier system was designed for a single person (nominally a lecturer) to upload
mp3 recordings to a server, and edit a first-draft transcript produced by Dragon Naturally Speaking.
Modifications were performed to extend this from a single-user system into a multi-user system, and (in order
to tie-in with module assessment) extensions were made to keep track of individual user's edits, storing
information about the length of time spent editing, etc. Screenshots of the system functionality described
below (with captions) can be found at http://www.cems.uwe.ac.uk/~p2phelps/heaics2010/.
The complete system incorporates MacOSX and winXP software: There are two separate MacOSX-based
Apache & PHP web interfaces:
 One "administrative interface" to control processing of lecture audio recordings uploaded to a digital-
dropbox;
 Another "student interface" to facilitate collaborative multi-user editing of first-draft transcripts.
The administrative web-interface monitors the status of each audio recording and transcript passing through
the system:
 Audio segmentation and time-coding is based on mp3splt (an open source cassette/vinyl track-split
detection algorithm);
 Speech recognition is performed by Dragon Naturally Speaking (running on a winXP machine);
 FTP is used to share data between the macOSX and winXP machines;
 autoIT is used to control the behaviour of Dragon Naturally Speaking during transcription;
 Various scripts (bash/winXP BAT) automate the transfer of transcripts (and mp3 audio) between the
two platforms.
The student web-interface allows multiple students to edit first-draft transcripts, and for staff to monitor and
moderate student submissions:
 Students log-in to the system by using their student number (then stored in a browser cookie);
 Students can immediately browse the lecture recordings, although accuracy of first-draft transcripts
will initially be low;
 Students are required to make edits to earn marks;
 Transcripts are composited from individual time-coded segments, which can be selected for editing;
 Segments can have one of three states:
o Untouched - text as produced by speech recognition first-draft;
o Allocated - a student is editing (segments are allocated on a first-come-first served
basis to ensure traceability when awarding marks);
155
o Approved - a student edit has been approved by a moderator.
 A "crawler" PHP script scans the pool of transcripts for approved segments periodically;
 The crawler builds an XML file (compatible with the TRS format used by Transcriber);
 Text in approved segments is merged into the TRS XML transcript;
 Rejected segments are ignored (an empty segment is inserted into the TRS XML file).
 Students and staff can monitor the status of entire lectures by looking at timelines for lectures, or by
using the "transcript viewer";
 The timelines provide a quick way to see how many approved segments a lecture has;
 The transcript viewer shows only approved text.
 The transcript viewer synchronises lecture audio and transcript text;
 PHP produces a document with timecoded tags embedded in the transcript text;
 A flash-based media player calls javascript to scroll timecoded text elements into view for the
current "playhead" position;
 The user can search for text using their browser's search function, and push a button to seek the
playhead to the matching timecode;
 In this way, timecodes can be converted to a standard URL such that specific points in an audio
file are easily shared.
6. SOFTWARE IMPROVEMENTS
A number of limitations have become apparent through use of the system, and we shall discuss possible
future improvements to the system at the August Conference.
7. CONCLUSION
Every project is a learning experience, and this has been no exception. Modifications and extensions to the
system have become apparent through use, and the issues of extending such a system into use by multiple
users have been challenging at times, with the project involving more work than originally anticipated.
We are heartened by the level of student engagement, and by their collaborative generation of a usable
resource for learning and revision. The transcripts have also caused the lecturer to reflect deeply on lecturing
style and personal idiosyncrasies of speech as well as providing a much more detailed basis than normal
practice for reflection on how the next iteration of the module might be improved in delivery and content.
8. REFERENCES
Craik, F. & Lockhart, R. (1972). Levels of processing: A framework for memory research. Journal of Verbal
Learning & Verbal Behavior, 11, 671-684.
Kearsley, G., (2010) Levels of Processing (F. Craik & R. Lockhart), The Theory Into Practice Database,
http://tip.psychology.org/craik.html (20th May 2010)
Lynch, M. & Phelps, P. (2009). Automated Transcription of Spoken Word Lecture Materials, Proceedings of
the Higher Education Academy for Information and Computer Sciences 10th Annual Conference, University of
Kent in Canterbury.
Marton, F., and Säljö, R. (1976a) On qualitative differences in learning: I. Outcome and process. British
Journal of Educational Psychology, 46, 4-11.
Marton, F., and Säljö, R.(1976b) On qualitative differences in learning: II. Outcome as a function of the
learner's conception of the task. British Journal of Educational Psychology, 46, 115-127.
McKeachie W. J. (1995) "Learning Styles Can Become Learning Strategies (University of Michigan)",
w2q1Vol.4 No.6. (Available from:http://www.ntlf.com/html/pi/9511/article1.htm)
156
TECHNOLOGY ENHANCED LEARNING: SYSTEMS TO INCREASE
ENGAGEMENT & PARTICIPATION DURING LECTURES
Andy Stephenson Liz Burd

andy.stephenson@durham.ac.uk liz.burd@durham.ac.uk
ABSTRACT
Context/Background - This study investigated the role of technology in engaging students both during
lectures and whilst reviewing lecture content.
Aims - The first aim of this study was to investigate whether taking lecture notes electronically on mobile
devices and allowing students to share them could lead to a higher level of engagement and participation
during and after lectures. In addition, this study evaluated a range of methods for increasing engagement and
participation during a lecture. Finally, the use of social network-style functionality was investigated to
determine whether it could increase engagement and participation when students review lecture content.
Methods - This project was split into two stages. Stage One focused on capturing students’ lecture notes
electronically and investigating methods of boosting engagement during lectures. Stage Two focused on
visualising notes captured using the system for use in reviewing content after the lecture.
Results - Stage One of this study showed a 23% rise in engagement amongst the cohort across three trials
as well as several different indicators showing a rise in participation during lectures. Results from Stage Two
indicated mixed usage of the lecture review system.
Conclusions - Qualitative results from this study indicated integrating different technological components can
increase engagement during lectures. Feedback from Stage Two showed students preferring a more social
network-style interface that illustrates the idea of a changing pedagogy from more traditional methods of
reviewing content, although this functionality was not used extensively and could be investigated further,
perhaps over an examination period.
Keywords
Technology Enhanced Learning, Engagement, Participation, Pedagogy, Social Networks, Electronic Note-
Taking.
1. INTRODUCTION
The aim of this study was to look at how engagement and participation levels could be improved within a class
of university students through the use of electronic, mobile technology.
During this study, engagement and participation levels were recorded as a result of implementing an
electronic note-taking system in lectures and a visualisation system to assist with lecture review. Engagement
was defined as how involved and connected students felt with a lecture, its contents and the process of
lecture review. Engagement was monitored qualitatively through questionnaires to gauge participants’
reactions. Participation, was defined as how involved the students were in lectures and the process of lecture
review. More specifically, it was measured quantitively by recording the number of lecture notes taken,
number of lecture slides reviewed as well as comments and ratings made and social networking tags added.
One aim of this study was to investigate engagement and participation in lectures, rather than students' ability
to understand and learn material. The result of this was that an increase in engagement or participation lead
to a more accessible and open learning environment. Widening the accessibility and improving the interaction
with education and learning materials could allow for an increase in students’ desire to learn, potentially
resulting in increases in learning ability through improved test scores or a deeper understanding of content
areas.
With reference to increasing engagement and participation in education, Biggs (2002) argues that this can
only really be achieved at Higher Education level. One study conducted by Lloyd (1968) suggested that
receptivity during a lecture featured two peaks: one five minutes into the lecture and another five minutes
before the end. Between the two peaks was a steady downward slope of both student receptivity and
performance. This is the type of scenario that this study aimed to improve: by adding technology as a catalyst,
student engagement and participation in a lecture would hopefully improve.
157
One common theme that appeared throughout studies researched was that by introducing different elements
of technology into the educational environment, changes began to emerge in students’ learning styles.
Perhaps one of the main reasons for the increase in participation and engagement whilst using technology
was that students were better able to identify the pedagogies they required specifically to understand the
content.
Overall, due to the rapidly advancing nature of this field and the different experiments being researched, there
are unfortunately very few secondary studies pointing to any form of definitive best practice with regards to
increasing student engagement and participation levels. The impact for this study is that the direction of any
research can only be defined by the most common trends within the industry, regardless of whether these turn
out to best practice.
The current state of university education is one where a lecturer will dictate a very linear-formed presentation
to a group of students (Laurillard, 2002). This very strict and traditional model that has been used for centuries
aims to allow students to almost learn through osmosis, whereby they are surrounded by new knowledge and
they must absorb as much of it as they can. Laurillard criticised this model, commenting:
“They [the students] must do the work to render the implicit structure to themselves, must reflect on
the relationship between what the lecturer is saying and what they previously understood, and decide
if it is different and how the difference is to be resolved.”
This idea that students must partake in a three-stage learning process illustrated for each idea a lecturer
mentions means that some concepts are likely to be missed. Given this process for disseminating information,
it is perhaps not surprising that within a typical lecture scenario, students’ note-taking efficiency is often as low
as 20-40% (Makany et al., 2009).
Integrating this idea into the context of university teaching could potentially set out a useful framework for
students to become more involved and to provoke them into questioning ideas and encouraging debate. This
concept could also potentially lead to a deeper level of learning.
The rise and ubiquitous nature of social networking websites in the past few years could provide a suitable
medium for this democratisation as it provides a framework that students are familiar with from other elements
of their life. Social networking principles - such as commenting, rating and tagging - represent a changing
pedagogy, moving away from 'traditional' learning methods towards a more collaborative approach.
According to Wheeler et al. (2008), there is evidence to suggest that user-generated software and content can
progress knowledge to a much deeper level. A growing trend towards student-led collaborative learning where
teachers are used more as points of reference rather than a single source of knowledge is backed up by the
more democratic style of education as defined by Hartnell-Young (2009) and Laurillard (2002).
Wheeler et al. (2008) showed that by allowing students to utilise wikis to generate, edit and synthesise
knowledge in an openly accessible space, students could share ideas and collectively gain knowledge that
they may previously have overlooked. In addition, students were able to receive near-immediate feedback
from their peers about their ideas.
2. SOLUTION
2.1 Stage One Design
Stage One was split into three trials: the first introduced the mobile note-taking platform, the second
introduced a live feed of students’ notes on the projector screen and the final trial added in a PRS (Personal
Response System).
In the literature, Laurillard (2002) commented on a three-stage learning process for each idea that a lecturer
dictates, meaning that some ideas or concepts are likely to be missed by students. Wheeler et al. (2008)
further promoted this philosophy, suggesting that the use of Wikis in education allowed students to share
ideas and collectively gain knowledge they may previously have overlooked. This was the concept upon which
the mobile note-taking platform was introduced: to reduce the risk of students missing out on important
concepts from lectures.
The second part of Stage One was a live feed of all the public notes taken during the lecture that were
automatically posted to the projector screen at the front of the lecture hall. This idea came from the need to
engage students more in lectures by changing traditional methods of lecturers teaching and students learning
158
(Biggs, 2002) as well as trying to extend the levels of student receptivity by constantly making them think
about the lecture's content and what their peers were writing, as was written by Lloyd (1968).
2.2 Stage Two Design
During Stage Two, the emphasis of the study changed to visualising the notes taken during Stage One, whilst
at the same time maintaining the core note-taking functionality of Stage One (to ensure there was enough
content to review). During Stage Two, the live note feed and the PRS were omitted to allow focus on post-
lecture review, rather than in-lecture engagement. In addition, users were permitted to use their own devices
to capture notes or given the option of an HTC mobile phone if they did not have another compatible device.
The three iterations in Stage Two were trialled with two cohorts. The first iteration displayed all publicly taken
lecture notes alongside a copy of the slides. The next trial integrated an audio recording of the lecture and the
final trial added social networking functionality, in the form of the ability for users to comment and rate notes
as well as tag slides with common themes. This functionality enabled the system to illustrate key themes that
emerged throughout each lecture in a tag cloud, aiming to engage students in a deeper level of understanding
in the topic compared to simply reviewing lecture notes.
The idea of creating a more social networking environment was developed from participants’ feedback before
the system was developed: there were strong opinions that an educational web-based system should adopt a
more familiar interface to one that students perhaps use on a daily basis in their social lives.
3. RESULTS
3.1 Log Analysis
Throughout the nine trials conducted during this study, there were 54 registered users, of which 28 took notes
during the system (52%). The trials were conducted across two subjects: Project Management (Stage One)
and Advanced Software Engineering (Stage Two). Between the two stages, there was some cross-over of
participants. During the nine trials, 177 notes were taken across 260 slides, although 54 notes (31%) were
deemed to be irrelevant to the subject matter. During Stage Two, only one note was commented on, although
this was irrelevant to the subject and only one participant rated a note, which had originally been written by
themselves.
3.2 Student Engagement Levels
At the end of Stage One, all participants in the trials were asked whether they felt more engaged after using
Stage One’s functionality, compared to a regular lecture or a Project Management lecture. The question
posed to participants included a five-point scale and asked users to select the number that they thought best
represented their views. On the scale, one was defined as not at all engaged and five was defined as
completely engaged. Figure 1 shows there was a 23% increase in engagement after using Stage One’s
functionality, compared with a regular lecture and a 16% increase in engagement between Stage One and a
Project Management lecture.
Figure 1 - Average student engagement increases as a result of introducing Stage One’s functionality
During Stage Two, participants offered mixed opinions on the lecture review system’s ability to engage them.
Overall, only 40% of participants (2/5) thought that using the lecture review system engaged them more in the
process of reviewing lectures, although a further 20% (1/5) thought that they were somewhat more engaged.
However, 80% of participants (4/5) enjoyed using the system to review their notes, with a further 20% (1/5)
agreeing that they somewhat enjoyed it.
159
3.3 Student Participation Levels
Table 1 compares the total number of edited notes taken throughout the study as well as adding in
comparisons to benchmark each trial’s results against each other. Starting with the first row, the trial number
and total number of edited notes are displayed. The ‘Notes/Participant’ calculates the total number of notes
taken, divided by the number of note-taking participants in the trial. The final column represents the slide
number (and total number of slides in brackets) that the last note was recorded on; the percentage defines the
proportion of the lecture after the last note where no notes were taken.
TABLE 1 - LECTURE NOTES CAPTURED ACROSS ALL TRIALS
Trial Number Total Notes Notes/Participant Slide Last Note Taken On (Total Slides)
Taken
1 17 4.25 16 (38) - 58% Slides Left
2 18 4.5 17 (20) - 15% Slides Left
3 5 2.5 18 (27) - 33% Slides Left
4 29 9.67 28 (35) - 20% Slides Left
5 14 2.33 21 (31) - 32% Slides Left
6 5 5 7 (8) - 13% Slides Left
7 28 5.6 17 (37) - 54% Slides Left
8 1 1 7 (41) - 83% Slides Left
9 6 2 28 (41) - 32% Slides Left
3.4 Social Networking Functionality In Stage Two

During Stage Two, only one comment was made in response to all the lecture notes in the system and this
comment was irrelevant to the subject matter. Further, two people rated notes during the system, although
one person rated their own note and the other person rated notes that were irrelevant to the subject matter.
During feedback, 40% of participants (2/5) suggested that the social networking-style interface helped engage
them more in the process of lecture review, with a further 40% (2/5) indicating that they felt somewhat more
engaged as well. 50% of participants (3/6) agreed that including a tag cloud allowed for a quicker
dissemination of a lecture’s content. Overall, 33% of participants (2/6) felt more engaged with the lecture
review as a result of having the tag cloud and a further 33% (2/6) felt somewhat engaged.
4. EVALUATION
The main limitation of this study was the accuracy of data and a confounding factor resides with the relatively
low numbers of trials, participants, notes taken, and questionnaire responses that were gained.
4.1 Student Engagement & Participation
Figure 1 showed increases in student engagement across Stage One of this study, although during Stage
Two only 40% (2/5) of participants felt more engaged after using the lecture review system in Stage Two.
Despite this outcome, 80% (4/5) of participants claimed to enjoy using the lecture review system and so even
though they may not have felt more engaged in the content, they were perhaps more inclined to review notes
taken electronically. This result was shared with Hartnell-Young (2009), who concluded that the use of
technology in education can help increase motivations to learn. Further, only 70% (7/10) of the lecture review
system’s users said that they currently review lecture notes by hand, showing a slight increase in participation
levels against traditional methods. This statistic highlights the benefits of making more educational websites
interactive, perhaps through the use of social networking functionality, in order to make them more accessible
to users.
Table One illustrates the different quantification methods. The biggest distinction is shown by the final column
that displays the proportion of lecture slides remaining after the last note was recorded per trial. The average
proportion of slides left was 42.5% of the lecture, which suggests that participants started by taking notes but
then stopped just after halfway through the slides. Whilst this may initially suggest that participants’
160
engagement levels fell during the lecture, there may also be other reasons for this fall in notes taken. Firstly,
there was a limiting factor with the WiFi connection on the HTC phones that sometimes struggled to connect
to the network. Secondly, the reduction may also be attributable to the time spent per slide during the first and
second halves of the lecture by the lecturer, since they may have inadvertently spent a disproportionate
amount on earlier slides and had to rush through the latter half of content. Lloyd (1968) noted that student
receptivity peaked five minutes into a lecture and then steadily decreased as the lecture progressed;
potentially what electronic note taking in this study has achieved is to extend the period of engagement
throughout over half the lecture, on average.
4.2 Taking Lecture Notes on Mobile Devices
Participants gave differing opinions of taking lecture notes via mobile devices: in some instances, the text-
input on mobile devices was restrictive and a better solution would have been a free text input solution. This
would have meant participants could write directly on the screen of their device using a stylus, allowing them
to capture non-standard characters as well as match their traditional note taking techniques more accurately
with the ability to underline, highlight and emphasise particular points on the slide.
As Stage Two progressed, more users switched to using their own mobile devices or laptops for taking notes
owing to the speed of input and the familiarity of their own device. The work of Wheeler et al. (2008) reaffirms
the result that familiarisation of new technology is critical to users’ adoption of it.
In addition to taking notes on mobile devices, participants mostly agreed that displaying a live feed of public
notes from the system was productive in both engaging them with the subject and also provoking debate
amongst the class. In one instance, a participant posted a question about something someone else had
posted. The lecturer seized the opportunity and a debate ensued throughout the class.
4.3 Use of Social Networking Concepts in Stage Two
The implementation of social-networking functionality was derived from user feedback, although when
implemented it was not used extensively. However, regarding the sharing of content, participants made a
case that they would be interested in a system that would allow them to control access to their own content,
sharing content openly with smaller, self-forming groups of their peers.
Feedback from participants suggested that they would have liked to review their lectures more and saw the
social networking functionality as a good way of achieving this, but they were too busy with other work to
participate fully.
5. CONCLUSION
Firstly, there were mixed results from efforts to increase student engagement and participation levels during
lectures: in Stage One, the inter-lecture trials showed a 23% rise in engagement levels, which was contrasted
by only 40% of participants feeling more engaged with a lecture’s content after using the lecture review
system in Stage Two. Several reasons led to low levels of participation during Stage Two, such as
participants’ time constraints. Whilst Stage One showed an increase in engagement, perhaps due to the
changing pedagogy in the way that students took lecture notes, further research could look into a link between
engagement in lectures and a deeper level of learning in order to show how much of a long-term effect this
technology could have on enabling deeper learning.
Secondly, results and feedback from this study suggest that taking lecture notes electronically should not be
restricted to specific mobile devices. Participants quickly realised that they preferred to use their own mobile
device or laptop/netbook to capture notes during lectures, rather than the HTC device given to them.
Finally, perhaps the most pertinent point illustrated in this study was that technology is best suited to use in an
assistive role to allow greater accessibility and engagement within education. Participation (and learning)
continue to be the responsibility of the individual in using the tools available to fit their own, specific,
pedagogical needs.
6. REFERENCES
 Biggs J., (2002), 'Teaching for Quality Learning at University’, Open University Press, Buckingham UK.
 Hartnell-Young, E., (2009), ‘The importance of teaching roles when introducing Personal Digital Assistants in a Year 6 classroom’,
Technology, Pedagogy and Education, vol. 18 issue 1, pp. 3-17.
 Lloyd, D., (1968), ‘A concept of improvement of learning response in the taught lesson.’, Visual Education, pp. 23-25.
 Makany, T., Kemp J., & Dror, I., (2009), ‘Optimising the use of note-taking as an external cognitive aid for increasing learning’, British
Journal of Educational Technology, vol. 40, issue 4, pp. 595-596.
 Laurillard, D., (2003), ‘Rethinking University Teaching: a framework for the effective use of educational technology’, London: Routledge.
 Wheeler, S., Yeomans, P., & Wheeler, D., (2008), ‘The good, the bad and the wiki: Evaluating student-generated content for collaborative
learning’. British Journal of Educational Technology, vol. 39, issue 6, pp.987-995.
161
INVESTIGATING ‘VOICE EMAIL’ TECHNOLOGY EFFICACY IN
INFORMATION MANAGEMENT ASSESSMENT
George Macgregor Alex Spiers Chris Taylor

Information Management Learning Development Unit Information Management
Liverpool Business School Liverpool John Moores University Liverpool Business School
Liverpool John Moores University a.spiers@ljmu.ac.uk Liverpool John Moores University
g.r.macgregor@ljmu.ac.uk c.j.taylor@ljmu.ac.uk
ABSTRACT
This paper introduces the use of ‘voice emails’ as an approach to improving formative feedback and describes
how such technology can be embedded within curricula. A quasi-experimental study was conducted with a
student sample (n = 42) comparing audio and written approaches to feedback delivery. Student performance
at formative and summative assessment points was analysed and the influence of the feedback format used
was studied. The ability of voice emails to better meet recognised theoretical models of ‘quality’ formative
feedback was also investigated using a specially designed web survey research instrument. Results reveal
that in most instances voice email can better meet the conditions of formative feedback thus enhancing the
student learning experience. Results from the study also suggest that voice email feedback, although offering
many positive applications to lecturers, may not significantly improve the learning of students.
Keywords
Audio feedback, voice emails, formative assessment, audio technology
1. INTRODUCTION
Formative feedback can be used by students to amend their learning behaviour thus promoting ‘deep
learning’ approaches to summative assessments [5]. Despite the recognised importance of formative
assessment [1, 5], few such opportunities are made available to students within Higher Education (HE). This
issue is often attributed to structural constraints, such as the limited time lecturers have within semester-
based systems to produce feedback, increased student-to-staff ratios and the demands of scholarly activity.
Gibbs and Simpson [1] propose a series of conditions to be met if formative feedback is to be effective and
used meaningfully by students. Included within these conditions are that it be sufficiently detailed (condition
4), understandable (condition 8) and received by the students “while it still matters” (condition 6). Structural
constraints often mean that when formative feedback is delivered it is insufficiently detailed, providing the
student with limited feedback on how their performance relates to course expectations and how it can be
improved. Furthermore, feedback will often be difficult to interpret, employing language and jargon that
students have difficulty understanding (e.g. “This is insufficiently critical”, etc.) [5]. This is further complicated
by assessment tasks themselves, the nature of which students rarely understand, thus making feedback
interpretation even more difficult [1]. Formative feedback may be sufficiently detailed and understandable but
will be of limited use if the student has little time to act on it prior to summative assessments.
The use of audio technologies to deliver feedback of all types has recently attracted attention from the
learning technology community and research has demonstrated student satisfaction with such technologies
[6]. In this paper we introduce the use of ‘voice emails’ as a potential solution to some of the above noted
difficulties and report on the findings of a study designed to evaluate the efficacy of voice emails in delivering
formative feedback to degree course students studying a business information management module. We
also evaluate the ability of voice emails to better meet recognised theoretical models of quality formative
feedback and investigate the potential of audio feedback to enhance student learning.
2. AUDIO FEEDBACK
Recent advances in audio technologies have enabled the creation of audio-based learning materials (e.g.
podcasts), often to promote m-learning. Research evaluating the effectiveness of such technologies to deliver
feedback unfortunately remains limited. Students’ perceptions and use of audio feedback have been
investigated by a number of researchers [e.g. 2, 4, 6], all of whom found students to respond positively to
audio feedback primarily owing to its personal nature and because it was more detailed and easier to interpret
than written feedback. Capturing greater feedback detail and improving feedback delivery times has also
been identified by several researchers as a potential advantage of audio feedback and a solution to the poor
formative assessment opportunities at HE; however, results remain variable [2, 3] and some studies lack
objective measurements [6].
Perhaps most interesting is the potential of audio feedback to affect improvements in learning. If audio
feedback provides opportunities for better meeting the conditions of quality formative feedback then it can be
hypothesised that students receiving it may experience improved learning prospects. Evidence gathered by a
number of studies has indicated that audio feedback may be capable of enhancing student learning more than
other feedback methods [e.g. 2, 3, 7]. However, evidence remains unsatisfactory and further research is
required to increase our understanding of the link between formative feedback delivered as audio and
enhanced student learning.
3. METHODOLOGY
Wimba Voice (http://www.wimba.com/products/wimba_voice/) was installed within Blackboard to deliver audio
feedback. Wimba is a web-based plug-in suitable for a variety of Virtual Learning Environments (VLE) and
provides a variety of audio tools (e.g. podcaster, voice enabled discussion fora, etc.). It also enables the
creation of voice emails. These are voice messages which can be recorded and delivered to students using
an email / tape recorder interface within a Java enabled browser. Students receive an email with a hyperlink
directing them to an audio file stored on a local server, thus obviating MP3 file size issues which can often
cause technical issues in feedback delivery [4]. Wimba also allows students to reply to tutors with their own
voice email. Wimba voice emails were used to deliver formative audio feedback to students in our study.
Voice
Male Female Sub-total Written (n=21) Male Female Sub-total
(n=21)
Age n % n % n % Age n % n % n %
18-24 3 14 14 67 17 81 18-24 3 14 18 86 21 100
25-29 0 0 3 14 3 14 25-29 0 0 0 0 0 0
30-34 0 0 1 5 1 5 30-34 0 0 0 0 0 0
Total 3 14 18 86 21 100 Total 3 14 18 86 21 100

Figure 1: Demographic details of study participants and composition of streamed groups.
Study participants were drawn from a first year cohort studying a web technologies and business information
module as part of a BA (Hons) Business and Public Relations degree course. Forty two students agreed to
participate in the research (Figure 1). All students received a Wimba orientation session to control for varying
levels of ICT efficacy and an instructional video was also made available via Blackboard.
The summative assessment for the module required students to submit an XHTML report. A formative
assessment point was created mid-way through the module for which students submitted an XHTML report
plan, thus enabling tutors to deliver formative feedback on student learning and understanding. Students
were then randomly streamed into a written group (control) (n = 21) and a voice (email) group (treatment) (n =
21) (Figure 1). Marking criteria were agreed among tutors and, where possible, feedback attempted to follow
recognised models of quality feedback [1, 5]. For the purposes of our research, students’ formative
assessment performance was recorded for future analysis. Note that this mark was not disclosed to student
participants. Formative feedback was then delivered to students (within a week), with written feedback
students receiving their feedback as an MS Word file and the voice group receiving a voice email.
A web-based survey instrument was administered during a timetabled IT lab one week after feedback was
delivered. The survey was designed to gather a wide range of data; however, of relevance to this paper was
section two of the survey in which students were required to indicate their responses to a series of statements
using a five point Likert scale, ranging from strongly agree (5) to strongly disagree (1). This section was
specially designed to determine how well formative feedback achieved its purposes and to detect effects on
student learning. The statements used in section two map to recognised formative feedback models [1, 5].
4. RESULTS AND DISCUSSION

Figure 2 sets out the results from section two of the web-based survey instrument. Responses from both
groups were generally positive, indicating a high level of student satisfaction with their formative feedback
irrespective of whether this was delivered as a voice email or in written form. Noteworthy median differences
can be observed for statements J (Mvoice = 2.476, MDNvoice = 2; Mwritten = 2.905, MDNwritten = 3), M (Mvoice =
3.905, MDNvoice = 4; Mwritten = 3.238, MDNwritten = 3) and O (Mvoice = 4.334, MDNvoice = 5; Mwritten = 3.952,
MDNwritten = 4). A Mann-Whitney U test was conducted to detect significant differences between group
163
responses (Figure 2). Statistically significant differences (p < 0.05) between group responses were found for
statements D, H, K and M, indicating students in the voice email group found their formative feedback to
better meet conditions of quality feedback in terms of being understandable, clarifying assessment
expectations, and inspiring motivational beliefs [1, 5]. Responses for K were significant at p < 0.01 (U = 128,
Z = -2.570, p = 0.01). It is also worth noting the generally positive mean scores recorded for many of the
voice email question statements.
Voice email Written a
Survey statements – section two U Z p-value
M Mdn M Mdn
a. I was satisfied with the feedback provided 4.286 4 4.095 4 181.5 -1.325 0.185
b. I found the feedback to be clear and understandable 4.286 4 4.095 4 175 -1.357 0.175
c. The feedback I received helped me 'troubleshoot' or
self-correct my performance on the module and the 3.952 4 3.762 4 193.5 -0.772 0.440
final assessment
d. The feedback clarified or made explicit what is
required of me in order to improve my academic 4.238 4 3.762 4 148 -1.981 0.048*
performance on the module and the final assessment
e. The feedback helped me reflect on my learning 3.810 4 3.714 4 211.5 -0.244 0.807
f. The feedback helped me understand where to focus
my efforts so that I can better improve my university 3.905 4 3.905 4 220.5 0.000 1.000
coursework
g. I considered the feedback to be sufficiently personal
4.238 4 3.905 4 163 -1.612 0.107
and relevant to me
h. I found the feedback to be easy to comprehend 4.190 4 3.809 4 157 -1.959 0.050*
i. I felt the feedback was sufficiently detailed 4 4 3.619 4 155 -1.814 0.070
j. I found the feedback to be too brief 2.476 2 2.905 3 170.5 -1.329 0.184
k. The feedback was cryptic or difficult to interpret 1.810 2 2.476 2 128 -2.570 0.010*
l. The feedback helped to increase my interest in the
2.952 3 2.905 3 206.5 -0.393 0.694
module I am studying
m. I felt motivated after reading/listening to my
3.905 4 3.238 3 132.5 -2.378 0.017*
feedback
n. The feedback was delivered in a timely fashion 4.095 4 4.286 4 189.5 -0.882 0.378
o. I intend to use the tutor feedback later in the module 4.334 5 3.952 4 157 -1.754 0.079
p. I was afforded sufficient opportunity to seek follow-
3.524 4 3.667 4 196.5 -0.679 0.497
up tutor feedback (e.g. Questions)
q. It is important that my feedback is delivered
3.667 4 3.619 4 208 -0.329 0.742
electronically
u. I prefer to receive my feedback electronically 3.524 4 3.905 4 182.5 -1.011 0.312
a
Two-tailed Mann-Whitney U test (adjusted for ties). Sig. at p < 0.05.
* p < 0.05.
Figure 2: Measures of central tendency and M-W U tests between groups for section two responses.
Students’ formative and summative assessment performance are presented in Figures 3 and 4. The
performance in the formative assessment was expected and was similar in both the voice email and written
groups, with slightly more dispersion around the mean in the voice group. This was confirmed by an unpaired
two tailed t-test at p = < 0.05 (t(40) = 1.36, p = 0.182).
Written
Formative Summative
Voice Voice Written
performance (n = performance
measure (n = 21) 21) measure (n = 21) (n = 21)
M 45.38 40.19 M 40.86 40.91
SD 14.89 9.12 SD 17.35 19.74
R 44 34 R 51 55
Figure 3: Student performance in formative and summative assessment (mean, standard deviation, range).
Unfortunately, performance for both groups in the summative assessment was poor. There was little
observed improvement from the formative assessment, with a mean percentage learning gain of -4.52 and
0.72 noted for the voice and written groups respectively. This result was unanticipated and is contrary to the
findings of a similar study [3]. It is nevertheless encouraging that marks of > 40% can be observed in the
summative performances of both groups, with numerous students in both groups reaching > 55% and some
164
students exceeding 70% (Figure 4). An unpaired two tailed t-test at p = < 0.05 revealed no significant
differences between group performances for the summative assessment (t(40) = 0.008, p = 0.993).
Figure 4: Student performance in formative and summative assessment (voice and written).
Students were expected to publish their XHTML report on the University server as part of the assessment.
Failure to do this correctly meant that tutors were unable to access the report via HTTP, for which marks were
assigned. A large proportion of students (52%, n = 22) failed this aspect of the assessment and as a direct
consequence achieved marks < 40%. These students were spread equally across both groups (n = 11). If
such outlying data is removed from the dataset we can observe improvements in the summative assessment
marks for both groups (Figures 5 and 6), with improved mean percentage learning gains (Mvoice = 10.52;
Mwritten = 19.81) and the graph profile of the written group mirroring formative performance. Although the data
indicates a superior summative performance and higher learning gains for the written group, the difference
between groups was not statistically significant (t(18) = -0.841, p = 0.413). This finding appears to
corroborate the findings of previous research [3].
Summative performance measure Voice (n = 10) Written (n = 10)
M 55.9 60
SD 11.51 8.79
R 31 31
Figure 5: Student performance in summative assessment (outliers removed).
Figure 6: Student performance in formative and summative assessment (outliers removed).

The time demands of tutors generating voice email feedback were considerably smaller and indicate that
using audio was 40% quicker to create and deliver (Figure 7). Less variability in the amount of tutor time
spent marking individual submissions was also observable from the reduced data dispersion (Mvoice = 0.17,
Rvoice = 1.06; Mwritten = 1.11, Rwritten = 4.40). To enable comparisons between feedback delivery times of
different tutors, a random sample of submissions (n =12) was used prior to data collection to detect for any
variance in tutors’ delivery of both audio and written feedback. The time taken for both tutors to complete
feedback for this test sample was similar and did not differ significantly for either voice emails (t(10) = 1.52, p
= 0.16) or written feedback (t(10) = -0.61, p = 0.56). The results presented in Figure 7 therefore appear to be
an accurate reflection of the time efficiencies possible when using audio feedback approaches. It is worth
commenting that the use of voice emails probably contributed to improvements in the time efficiencies by
165
obviating any need for specialist audio software or recording equipment and by removing file transfer issues
and other technical issues (e.g. difficulties attaching large MP3 files to emails) which can often limit the time
reductions possible [6].
M SD R
Feedback group
Dec. (5dp) Min/Sec Dec. (5dp) Min/Sec Dec. (5dp) Min/Sec
Voice 0.06407 3.50 0.00475 0.17 0.01861 1.06
Written 0.10309 6.11 0.01999 1.11 0.07778 4.40
Figure 7: Time requirements for delivering voice email and written feedback.
5. CONCLUSION
Our quasi-experimental study compared the efficacy of audio and written approaches to formative feedback
delivery and introduced the use of voice emails as a means of delivering formative feedback to students
undertaking an information management module. The results indicate that voice emails better meet
recognised theoretical models of quality formative feedback thus enhancing the student learning experience.
Voice emails were found to clarify assessment expectations, to be more understandable, and to inspire
motivational beliefs. It was disappointing that the increased detail and personal nature of voice emails was
not found to be statistically significant; however differences in mean responses were observed indicating
enhancements in this respect. Results from the current study therefore suggest that although audio feedback
may enhance the learning experience, it may not significantly improve student learning. It is nevertheless
encouraging to note that voice emails appear to promote greater use of formative assessment by reducing the
time commitments of formative feedback delivery. It is our intention to merge data from this study with that of
a previous study [3]. Extensive qualitative data was also gathered from student participants using semi-
structured interviews, the coding of which provided useful additional data on the learning effects of voice email
in formative assessment. We expect the results of both to be published in the academic literature in due
course.
6. ACKNOWLEDGEMENTS
This research was conducted as part of a wider project (ExAEF Project) funded by the HEA Subject Centre
for Information and Computer Science (2009). Their support is gratefully acknowledged by the authors.
7. REFERENCES
1. Gibbs, G., and Simpson, C., (2004), Conditions under which assessment supports learning, Learning
and Teaching in Higher Education, 1 (1), pp 3-31.
2. Ice, P., Reagan, C., Perry, P. and, Wells, J., (2007), Using synchronous audio feedback to enhance
teaching presence and students’ sense of community, Journal of Asynchronous Learning Networks,
11 (2), pp 3-25.
3. Spiers, A., and Macgregor, G., (2009), Using audio email feedback in formative assessment, Audio
Feedback: A Word In Your Ear Conference, 18 December 2009,
http://www.staff.ljmu.ac.uk/bsngmacg/exaef/docs/LJMU-presentatio1n-new.pdf (accessed
10/05/2010)
4. Merry, S., and Orsmond, P., (2008), Students’ attitudes to and usage of academic feedback provided
via audio files, Bioscience Education, 11 (3),
http://www.bioscience.heacademy.ac.uk/journal/vol11/beej-11-3.pdf (accessed 10/05/2010)
5. Nicol, D.J., and Macfarlane-Dick, D., (2006), Formative assessment and self-regulated learning: a
model and seven principles of good feedback practice, Studies in Higher Education, 31 (2), pp 199-
218.
6. Rotheram, B., (2009), Sounds Good: Quicker, better assessment using audio feedback (Final Report
– JISC Project), Joint Information Systems Committee, London,
http://sites.google.com/site/soundsgooduk/ (accessed 10/05/2010)
7. Sipple, S., (2007), Ideas in practice: development writers’ attitudes towards audio and written
feedback, Journal of Developmental Education, 30 (3), pp 22-31.
166
THE IMPACT OF PEER ASSESSMENT ON ACADEMIC SUCCESS
Marie Devlin Chris Phillips Lindsay Marshall

Newcastle University Newcastle University Newcastle University
School of Computing Science School of Computing Science School of Computing Science
Newcastle, UK Newcastle, UK Newcastle, UK
marie.devlin@ncl.ac.uk chris.phillips@ncl.ac.uk lindsay.marshall@ncl.ac.uk
ABSTRACT
A student’s learning achievements are still interpreted, largely, in terms of their quantitative scores on
assessment tasks completed during or at the end of a module of study. Associated with this interpretation is
the somewhat tacit assumption that, if a student scores well on assessments, they must be highly engaged
with the learning methods and materials associated with that module and their level of engagement has
contributed to their success. In this paper, we reflect on peer assessment methods used in group work and try
to determine if and how these can impact on student engagement and academic achievement. We review and
analyse the quantitative and qualitative results from two undergraduate modules that use different peer
assessment methods. We try to determine how the use of peer assessment in each of the modules has
impacted on students’ engagement, their learning outcomes and their overall academic achievement. We
then discuss how greater engagement could be fostered using a wider range of peer and self-assessment
methods and we also give examples of how these methods can be incorporated into current assessment
frameworks to facilitate greater student learning and a more accurate measurement of student achievement.
Keywords
Peer assessment, student engagement, academic achievement
1. INTRODUCTION
Team work assignments have become the norm in most HEI undergraduate programs for the sound
pedagogical reason that university education is not just about developing what people know and understand
in isolation but about learning from and with others, for the benefit of society. Teamworking helps students
“shift away from simple academic achievement to much broader goals – preparing them for their working
lives” (Leik et al, 1996). However, assessing individual performance in teamwork can be complicated because
we need to measure the effectiveness of the team process as well as the product/s delivered at the end of the
learning activity. Accreditation and certification requirements ultimately mean we have to generate a
summative mark that fairly represents the effort each student has put into the team activity and the level of
success they have achieved based on the expected standard for their level of study. However, converting a
student’s contribution on a group task into a numeric grade “is a complicated and problematic task”, (Leik et
al, 1996). At Newcastle we have often used peer assessment as one way to understand the team process
more accurately. We view it as a good mechanism for getting a more rounded picture of how teams have
interacted both formally and informally when trying to get their work done. With any assessment, the mark
given is the final interpretation of learning achievement but often associated with this mark is the assumption,
somewhat tacit, that if a student has performed well and achieved a high personal score, they have been
highly engaged in the teamwork process and this level of engagement has contributed to their success. But is
this true? We wanted to know if the peer assessment methods we use were accurate measures of a student’s
contribution to teamwork and their engagement in the activity. We also wanted to find out if the methods
themselves had impacted on the students’ engagement in any way. In this paper we review and analyse the
quantitative and qualitative results of two different methods of peer assessment used in our undergraduate
teamwork activities. We try to determine the impact of these methods on students’ engagement with the
activities and their overall academic achievement. We then outline some alternative methods of peer and self-
assessment that could be used to increase engagement and give examples of how these methods can be
incorporated into current assessment frameworks to facilitate greater student learning and a more accurate
measure of student achievement.
2. PEER ASSESSMENT AT NEWCASTLE
2.1 Method 1: Allocating Percentages
The Software Engineering Team project at Newcastle is a year-long project activity undertaken by our level 2
students. Students are formed into teams and must create a large piece of software and documentation by
the end of the academic year. Student teams are expected to work almost independently but they are given
support when needed. For peer assessment in this project we ask students to divide 100% between their
team members based on their perception of how much effort and contribution to the project processes and
deliverables each team member has made. Students complete the peer assessment twice during the project,
once, quite early on in the project – approximately half way through the first semester and again at the end of
the second semester when all the major deliverables have been submitted and demonstrated. For both
instances students are asked to distribute 100% between team members, including themselves. An example
percentage allocation for a 5 person team is illustrated in Figure1:
Anonymous Peer Percentages total/5 to 1dp

1 2 3 4 5
Student 1 24 20 25 16 28 22.6
Student 2 15 20 17 20 20 18.4
Student 3 16 20 17 19 16 17.6
Student 4 20 20 26 16 15 19.4
Student 5 25 20 25 29 21 24
100 100 100 100 100
Figure 1: An example of percentage allocation
In this example, the total column actually adds up to more than 100% for the whole team but this fact is largely
deemed insignificant, it is hard to be precise but we round up above 0.5. The marks for all team assignments
are then multiplied by these two peer assessment marks to get the student’s overall individual mark for the
team deliverables e.g. if a design document received a mark of 77%, Student 1 would receive 22.6% of the
group mark of 77% for the document = 17.04. We use both individual and team assignments during the
module but also monitoring and observation of teams by a member of staff. The member of staff allocated as
a monitor attends each team’s weekly formal meeting and assesses student’s effectiveness during the
project, as a team and as an individual, based on their performance in meetings throughout the year. It is the
combination of marks from individual assignments, team assignments, monitor observations and peer
assessments that are used to calculate an individual’s module mark at the end of the year.
2.2 Method 2: “Show Me the Money!”

The second project in which students peer assess is a video filming project that students undertake in the
Information Handling module. Their task is create a 5 minute video relating to their chosen theme e.g. Student
Life. The aim of the project is to illustrate the legal issues that arise e.g. copyright of images, filming in public
places but also the technical issues of film editing and of course the important aspect of working in a team.
Students are placed into small teams of 5-6 students and asked to produce a series of assignments relating to
their final video submission. These assignments include writing storyboards and shooting scripts. As part of
the module students are asked to review their team process via peer assessment. They are asked to
distribute an imaginary £1000 bonus between group members according to their perception of each person’s
effort and contribution to the group goals. Each student allocates a proportion of the £1000 to their team
mates again, including themselves. The tutor then finds the maximum amount allocated to one student and
that student gets all the mark given e.g. 20 out of 20. Everyone else gets the marks times their total divided by
the maximum amount. So, for example, if a maximum mark is 19/20 for a piece of team coursework the
allocations for a team of 5 students would be as follows in Figure 2. In this example students B, C and E
would all get the full 19/20 and students D and A would get 17.9/20 each.
168
Student A (£200) (£200) (142.86) (£200) (£200) 942.86 17.9
Student B (£200) (£200) (£200) (£200) (£200) 1000 19.0
Student C (£200) (£200) (£200) (£200) (£200) 1000 19.0
Student D (142.86) (£200) (£200) (£200) (£200) 942.86 17.9
Student E (£200) (£200) (£200) (£200) (£200) 1000 19.0
Student A’s mark: 19 * 942.86/1000 = 17.9
Figure 2: Allocation of money in Information Handling
3. IMPACT ON ENGAGEMENT & SUCCESS

Carini et al, in an analysis of student surveys from 14 institutions and 1352 students defined some student
engagement indicators as positive views on
 the level of academic challenge in a programme,

 the level of student-staff interaction,
 a supportive climate
 the level of active and collaborative learning.
They found that whilst student engagement is linked positively to learning outcomes such as critical thinking
and grades the relationship between engagement and academic performance was not as robust as they had
hoped. Student engagement constitutes a constellation of institutional processes that may add value to
student learning, (Carini et al, 2006). At Newcastle, both the Software Engineering and Information Handling
modules have mainly received positive student feedback. Students enjoy doing something different, they like
working in teams and they are eager to take on the challenge of creating a video or a large piece of software.
Student performance is also generally what is expected within a level 2 class. There are however, quite a few
disagreements in teams when it comes to allocating marks for peer assessment and students often complain
about this process to module leaders. We want to change peer assessment so that students have greater
engagement with their team projects and to ensure the process itself helps students with their learning.
We have been using the percentage sharing method in the Software Engineering Team Project for over 5
years. We compared a small sample of module marks from 12 teams during one academic year (2008/09).
There were 106 students in the class. We examined the interim and final peer percentages for the module to
see if there was any indication that peer percentage was a reflection of the student’s overall mark for the
module. The average team size was 9. The lowest interim percentage mark give was 1.55% whereas the
lowest final percentage was 0. The highest percentage given to a team member at the interim was 19% and
the highest mark for the final set of peer assessments was 24. There was a lot of variation between teams in
the second semester, compared to the first. We found that students were more inclined to give each other
equal marks in the first half of the project when only one major deliverable is submitted i.e. the requirements
document. The second semester is when the project enters the major implementation period. This is often a
tense time for the students, especially if they have not stuck to their original plan or have poor team cohesion
or morale. We found that there was a wider range of marks used by each team for all second semester
percentages and that peer assessments strongly reflected students’ final mark in the majority of cases. Also 8
of the 10 best performing students in the class received a poorer rating from their peers in the second
semester, with 4.87 being the biggest percentage fall. This result has led us to believe our design and
schedule for the project may have caused unnecessary stress and contributed negatively to performance and
peer assessment outcomes during the second semester.
We have only been using the peer assessment in the video project for one year and therefore believe it is too
early to draw any conclusions on the quantitative results. One interesting observation we have for this class is
how the use of money rather than marks in the peer assessment has affected students’ approach to peer
assessment. Anecdotally, we have found that students are much more focused on the assessment task when
it comes to allocating money and less focused on their overall module mark and the outcome of the project in
terms of their final grade. Students also seem to be more objective in their allocations. We have observed that
money is less emotive for them than marks and they are inclined to be honest and forthcoming about their
own contribution when it comes to a discussion on how the £1000 should be distributed. We think that this is
because the money is seen as unconnected with their final degree classification and is viewed as more of a
169
bonus mechanism that graduates could encounter in their working lives on a real project. This is in complete
contrast to how peer percentages are viewed in the Software Engineering project where discussions often get
very heated when it comes to allocation of marks. There is a reliance on students to be honest for this peer
assessment and although we think students are honest, some safety measures need to be taken to maintain
academic integrity and quality standards. One such safety measure could be the use of contribution matrices,
(Devlin et al, 2008). A contribution matrix is a grid where each task the team completes is broken down into
sub-sections and students must note down all the sub-tasks they contributed to. The matrix is then agreed by
all team members and submitted along with the assignment/deliverables. We are going to observe the
allocation of money in the Information Handling module next time round to see if our observations are
repeated. We believe the move away from allocating marks on this peer assessment task has had a positive
effect.
3.1 Feedback from students

In both projects, peer assessment is used as a summative measure of contribution towards the end of the
learning activities. However, in the case of Software Engineering Team project, peer assessment is also used
as a formative process in the first semester. It is the summative peer assessment that seems to cause the
most friction between teams. Feedback from both modules on the peer assessment exercises has been quite
negative as can be seen in the Figure 3:
“I was not happy with my original peer percentage for the second semester which was 14.4 mostly
because I felt like I had done more work than some others. After listening to my objections, my mark was
increased by the team. However, this meant that two other students lost some marks as a result.
Obviously they were not happy about this. Eventually we all (reluctantly) agreed on a mark that was fair.
Personally I don’t think this is a good system as it can be abused easily, for example two members could
unfairly rate each other.”
Figure 3: Student Comment
This student who commented in Figure 3 took part in the Software Engineering project. We specify that
students should conduct the peer assessment exercises during formal meetings when their monitor will be
present. This is one way we try to ensure that members are not forced to agree to a mark that is unfair but
also, because we try to avoid the situation of collusion that this student mentions. Unfortunately, we cannot be
fully sure that collusion has been avoided. There is also some evidence of students performing differently in
observed and unobserved team meetings. Other student comments have made us think about the
assignment tasks that we set for the students. We ask students to choose team roles based on their skill
strengths and weaknesses, (Devlin et al, 2007), but the choice of roles and tasks means that some students
may not get a fair allocation in the first set of peer assessment percentages, through no fault of their own.
Evidence of this impact can be seen in the statement in Figure 4 where the scheduling of tasks and
assessment deliverables has had an impact on the first set of marks some students receive. This may
account for some of the differences between ratings in the first and second semester for the high-scoring
students mentioned earlier.
“At the moment much of the work is technical and it is clear that certain members of the group have
stronger ability in this area. We believe that as the project progresses each team member will have more of
an opportunity to stand out and achieve a higher personal percentage.” Student Comment
Figure 4: Comment on Module Design
Some teams also noted that peer assessment had a demoralizing effect on the team and affected
relationships, which was not really optimal for the continuation of work during the second semester. The
reason we introduced peer assessment in the semester of these projects was for the purposes of giving
formative feedback to students i.e. to help further learning and to allow them to modify their behaviour to
improve their learning outcomes during the project and ultimately their final mark at the end (Freeman et al,
2006). The use of peer assessment at the end of each of the projects was, in part, to help us estimate an
individual’s contribution for summative purpose, but mainly to help students reflect on their performance and
what they have achieved and learned. However our students’ experiences of peer assessment are somewhat
negative and we need to improve this. It may be that project scheduling, the methods we use for peer
170
assessment, (especially the one for allocation of percentages) and the methods that we use for monitoring
performance during meetings are not working as well as we had hoped in combating collusion but also in
terms of engaging students with the process and reassuring them about fairness. More importantly the
methods are not helping students to focus on their learning or to assess their skills development throughout
the project.
4. GREATER ENGAGEMENT VIA PEER ASSESSMENT

According to Falchikov, when it comes to peer assessment, students need guidance, (Falchikov, 1988). One
of the problems with our methods is that we do not give our students much guidance on peer assessment. We
do give them tips about making fair decisions e.g. we ask students to focus more on tasks, attendance,
performance rather than personality traits and behaviour. However, this is not enough, we admit that.
Students are also very mark oriented and any assessment should be closely tied to feedback and aimed at
motivating students (Falchikov, 1988). We do not give them any guidance or practice on peer assessment
before they have to do it for real. We ask students to judge a person’s performance but do not give them any
real criteria to do so. Students are therefore unsure of the reasons for peer assessment and find it difficult to
do. Some ways to increase students’ engagement with the peer assessment process is to illustrate the real
value placed on evaluating performance as a skill needed in the real world. We could make the process more
realistic in terms of an appraisal process students will experience in their future careers. Goldfinch and
Conway both suggest the use of matrices outlining expected task performance and behaviour but both these
approaches still ask students to assign values and marks to their peers and Goldfinch’s approach uses
negative marking, which might make peer assessment more emotive than it is already, (Goldfinch, 1994,
Conway et al, 1993). Alternatively, Smith and Smarkusky’s approach to peer assessment is similar in that they
use matrices however, their competency matrices focus on skills and competencies and ask students to rate
themselves using common degree classifications rather than actual marks, (Smith and Smarkusky, 2005).
This is almost a variation on our money distribution approach, (see Figure 6). These matrices or an adaption
of them would help to capture learning and give students rich feedback on their abilities and skill levels more
than our current approaches.
CLASS RANK 1ST 2ND 3RD

Process Steps required to complete project
Task Performance Exhibits on tasks Supports others in completing tasks Motivates others to independently stay on task
behaviour consistently
Leadership skills Learns about leadership Rehearses leadership skills Exercises leadership skills
skills
Figure 5: Smith and Smarkusky’s Competency Matrix
A competency matrix captures team knowledge and skills in various categories (process, communication,
interaction, contribution and responsibility). These matrices then allow an assessor to assign a numerical
range of proficiency in each specified competency – individuals are evaluated by selecting a class rank to
indicate the baseline competencies expected of the individual. In this work, peers assess whether an
individual has met the expectations, exceeded the expectation by various amounts or requires improvement
(varying amounts of improvement can be denoted). Using a similar approach would enable students to focus
on behaviour during the project and the allocation of marks would be left to the tutor. Students would have to
evaluate themselves and the performance of others in their team but not assign grades. This would support
formative learning better and remove the emotive aspect that assigning marks often evokes. Students could
be provided with examples of how to assess in a workshop before the project commences. We could show
them how to approach a peer review and how to fill in the matrices for themselves and others. The use of
competency matrices would also introduce an important element of self-assessment, early on in the projects
and give students the language to assess their skills and performance throughout the year. Staff could use
the same matrices and criteria to assess students as the students themselves and then compare matrices in
order to assign grades based on a scale defining if they have met expectations, exceeded expectations or if
they need to work on certain areas. This would give a sense of continuity, a transparency in assessment and
a shared language for staff and students to discuss performance. This method would also give students some
experience of ‘social comparison’. Students very rarely get to view the results and feedback given to others
and therefore find it hard to compare their performance to that of others in their class in a formal or managed
way. The method would also avoid the collusion problems we have experienced and help students to focus on
the learning or ‘apprenticeship’ aspects of a project. We could use the matrices periodically to get students to
evaluate how they think they are performing along a set of pre-defined competency areas. For software
171
engineering and information handling students we could use competency matrices based on subject-specific
competencies identified by Turley and Beiman, for example on “using knowledge, researching, investigating
and problem-solving, communication and developing solutions’, (Turley and Bieman, 1995), but the work
outlined here could be adapted quite readily for other disciplines and not just Software Engineering or
Information Handling. The matrices would be used in conjunction with the traditional assessment of technical
work products and tangible deliverables such as code and documentation.
5. CONCLUSION
All team project experiences are designed to give students a realistic experience of working on a large piece
of software and an insight into what it is like to work on real problems within development teams. The
alternative methods of peer assessment we outline here offer an opportunity for students to receive higher
quality feedback on their progress and development and to determine how to further develop their skills
without focusing on grades which may help with their engagement in peer assessment and in their overall
project tasks. The current matrices that we have proposed focus on skills and knowledge needed in our
modules but it is feasible to identify generic skills and competencies required in team working scenarios for
other disciplines. Rarely does a professional person work in isolation these days and generic skills such as
communication, leadership, negotiation and problem-solving are required of most professionals in the modern
workplace. These generic team working skills could be used in competency matrices that focus on skills
rather than scores and substituted for the peer assessment methods we have outlined here. We believe that
the focus on skills rather than marks might help students engage more with their learning and with peer
assessment exercises which should have an impact on their learning outcomes and achievements.
6. REFERENCES
Carini, R., M., Kuh, G., D. and Klein, S., P., (2006), Student Engagement and Student Learning, Research In
Higher Education, Vol. 47. No.1. Feb 2006, pp 1-32.
Conway, R., Kember, D., Sivan, A., and Wu, M., (1993), Peer assessment of an individual’s contribution to a
group project, Assessment and Evaluation in Higher Education, 18(1), 1993, pp 45-56.
Devlin, M., Drummond, S., Phillips, C. and Marshall, L., (2008), Improving Assessment in Software
Engineering Student Team Projects, In 9th Annual Conference of the Subject Centre for Information and
Computer Sciences, 26th-28th August 2008, Liverpool Hope University, White, H. (ed.), Higher Education
Academy, Subject Centre for ICS, 2008, pp 133-139
Devlin, M., Phillips, C. and Marshall, L., (2007), Making Computing Science Students More Employable with
Problem-Based Learning and Cross-Site Teamwork, In International Conference on Engineering Education
and Research (iCEER) 2007, Melbourne, Australia, 2-7 December 2007 , International Network for
Engineering and Education Research, 2007, Notes : Proceedings on CD-ROM. Session : Industry, Problem
and Project Based Learning. Paper no. 5. 11 pp.
Falchikov, N., (1988), Self and peer assessment of a group project designed to promote the skills of
capability, Programmed Learning and Educational Technology, 25(4), November 1988 , pages 327 – 339
Freeman, M., Hutchinson, D., Treleaven, L., Sykes, C., (2006), Iterative Learning: Self and Peer Assessment
rd
of group work, Proceedings of 23 Annual Conference Who’s Learning, Whose Technology?, 2006, pp257-
266.
Goldfinch, J., (1994), Further developments in peer assessment of group projects, Assessment and
Evaluation in Higher Education, 19(1), pp29-35.
Lejk, M., Wyvil, M., Farrow, S., (1996) Survey of Methods Deriving Individual Grades from Group
Assessments, Journal of Assessment & Evaluation in Higher Education, Vol. 21, No. 3, pp267-280.
Smith, H. and Smarkusky, D. L., (2005), Competency Matrices for Peer Assessment of Individuals in Team
Projects in Proceedings of SIGITE ’05, 2005, pp155-161.
Turley, R., T. and Bieman, J., M., (1995) Competencies of Exceptional and Non-exceptional Software
Engineers, Journal of Systems and Software, A Survey of Methods Deriving Individual Grades from Group
Assessments, Vol. 28, issue 1, pp 19-38.
172
STAFF LED INDIVIDUALISED ASSESSMENT – A CASE STUDY
Thomas Lancaster Robert Clarke
School of Computing, Telecommunications and School of Computing, Telecommunications and
Networks Networks
Faculty of Technology, Engineering and the Faculty of Technology, Engineering and the
Environment Environment
Birmingham City University Birmingham City University
thomas.lancaster@bcu.ac.uk robert.clarke@bcu.ac.uk
ABSTRACT
The problem of maintaining academic integrity when setting and marking assignments is of continued
concern. Assignment specifications need to be structured so as to minimise the opportunities available for
students to cheat. This includes students colluding with their peers and through contract cheating. One
solution is to individualise the assessment taken by each student, so that each student within a cohort works
on a different problem. This offers an additional benefit of adding elements that allow an assignment
specification found placed on a contract cheating auction site to be traced back to the culprit.
This paper presents a number of methods through which assessment can be individualised. The focus is on
individualisation that is led by staff. Four case studies of computing assignment specifications that the authors
have individualised are presented. The intention of this paper is to assist tutors who are considering
implementing their own individualisation strategies.
Keywords
assessment, individualisation, plagiarism, contract cheating, academic integrity.
1. THE TRADITIONAL ASSIGNMENT

The traditional written assignment, where a tutor compiles a standard brief to which each student produces a
solution for assessment, is well established as a mainstay of education. The written assessment is commonly
used in education at all levels, including university. Written assignments are necessary in many cases to test
abilities which cannot be evaluated solely through examination. For instance, seeing a student work on a
prolonged software development problem may provide a tutor with a much higher level of confidence in the
student’s abilities than would be gained through an examination.
Despite this, problems with traditional assignments are well known. Assignment specifications can be reused
to the point at which standard solutions are easily accessible via essay bank sites (Clarke & Lancaster, 2006).
Such reuse could be said to be actively encouraging students to cheat and plagiarise. Other types of
plagiarism are equally possible. Students can copy from their peers, or from others who have studied the
class in previous years. At any point when student work is produced away from a supervised setting, there is
a danger that the student is not the same person who is actually completing the work. In particular, the issue
of contract cheating, where assignment completion is outsourced, has been documented as a key problem in
the computing education literature (Lancaster & Clarke, 2007b).
A number of good practice solutions to both plagiarism and contract cheating are commonly recommended
(Culwin & Lancaster, 2001; Lancaster & Clarke, 2007a). The continued use of these recommendations should
be strongly encouraged. Examples include: always setting an original assignment specification not found
online or in textbooks; monitoring known cheating sites; looking for unusual indicators in student work; and,
supporting written submissions with a practical examination or viva. In order to preserve academic integrity it
is necessary for tutors to put the same effort into setting assignments as would be expected from students
completing them.
In the modern world, when students are highly technologically literate, such approaches are not always
enough. Purely following good practice will not stop a student from paying someone else to complete the
assignment on their behalf. Such approaches may also not motivate students to work to the best of their
abilities. The argument from an individual student, that if a tutor is marking 200 answers to the same brief,
173
why would the tutor give their work full attention, can be persuasive. Educators need to continue to develop
improved approaches.
A key recommendation is individualised assessment (Fincher et al, 2006). This is where the exact
requirements a student needs to complete are tailored to them as an individual. Such an approach
discourages collusion within a class; since each student has their own individualised solution to produce,
merely copying from a peer will not allow them to do this. Knowing that tutors are taking assessment seriously
can be highly motivating to students who do not want to see their own hard work devalued by cheating from
others.
Individualising an assignment has a secondary key benefit, that of identification. If a common assessment is
placed onto a contract cheating site it is often impossible to identify which student from a large cohort posted
it. A set of unique requirements can be traced back to an individual. This has both deterrent and detection
advantages, two key parts of any successful anti-plagiarism approach. A student knowing that their
misconduct is easily traceable is discouraged from putting their specification online. If the work does appear
online, the miscreant can be traced.
This paper considers ways in which assignment specifications can be individualised. Seven common
individualisation mechanisms are first presented and particularly favoured methods are highlighted. Four case
studies of the authors’ own assignments are discussed in more detail, designed to illustrate the
individualisation techniques. The paper concludes with recommendations for tutors and identifies issues that
require further understanding.
2. APPROACHES TO INDIVIDUALISATION ASSESSMENT

There are many ways in which assignment specifications can be individualised. For the purpose of examples
in this paper, these are split into student led and tutor led approaches. Student led approaches are primarily
those where a student selects an area on which to focus. This choice may be restricted to a greater or lesser
extent, for instance by an overall module topic, but a selection is not usually forced upon a student. Tutor led
approaches are those where a tutor decides exactly what each individual student will work on. This may be
done by hand or automated.
Figures 1 and 2 set out some key classifications of ways in which each type of individualisation may be
approached. Each is accompanied by a brief description and the outline text of an example. The lists are not
intended to be exhaustive, as there are many possibilities about how exact assignment specifications may
differ. Instead, they are intended to be illustrative and offer a flavour of the different methods. Some major
differences between the approaches, with respect to academic integrity, are identified.
Classification Description Outline Example
Open Students are given an overall subject for an Write a computer program to solve a
assignment. They then choose an area on problem of your choice.
which to focus.
Restricted A refinement of Open. A number of options Choose from the following problems…
are presented. Each student chooses from Write a computer program to solve the
these options. problem.
Unique A refinement of Restricted. A number of Choose one of the following topics…
options are presented. Each student chooses Notify your selection to a tutor (if this has
from these options. A maximum of one been taken you must reselect).
student is allowed per option.
Write a computer program to solve the
problem you have selected.
Figure 1 – Student Led Individualised Assessment Approaches
The methods identified in Figure 1 may all be valid in certain situations, but there are some common dangers
to all of these. The main one is that a student may have already found a solution before choosing a topic. This
is particularly problematic for a totally open choice, when a student can make a selection at any point up to a
final deadline. Where a lesser number of options are available, it may be possible to ensure that prewritten
solutions are not available, but this cannot totally avoid the possibility of student collusion.
Allowing only one student per topic is the most secure. Unique topics can make it more difficult for a student
to select a ready-made solution, as well as make it possible to trace attempts to outsource work found online.
But coming up with sufficient, well-balanced, examples can be onerous for tutors.
Despite the disadvantages, allowing students a fairly open selection can be useful in certain situations.
174
An illustrative example is student projects. Many universities give their students an open selection of areas on
which to work. There can always be the danger of a prewritten solution or a solution completed piecemeal
through an outsourcer, but this can be minimised by a carefully considered project process. The project
system used in the Computing Department at Birmingham City University goes one step further, requiring staff
to compile a set of project outlines, and for students to select from these on a first-come first-served basis.
This has the advantage of ensuring that pre-produced solutions are highly unlikely to be available, especially
if a student is slow with selection. It also allows for the suitability of each project topic to be monitored prior to
allocations, ensuring that appropriate topics are available for all students.
Classification Description Outline Example
Single Students are given an overall subject for an Your individual problem on which you have
assignment. Each is then allocated a unique been allocated to work is …
area on which to work. Write a computer program to solve the
problem.
Subset Staff prepare a list of activities. Each student is Write a computer program which
required to work on a different subset of these demonstrates activities 2, 4, 11, 13 and 17.
activities.
Cycled An assignment is constructed as a series of Write a computer program which
activities, each with different options. Students demonstrates activities A3, B1, C5, D2,
are allocated an option from each activity. E2.
Data Each student is given a set of unique data, on Write a computer program to process your
which they perform a common set of tasks. unique dataset and comment on the
significance of the results.
Figure 2 – Staff Led Individualised Assessment Approaches
The examples on which this paper focuses are all staff directed and largely in line with the classifications
shown in Figure 2. This is generally for the reasons of academic integrity already discussed. Further, each of
these staff led approaches immediately adds traceability to work that is found online. Providing that the
combinations are unique, it is possible to identify an individual student.
A common criticism of staff led individualised assessment is that the amount of work required on the part of a
tutor may be onerous. This can be immediately addressed. Certainly, there is a need to prepare a greater
number of activities than would be needed for a single assignment completed by everyone. These would
allow for the larger number of combinations of activities necessary. But there do not need to be a huge
number of activities prepared for this. Further, these individual activities do not need to differ greatly.
For example, consider an assignment specification produced using the cycled option. The specification,
consisting of four sections, each with four different activities (of which students complete one activity per
section), would offer 256 combinations, enough for all but the biggest classes. Activities do not even need to
be massively distinct, for instance a computer system specification problem could just rely on different
combinations of core equipment required, alongside different budgets. The cyclic option can also allow for
resubmitted work, providing these students are allocated a combination to work on different from their original
one. This has side advantages, ensuring that first sit and resit work is of the same difficulty, and removing the
necessity for a tutor to write an entirely fresh assignment for what is ideally a small number of students.
Care does need to be taken when writing the assignment briefs. When an assignment is intended to assess a
range of learning outcomes, the generation method must also ensure that every combination covers these
learning outcomes. It is also important that the difficulty of the work allocated to each student is equivalent.
Although the assignments generated may appear crude based solely on the outline examples in Figure 2, this
need not be the case. A common method would be to present these to students in a single pre-merged file.
This could be done simply by using the MS Word mail merge facility. Generating individual versions in
advance has the benefit of only showing the students their own unique assessment, preventing potential
confusion and ensuring that students cannot see what their peers are working on without their express
permission.
In the authors’ cases, most of their individualised assignments have been produced with the aid of an MS
Excel or MS Word document with a Visual Basic back end. This file be passed on directly to students as an
Excel file (for data processing), or converted into PDFs (this offers enhanced security and the option of
embedded visible and hidden fingerprints for traceability purposes). Further, many individualised assignments
have been produced using a combination of these methods. Section 3 of the paper explores four of them in
more detail.
175
3. CASE STUDIES
3.1 Number Systems Quiz
The Number Systems Quiz is an example which has been used successfully for many years across hardware
modules on a range of courses using the Data Individualisation approach.
Students enter their student ID into an Excel spreadsheet and an individualised set of data is generated in the
form of a hexadecimal memory dump. An individual set of questions is generated alongside the data, relating
to the exact memory locations to be identified and the conversions to be performed on them. A separate Excel
answer spreadsheet is used by the tutor to generate the answers and the work is marked by hand.
This assignment specification has been safely reused by changing the hashing algorithm used to work out the
data that the student sees. This ensures that, even if a student retakes the assignment, they will receive a
different memory dump and different questions, thus extending the lifetime of the assignment.
Students are required to submit handwritten pages showing how they have worked out the answers to each of
the questions. This allows marks to be allocated to students who show an understanding of process but fail to
get the correct answer. Handwritten work also allows for another level of checking for academic integrity.
With small cohorts of students, the tutors allow students to have their work checked to find out which of their
answers are correct. This can be done quickly using the marking spreadsheet and ensures that the students
receive formative feedback. This encourages increased participation by students.
3.2 Library Stock

The Library Stock case study has been used successfully in a series of mathematical and statistical modules.
Students work on a unique set of data concerning the levels of book stock in a library. They are then asked to
calculate various values from their data, such as averages and percentage increase corrected for RPI. They
are also asked to draw graphs of the data, one example being a regression graph. As with the Number
Systems Quiz, a marking spreadsheet assists the tutor, providing numeric answers and sample graphs.
Unlike the Number Systems Quiz, the students are also asked to provide qualitative responses, with
questions along the lines of “comment on the significance of the result”. This allows the understanding of the
student to be tested at a deeper level, although this can lead to an overhead for the tutor marking the work.
As with the Number Systems Quiz, the assignment can be reused by giving students new data. In actuality,
the topic of the data is easily changed. For example, the library stock could work equally well as sales figures.
Only minor changes are needed to the overall spreadsheet.
3.3 Computer Network Quiz

The Computer Network Quiz assignment uses both the Subset and Cycled approaches.
Students are initially presented with a selection of short answer questions, drawn from a selection on a
spreadsheet. They are also given a longer topic which requires more in-depth research and investigation.
Although each student gets a different topic to work on, the individualised assessments all have the same
format, which helps to allow students to see that they are of the same level of difficulty. Students are required
to be selective about their responses and to justify their decisions.
This case study requires the most effort from the tutor out of those identified so far, both to prepare and mark.
Efforts have been made to make this marking easier by the use of a grid in which students place their
answers. This reduces the amount of reading they need to do when marking. An example is shown in Figure
3.
176
Figure 3 – Computer Network Quiz Example
Although this example comes from a Computer Network module, a similar approach has been used in other
subjects.
3.4 Self-Marked Quiz

The Self-Marked Quiz works by using a large bank of questions, each of which is in a multiple choice format.
Here, a spreadsheet extracts an individual set of questions and presents them to students. The student enters
their answers as letters and presses a button, which ‘marks’ the work.
The computer responds with an overall score, but does not indicate which questions are right or wrong.
The spreadsheet allows the student multiple attempts in an effort bolster their score. To prevent students just
randomly trying to guess, the ‘mark’ button is engineered to disappear once pressed. To retry the student
must reload the spreadsheet and re-type the responses.
When the student is happy with the score that they have they printout the spreadsheet with their mark
recorded on it. To ensure that the printout is not tampered with a series of check values are also printed out.
This type of assignment is generally used for a small proportion of the marks for a module, but can be
adapted.
4. RECOMMENDATIONS
The case studies presented here represent a number of products developed by the authors over the course of
10 years. Each is still hand-crafted to a degree to implement changes in trends, module allocations and good
practice recommendations. Each also represents a base from which many types of individualised
assignments can be generated.
A completely new assignment does take a long time to prepare. As such, it is important that it is produced with
re-use in mind and that it will offer a long shelf life. However, implementing new individualised assessments
using methods like these is not feasible for all tutors. A level of technical ability is required. Ideally, a toolkit
could be developed to make the process more productive and easier for non-technical tutors to use. This
could be linked with the different staff led individualisation approaches.
Individualised assessments like these have the potential to increase student engagement if development
continues. Support can be provided directly to students based on their own individualised data. Self-marking
can also take place to offer either formative or summative results.
One further issue still needs to be explored further, that of traceability. Much of the contract cheating detective
work relies on using Google to find text within an assignment specification (Clarke & Lancaster, 2007).
Therefore, it is important that fixed phrases are embedded within the assignments. Ideally, each should allow
an assignment specification to be traced to an individual institution and tutor.
It is anticipated that this paper will further open the discussion about individualised assessment. The approach
should form part of an overall best practice study of assessment across the computing sector of higher
education. If nothing else, it is intended that the individualisation approaches should provide a potent way of
restricting the opportunities of students to cheat.
5. REFERENCES
Clarke, R. & Lancaster, T. (2006). Eliminating the successor to plagiarism? Identifying the usage of contract
cheating sites, in Proceedings of 2nd Plagiarism: Prevention, Practice and Policy Conference 2006
Clarke, R. & Lancaster, T. (2007). Establishing a Systematic Six-Stage Process for Detecting Contract
Cheating, in Proceedings of 2nd International Conference on Pervasive Computing and Applications
Culwin, F. & Lancaster, T. (2001), Plagiarism Issues for Higher Education, Vine 123
Fincher, S., Barnes, D., Bibby, P., Bown, J., Bush, V., Campbell, P., Cutts, Q., Jamieson, S., Jenkins, T.,
Jones, M., Kazatov, D., Lancaster, T., Ratcliffe, M., Seisenberg, M, Shinner-Kennedy, D., Wagstaff, C., White,
L. & Whyley, C. (2006). Some Good Ideas from the Disciplinary Commons, in Proceedings of 7th Annual
Conference for Information and Computer Sciences
Lancaster, T. & Clarke, R. (2007a), The Phenomena of Contract Cheating, in Student Plagiarism in an Online
World: Problems and solutions, Roberts, T. S. (editor), USA: Idea Group Inc, Hershey, Pennsylvania
Lancaster, T. & Clarke R. (2007b), Assessing Contract Cheating through Auction Sites – A Computing
Perspective, in Proceedings of 8th Annual Conference for Information and Computer Sciences
177
THE INFLUENCE OF PRESENTATION UPON EXAMINATION MARKS
J Hughes S Akbar
School of Computing School of Computing
University of Dundee University of Dundee
Dundee Dundee
jhughes@computing.dundee.ac.uk s.akbar@dundee.ac.uk
www.computing.dundee.ac.uk www.computing.dundee.ac.uk
ABSTRACT
An investigation was made into how the appearance of answer scripts provided in a computing-related
assessment can affect the grades that students are then awarded, in particular comparing typed and
traditional handwritten formats. The comparison was made using the answers to an Information Technology
examination taken by first year computing undergraduate students. Each student completed the Information
Technology exam using a PC and basic text editor to type their answers. Original typed scripts were then
transcribed into a handwritten version and a grammatically-correct and spell-checked version. All three
versions were marked by independent assessors. In an adaptation of the methodology adopted by Russell
and Tao (2004b), each assessor was assigned one presentation format only.
Analysis of the marks awarded indicated that the original typed answers were rated significantly more highly
than handwritten versions of the same answers. This may be because the assessors were more comfortable
marking an answer produced in a word processed format, since they were specialists in the computing field.
However, the set of “perfected” typed answers (spelling and grammatical mistakes had been removed)
surprisingly were awarded lower grades than the original typed answers that did contain spelling and
grammatical mistakes. Possible explanations for this are considered, for example, grammatically-correct
scripts may have indicated a high level of professionalism to the assessors and hence raised their
expectations of in-depth answers. Variation between marks awarded by assessors was higher than expected.
Keywords
Assessment; e-examination
1. INTRODUCTION
Research comparing handwriting with word processing for the presentation of assessment answers can
broadly be identified as having two main phases. In the initial phase of the early 1990s, a small number of
groups investigated the effects of two modes of answering an examination questions – handwritten or word
processed – upon students’ scores. One notable work in this phase was by Powers, Fowles, Farnum and
Ramsey (1994). These authors converted a sample of original handwritten essay answers into word
processed versions and transcribed a sample of original word processed essay answers into handwritten
versions. Analysis of the scoring revealed that handwritten answers were awarded higher average scores
than word processed answers, irrespective of the original mode in which the answers were produced. This
replicated findings of Arnold, Legas, Obler, Pacheco, Russell and Umbdenstock (1990) that student papers
converted to word-processed versions received lower scores than did the original handwritten versions.
Arnold et al suggested that the reviewers may have had higher expectations of the word processed work, less
empathy with the authors of word processed work or may have been less likely to give the benefit of any
doubt. Powers et al suggested other possible factors including lack of evidence in the word processed
versions of evidence to revise work, greater visibility of typographical errors, or apparently shorter answers.
A second phase of research in this area resulted in a series of publications from 2004 onwards. Principal
authors have included Russell and Tao (2004a, 2004b) who found some evidence in support of the Powers et
al work of 1994, suggesting that the lower scores for computer printed work may have resulted from factors
identified by increased visibility of any errors, higher expectations of readers and less empathy with students
(“felt a stronger connection to the writer because of the handwriting”).
 Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided
that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on
the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission.
 © 2010 Higher Education Academy
 Subject Centre for Information and Computer Sciences
178
Shaw (2005) conducted a small study in which a group of IELTS examiners were presented with handwritten
and typed format scripts. One factor commented upon was the legibility of the scripts: one examiner
observed that “Lots of typos have a more negative affect than lots of handwritten ones …” and Shaw noted
that markers give ‘benefit of doubt’ where legibility is an issue. Concerned that an effect appears to result from
the mode of presenting assessment answers, Russell and Tao (1994b) were prompted to investigate if
training of markers could eliminate any such differentials in assessment. The authors concluded this was
possible although they recommended more research in this area. More recent work in the United Kingdom by
researchers at the University of Edinburgh (Mogey, Paterson, Burk and Purcell, 2010) compared transcribed
scripts of first year students in a mock examination: handwritten scripts were transcribed into typed format and
typed scripts were transcribed into handwritten format. Mogey et al found “weak evidence” that handwritten
scripts generally scored slightly higher than typed scripts.
2. RESEARCH QUESTION
Most of the studies described in section 1 applied to students of non-technical subjects. Powers et al (1994)
used a sample of college or university students principally from business, social sciences, humanities/arts and
education. Russell and Tao’s 2004 work featured school students of Language Arts. Mogey et al (2010)
investigated the work by divinity students. Although almost by definition most ‘digital native’ students are
familiar with word processing, it seems unlikely that students of non-technical subjects will be as confident
and competent using a keyboard to present answers as students of computing, who typically use keyboards
on a daily basis for most aspects of their practical learning. The work described in this paper is part of an
investigation comparing on-paper assessment with on-screen assessment in the subject area of computing.
In particular for this paper, the focus was whether handwritten scripts are judged differently to typed scripts in
the subject area of computing. In the style of Mogey et al (2010), the research question was: is the mark
awarded to an examination answer influenced by its mode of presentation?
3. METHODOLOGY
The methodology adopted was based upon the study conducted by Russell and Tao (2004b). Assessments
of 40 undergraduate students in their first year of study were used to test the comparison of handwritten and
computer-based examination answers. All were studying computing at the University of Dundee in the School
of Computing. All the students had regularly been using computers throughout the academic session and
therefore were all experienced with typing, although not all were touch-typists.
The examination (of an I.T. module) took place at the end of semester 2 in academic session 2008/9.
Questions were designed to assess knowledge and understanding of databases, human computer
interaction, programming in general and experimental methods. The questions were delivered and answers
obtained using an assessment management system (Questionmark Perception) that permits a secure
scheduled delivery of exams, locked down to prevent internet access. Use of this system for high-stakes
assessment had been piloted the previous semester with a different IT class. The examination was set and
quality assurance checks performed in the normal way.
Students were provided with the short-answer style examination questions on-screen and were able to type
answers using a rudimentary text editor built into the system. No spell checker is provided with the system,
therefore spelling mistakes were as likely to occur as in a handwritten exam. The questions were presented
one at a time with a feature which enabled the students easily to answer questions in whichever order they
preferred (Figure 1).
In addition, students were provided with a standard answer booklet to use if they wished to sketch out
answers, prepare rough notes or create diagrams. Answers typed were stored on the hard drive. Library and
Learning Centre staff confirmed that all candidates’ answers were stored, and later provided academic staff
with the answer files. Anonymised answer scripts were printed and, for the purposes of the actual
examination, distributed to academic staff for marking.
For the purposes of the research, the typed answers then were transcribed by a set of paid volunteers into
handwritten format. In order to reflect the general quality and style of student handwriting, each transcriber
was given a sample of each student’s handwriting in order that they could mimic the handwriting style. The
transcribers also were instructed to reproduce the exam exactly as originally typed and not to correct spelling
or grammatical mistakes. However any obvious typographical errors were to be corrected. For example a
mistake such as “teh” would have been corrected to “the” as it was judged to be a mistake due to the
student’s typing and therefore would not have occurred in a handwritten assessment. In addition to this, a
further transcription was produced in which spelling mistakes and typographical errors were eliminated whilst
keeping the typed format (Figure 2).
179
Figure 1. Examples of on-screen assessment question and answer system
Figure 2. Transcription processes
Computing PhD students familiar with I.T. were recruited as assessors. An agreed marking scheme was
used to support independent marking and a moderator was used to resolve any discrepancies. Altogether
there were 240 scripts: 40 original typed for each of two assessors; 40 transcribed (handwritten) for each of a
further two assessors; an additional set of 40 were produced for each of a final two assessors to analyse the
effect of correction of spelling mistakes. Scripts were allocated for marking to ensure that each assessor saw
only one presentation format, i.e. to keep the markers as blind as possible to the expected outcomes of the
study (Figure 3). Any question originally leading to the creation of a diagram was excluded from the
subsequent analysis, thus resulting in a possible maximum score of 89 to be awarded. Scores awarded by
each rater in a given pair then were aggregated into a single score by adding the two single scores.
Exam Presentation format

Level 1 I.T. Handwritten Typed Spell-checked typed
(40 scripts)
Assessor A Assessor C Assessor E
Assessor B Assessor D Assessor F
Figure 3. Order of presentation to assessors
4. RESULTS
Handwritten scripts generally scored lower than either typed or spell-checked typed scripts (Figure 4). The
results showed a significant effect of the format for the total score: F(2,117) = 6.081, p < .01. Post-hoc
comparisons (Tukey HSD) confirm that the scores differ significantly for (i) handwritten and typed scripts and
(ii) typed and spell-checked typed (Figure 5).
180
N Mean SD
Handwritten 40 69.1 22.57
Typed 40 86.6 23.32
Spell-checked typed 40 73.6 24.01
Figure 4. Descriptive statistics for marks scored in each format
Mean Difference Std. Error Sig.

Presentation format: F(2,117) = 6.081, p < 0.01
Handwritten vs typed -17.50* 5.21 .003
Handwritten vs spell-checked typed -4.50 5.21 .664
Typed vs spell-checked typed 13.00* 5.21 .037
*. The mean difference is significant at the 0.05 level.
Figure 5. Comparisons between marks scored in each format
This evidence suggests that higher marks were awarded to any form of typed presentation for exam scripts
than to handwritten formats, but that spell-checking and removing errors resulted in less benefit than
uncorrected typing. Mark variability might be due to the differences in the way that the different markers
interpreted or applied the marking scheme. This possibility was also considered by Mogey et al (2010) and
therefore, replicating that approach, the distribution of marks awarded by each marker was investigated
(Figure 6). A broadly similar range, skew and standard deviation was shown across the markers.
Figure 6. Boxplot of score by marker
However, when the marks were categorised (1st, 2.1, 2.2, 3rd or fail), the inter-rater reliability for each of the
three pairs was found to have fair agreement only (Landis and Koch, 1977). This finding corresponds with
that of Mogey et al (2010), who noted the importance of the “differences between the markers”.
5. DISCUSSION
Analysis of the marks awarded indicated two findings. Firstly, the original typed answers were rated
significantly more highly than handwritten versions of the same answers. This may be because the assessors
were more comfortable marking an answer produced in a word processed format, since they were specialists
in the computing field. More surprisingly, the second finding is that the set of “perfected” typed answers
(spelling and grammatical mistakes had been removed) were awarded lower grades than the original typed
answers that did contain spelling and grammatical mistakes. Possible explanations for this include
expectations of the assessors and empathy from assessors. For the former, grammatically-correct scripts
181
may have indicated a high level of professionalism to the assessors and hence raised their expectations of in-
depth answers – a computer-student version of the effect observed by Russell and Tao (2004a). Considering
the latter finding, there may be an effect specific to computing with the juxtaposition of visibility effect (Powers
et al, 1994, considered that greater visibility of spelling and punctuation errors may adversely affect rater
scores) and empathy effect (Russel and Tao suggested that raters might be more lenient if they felt a stronger
connection to the writer). In this case, it may be that computing markers felt a stronger connection to a writer
who made spelling and punctuation errors in their typed answers than to a writer whose typing was ‘perfect’
but whose computing knowledge was flawed.
6. CONCLUSION
This study was designed to investigate the effect of different presentation formats upon the marks awarded to
examination assessments. Results suggest that in the discipline of computing, students may benefit
somewhat from typing answers rather than handwriting them, irrespective of the quality of the answer itself.
Moreover, markers may favour somewhat flawed typed answers over scripts with answers with spelling or
grammatical errors. Given the choice of doing high-stakes assessments on computer or by handwriting,
students may well be advised to choose to type their examination answers. Examination boards may wish to
consider requiring one presentation format, rather than giving students a choice. Finally, there is a strong
indication that notable marker variability existed in this study: programme leaders may wish to review quality
control processes that exist for marking.
7. LIMITATIONS
The principal limitation of this work lies in the design: selected to maximise the preservation of blind marking,
it therefore was not counterbalanced. Had it been, more control could have been exercised over any
differences in interpretation of the marking scheme by the different markers: each of the markers could have
scored one third of each presentation format. Nonetheless, the design chosen perhaps has resulted in the
identification of an unexpected issue, that of variation between the markers. Further work will include a
revision to the design with further data collection and analysis. The fact that the subjects were from one
discipline area only is not judged to be a limitation, since it was the presentation format for that discipline
which was under investigation.
8. ACKNOWLEDGEMENT
This work was funded in part with the support of the Nuffield Undergraduate Research Bursary. Thanks are
also due to the student transcribers and assessors.
9. REFERENCES
Arnold, V., Legas, J., Obler, S., Pacheco, M.A., Russell, C. and Umbdenstock, L. (1990) Do students get
higher scores on their word-processed papers? A study of bias in scoring hand-written versus word-
processed papers. Unpublished manuscript cited in Powers et al (1994).
Landis, J.R. and Koch, G.G. (1977) The measurement of observer agreement for categorical data. Biometrics,
33 pp159-74.
Mogey, N., Paterson J., Burk J. and Purcell, M. (2010) Typing compared with handwriting for essay
examinations at university: letting the students choose. ALT-J, 18 (1), pp 29-47.
Powers, D.E., Fowles, M.E., Farnum, M. and Ramsey, P. (1994) Will They Think Less of My Handwritten
Essay If Others Word Process Theirs? Effects on Essay Scores of Intermingling Handwritten and Word-
Processed Essays. Journal of Educational Measurement, 31 (3), pp 220-233.]
Questionmark Perception, http://www.questionmark.co.uk, (last accessed 21.05.2010)
Russell, M. and Tao, W. (2004a) Effects of handwriting and computer-print on composition scores: a follow-up
to Powers, Fowles, Farnum & Ramsey. Practical Assessment, Research & Evaluation, 9 (1).
Russell, M. and Tao, W. (2004b) The influence of computer-print on marker scores. Practical Assessment,
Research & Evaluation, 9 (10).
Shaw, S.D. (2005) The impact of word processed text on marker behaviour: a review of the literature.
Cambridge ESOL internal report no. 670.
182
AN EVALUATION OF THE DURHAM TEC MOBILE EDUCATIONAL
SOFTWARE SYSTEM IN THE CONTEXT OF PASK-LAURILLARD
CONVERSATIONAL FRAMEWORKS
Michael Batty Phyo Kyaw

UK UK
michael.batty@durham.ac.uk phyo.kyaw@durham.ac.uk
http://tel1.dur.ac.uk/tec/ http://tel1.dur.ac.uk/tec/
ABSTRACT
Modern mobile devices provide students with ability to engage in a wide spectrum of activities, including
social networking and mobile learning. This paper describes the development and evaluation of a context-
aware mobile application, called Technology Enhanced Campus (TEC). It presents how a new context-aware
technology is developed using vector-based location identification using the existing Wi-Fi infrastructure within
the University Campus. It details how a folksonomy-based information visualisation is used to present the
context-sensitive and personalised information to the students. This paper attempts to discuss the
distinguishing features of mobile computing in the context of the TEC project and evaluate the educational
features of the system in a framework for mobile computing due to Laurillard, based on Pask's conversation
theory. It also proposes further amendments to the TEC system to support the framework.
Keywords
Mobile learning, mobile software, student engagement, context-aware, location-aware methods, conversation
theory.
1. INTRODUCTION
1.1 The Durham Technology Enhanced Campus Project
This paper describes the work of the Durham University Technology Enhanced Campus project. The purpose
of the project is to provide a service to students that can deliver content to the students, using their location as
a filter. Initially the design of the system used a subscription model - students could subscribe to certain
locations and receive messages for that location. Locations were divided into zones, buildings and campuses
in a pre-ordained way. For instance, students could subscribe to the library channel to find out
announcements about the library. Thus the system combined personalisation with context-awareness based
on the location. It also introduced a third aspect, social networking. Like other social networking applications
you can send friend requests, accept or reject them. But what is different with mobile social applications is
that you also see the location of your friends on a map.
183
1.2 Disadvantages of the Initial Design
The initial design had several drawbacks. Its location determination algorithm was based upon selection of the
Wi-Fi signal with the greatest strength. This often led to inaccuracies, particularly in determining which floor of
a building a student is on, undermining the subscription model of the system. It also carries no notion of
proximity of Wi-Fi access points; the access points are simply treated as isolated points in space. But what
was perhaps a more serious problem was that the Wi-Fi access point data had to be gathered, associated
with zones and buildings, and this was very time consuming. Moreover, it is unlikely that messages can be so
simply linked with locations, as most locations are used for many different purposes. For example, a lecture
theatre in the department of Physics might be used for Computer Science lectures at certain times.
Furthermore, locations mean different things to different people. The same lecture theatre could be used for a
university society after hours, which has nothing to do with either physics or computer science. What was
needed was a way of associating contexts, meanings, to locations in a way that
 different meanings can be assigned to the same location, perhaps with different strengths (for
example the above physics lecture theatre would probably be 'more associated' with physics than
computer science
 messages (documents) can be ranked, in a systematic and dynamic fashion, based upon these
contexts
 the location context can be personalised and provided by user-generated content
1.3 A Folksonomy-based Approach

To meet these requirements, a new system has been designed and implemented which uses a location-
based folksonomy model. Based on these requirements, the initial design was replaced with a vector location
model (Batty and Kyaw, 2009). Locations are assigned with tags on mobile devices by the various users of
the system (precisely how this operates will be described later). The information is sent to a central server
which stores these tags in a database. The idea is that, with a sufficient corpus of data, it should be possible
to use the semantic context supplied by the tags to variably rank content throughout the continuum of the
university campus, and since the identities of those performing the tagging are also recorded, it can also be
personalised.
2. M-LEARNING
2.1 Distinguishing Factors of Mobile Computing
What is different about mobile computing to standard computing? One factor is that mobile devices are
omnipresent (assuming that the mobile devices are present and operational), which of course means that
resources can be accessed in marginal time, for example while travelling or waiting in queues. The other
important difference is that the modern mobile device is aware of its location (although the location can be
determined in various ways).
2.2 Learning Spaces

Laurillard has analysed what distinguishes mobile learning (m-learning) from other types of learning
(Laurillard, 2007). She points out that it is more than just the building of knowledge in different contexts (in
particular those of time and space), and more than just a different set of working and learning patterns. These
are available in other types of learning. She also recalls a suggestion of N. Winters that m-learning lies at the
interface1 between three spaces: “3-dimensional physical space, social network space, and 'fluid space': the
space of learners, relations and the object of learning” (Winters, 2007). Furthermore, all of these spaces
evolve over time. M-learning seems to take place at the interface of these three domains. But the main point
of the paper is to provide a framework which might be used to decide whether a mobile learning system may
be amenable to conversational type analysis in the sense of Pask (Pask, 1975; Motiwalla, 2007; Atherton,
2009). The main point made is that unless m-learning is evaluated in educational terms there are dangers of
1 meaning in terms of abstract modelling
184
(a) failing to “optimize its value” by not understanding its potential (b) “over-adapting education to
accommodate what it offers”. Laurillard poses the following intriguing question:
“What are the pedagogic forms specific to m-learning that both fully support the learning process and exploit
the richness of the remote environment”.
2.2 Mathematical Interpretation

The notions of pedagogic form and fluid space could be interpreted mathematically. A learning process can
be thought of as a path through the different types of learning such as reflection, analysis, questioning, etc.
which take place between the different stakeholders in the learning process, such as “teacher”, “student”,
“peer” or “observable world”. The paths live, strictly speaking, not in the graph connecting in these objects but
rather in some covering space2 of the graph (in the sense of homotopy theory and algebraic topology), as the
paths do not loop back to the same place, but rather to a place with the same stakeholders but an increased
level of knowledge. Polarizing through a mathematical lens, pedagogic forms may be thought of as paths (or
homotopy classes thereof) in the fluid space and with this interpretation it seems that the challenge in
addressing Laurillard's question would be (a) to design a m-learning system capable of exposing a sufficient
variation of paths to the learner, and (b) to devise sensitive enough metrics to evaluate the relative merits of
each path in the context of learning.
3. CONTEXT-AWARENESS, FOLKSONOMIES AND SOCIAL NETWORKING

3.1 Context-Awareness
The notion of context awareness in computing has been around for at least 20 years (Chen and Kotz, 2000).
Many Web 2.0 applications are context-aware in the sense of user context, or personalisation and this is one
of the distinguishing features of so-called social networking sites. For instance, profile pages, lists of friends,
user histories (in the form of blogs, tweets or status updates) and comments are all common features, the
latter also being sensitive to the context of time. In a sense, so are e-commerce sites, or indeed most web
sites with a log-in system. As regards location awareness, literal knowledge of physical location is of limited
educational interest. Knowledge of latitude and longitude (and possibly altitude) and where matter is and
where space is only has importance when coupled with some representation of the nature of a location,
semantic or otherwise. How a location is used, for instance.
3.2 A Vector Folksonomy Model

The TEC system sought to capture this metadata in the form of tags, as described in the introduction. Within
the physical arena of interest (in our case the Durham University Science Campus), a corpus of sample points
was prepopulated before the software was given to the students. This data was fixed after this point, and
could not be amended by the students. When a student stands in a physical location and enters a tag, the
software associates the tags to a linear combination of the sample points, which is calculated in real time by
projecting the instantaneous Wi-Fi vector onto the vectors recorded for the sample points. A cumulative
tagging table is computed via a stored procedure in the central server's database, which may then be used,
after a sync process is carried out, to display tag clouds for the location. In this way the model may be
considered that of a vector folksonomy, more general than a classical folksonomy (Vander Wal, 2005). In a
vector folksonomy, it is linear combinations of resources which are tagged, not just resources themselves.
This allows for weighted simultaneous tagging, and could conceivably result in more accurate tag clouds for
individual resources. This is being investigated in the context of precision and recall in web search (Batty and
Cummins, 2010).
4. EVALUATION OF TEC IN TERMS OF THE CONVERSATIONAL FRAMEWORK

4.1 Informal Learning
Laurillard's conversational framework for informal learning (i.e. without the intervention of or structure imposed
by a teacher) is based upon interactions between a learner, other learners and the external world of
2 but perhaps not the universal covering space – one would expect different paths to lead to the same learning outcome.
Perhaps this can be modelled by a homotopy relation.
185
experience, (Laurillard, 2007, figure 6.2). It requires facility for learners to question and comment upon other
learners, the reflection of both parties and adaption of actions accordingly.
4.2 E-Graffiti and Lecture Notes

Whilst the TEC system currently allows context to be associated with locations in the form of tags, and these
tags may be used to filter messages depending on location, what is also needed in order to support the
framework are extra aspects of the system to increase interaction and reflection. A means of associating
notes to a location called e-Graffiti has been previously evaluated on a university campus (Burrell and Gay,
2002). Various difficulties are highlighted, such as lack of use, which is attributed to (a) the need for user-
generated content and (b) the lack of contextual focus. This system relies on the implicit context supplied by
the notes themselves and would be unable to use the overall context of a location to filter content. We
therefore propose a system for context-aware lecture-note taking which uses an e-graffiti approach combined
with vector folksonomies. With such a system it would be possible, for instance, for someone to take lecture
notes about mathematics in a history lecture theatre, and these notes would then be highly ranked to
someone who is located somewhere to which mathematical context has been associated by tagging (for
instance the mathematics department common room, or other rooms in the mathematics department). The
system would at least
 auto-tag lecture notes (and possibly also allows human tagging).
 associate tags with locations.
 allow users to add lecture notes at a location.
 allow users to retrieve lecture notes based not only on location but also on the context of the location
and comment upon them.
Such a system could potentially expose useful pedagogical forms specific to m-learning in the covers of the
informal conversational framework, as these would be visible from the timelines of the tagging and
note/comment threads.
REFERENCES
Atherton J. S., (2009), Learning and Teaching; Conversational learning theory; Pask and Laurillard available
at: http://www.learningandteaching.info/learning/pask.htm, accessed: 10 March 2010.
Batty, M. and Cummins, S. (2010), Vector Folksonomy Models Applied to Document Search and
Categorization, in preparation.
Batty, M., Kyaw, P., (2009), Vector-based Location Finding for Context-aware Campus. ICWMC, Cannes,
France.
Burrell, J., Gay, G. K., (2002), E-graffiti: evaluating real-world use of a context-aware system, Interacting with
Computers, 14 (4), pp301-302.
Chen, G. and Kotz, D., (2000), A Survey of Context-Aware Mobile Computing Research, Dartmouth College
Computer Science Technical Report TR2000-381.
Laurillard, D., (2007), Pedagogical Forms for Mobile Learning in Mobile Learning: Towards a Research
Agenda, edited by Pachler, N., London: WLE Centre for Excellence, Institute of Education.
Motiwalla L. F., (2007), Mobile Learning: a Framework and Evaluation, Computers & Education Volume,
49(3), pp581-596.
Pask, G. (1975), Conversation, Cognition and Learning: A Cybernetic Theory and Methodology, Elsevier.
Vander Wal, T., (2005), Folksonomy, available at: http://www.vanderwal.net/essays/05113/folksonomy.pdf,
accessed 16 February 2010.
Winters, N., (2007), What is mobile learning? In Big Issues in Mobile Learning, edited by M. Sharples (Ed.),
LSRI, University of Nottingham.
186
THE USE OF SIMULATION IN DIGITAL FORENSICS TEACHING
Jonathan Crellin Mo Adda Emma Duke-Williams

School of Computing, University of School of Computing, University of School of Computing, University of
Portsmouth Portsmouth Portsmouth
Buckingham Building Buckingham Building Buckingham Building
Portsmouth, PO1 3HE Portsmouth, PO1 3HE Portsmouth, PO1 3HE
jonathan.crellin@port.ac.uk mo.adda@port.ac.uk emma.duke-williams@port.ac.uk
http://userweb.port.ac.uk/~duke-
http://userweb.port.ac.uk/~crellinj/in wie/blog/
dexd.php
ABSTRACT
Simulation at different levels of fidelity has been used in education and training for many years. This paper
will look at the use of simulation in this area of Computer Science education, and reports on a number of
different simulation tools in use in the University of Portsmouth. Central to digital forensics education is a life
cycle that involves forensically safe seizure of digital evidence, imaging of digital devices, their investigation,
report writing, and acting as an expert witness in court. Several of these activities are suitable for simulation
based education. In teaching of digital forensics at the University of Portsmouth simulation of forensically safe
seizure has used simulation. This usually involves constructing a scenario where a running computer is
seized and imaged, which involves using an office like room, and providing a computer that has been suitably
prepared. Many universities now have 'forensic houses' which are simulated crime scenes, that are used to
support a variety of forensic disciplines, and these can provide a suitable context for seizure simulations. The
University of Portsmouth has operated a forensic house for about five years. Another area of simulation is in
court room work. Many digital forensic specialists will be required to give evidence in court at some stage of
their career. Again, a number of universities have set up simulated court rooms, and the University of
Portsmouth opened its crown court room simulation in February 2010. The simulated court room is a room
laid out as a crown court, with video recording facilities, and external features such as jury rooms, interview
rooms and facilities to support the giving of video evidence. This year the digital forensic students were
required to give evidence in this court as part of their assessment. Although the court was not fully simulated
(only roles of defence and prosecution council, judge, usher, and accused were represented) the experience
was reported by students as being quite intense.
The benefits brought to the unit by simulation were mainly in terms of increasing enjoyment and motivation
among the students. All the staff involved in the teaching were surprised by the almost ecstatic reaction of
many students on the unit to the simulation experience. The main disadvantages were the costs involved in
setting up simulations, which are quite high, especially for the seizure simulation. Also for individual
simulation the time involved was high. For example the court simulation required each student to give
evidence independently, and be cross examined, even with a small MSc class this took nearly three hours.
Seizure simulation involved team work, with only one student actually directly performing the seizure (under
direction from her peers).
In 2009-2010 we have investigated the use of virtual world simulations for digital seizure, constructing a
number of test examples (with the help of students studying an interaction design unit). These potentially
allow a low cost replaying of seizure scenarios, allowing individual students to experience different scenarios
on demand. A similar opportunity exists for familiarisation with court room procedure. We will evaluate these
low fidelity simulations with students in 2010-2011.
Keywords
simulation, digital forensics, teaching, virtual worlds
187
1. INTRODUCTION
1.1 Simulation in computer science teaching
Simulation forms a important part of many aspects of computer science teaching. It covers low fidelity
simulation (for example the specification for a programming task) to high fidelity simulation (for example
requirements capture from staff simulating business roles for system analysis). Simulation has two important
aspects. The first is that at least one important aspect of the target context is accurately mimicked, and
second that it is safe, the participant cannot suffer any significant damage (but may experience some of the
emotions or challenges of the real situation). A flight simulator recreates the changes in force as an aircraft
changes orientation and speed, the appearance of the ground from the cockpit, and the controls available in
the aircraft. But a flight simulator is safe. Although the simulated aircraft may crash, the physical forces
experienced by the participant will never be dangerous. The emotional experience may be intense.
Simulations form one important class of game (either in a computer simulation, or in a realistic role play)
because of the opportunity to experience intense emotions in a safe context.
This safe experience of dangerous contexts also allows for education and training. Flight simulators have
been used since the early twentieth century to introduce trainee pilots to the difficult aspects of flying without
putting the trainees at risk (Link Simulator, L-3 Communications, 2009). In more 'academic' domains,
simulations may be used in operating theatre, or intensive care ward practice (ExPERT Centre, 2008), and
virtual worlds have been used in computer science education in a number of ways (Crellin, Chandler, Duke-
Williams, & Collinson, 2009).
1.2 Digital forensics investigation lifecycle

Central to digital forensics education is a life cycle that involves four main steps. The first is the forensically
safe seizure of digital evidence. This means data stored on a variety of devices, typically computer hard
disks, or flash memory devices. Increasingly mobile devices are important for investigation. The core
principal of any forensic investigation is that the process of collecting evidence must not change the evidence
in any way. Where evidence is in a dynamic state (for example data on a running computer) the seizure
process is more complicated. The Association of Chief Police Officers (ACPO, undated) defined a process for
forensically safe seizure of evidence, and whilst this is becoming less appropriate as computer systems
evolve (particularly the increased use of disk encryption), it does provide a baseline for approaching the
problem.
Once collected devices must be examined and analysed. This usually involves looking for particular data on
the device. The data is often hidden, and novice investigators will need to undertake a number of
investigations of increasing complexity in order to acquire the necessary skills. The provenance of the
material found is as important as it presence. The investigation process is reported in detail, as well as the
findings.
A court report will detail the findings, and the process in a replicable form, since expert witnesses working for
the defence may wish to critically evaluate the investigation process.
Finally an expert witness will give evidence in court. In a Crown Court, evidence is given to the judge and
jury, but is elicited through questions from council for defence and prosecution. Hence the expert witness
cannot usually make a presentation to the judge and jury, but must rely on the questions of council to elicit the
key points of evidence. It is a strange and artificial process, and one that requires quick reactions to the
opportunities presented by the questions.
2. USE OF SIMULATION IN OUR TEACHING OF FORENSICS

2.1 Seizure simulations
A seizure simulation involves presenting students with a simulated environment which contains data devices
of one sort or another, with the challenge that these be collected in a forensically safe way. In practice digital
forensic specialists may not be present at the scene, however they may be called in to deal with evidence that
scene of crime officers cannot deal with. One example might be a running computer. A running computer
provides several challenges. Closing a computer down may inadvertently lead to loss of evidence (and the
computer may be designed to remove incriminating evidence under some circumstances). Closing a
computer down normally will produce a great deal of disk activity, with access and updates to many files. In
some cases a running computer may provide the best opportunity for accessing concealed or encrypted files.
A computer is also a communications device and a running computer may be connected to a site of interest to
the investigator.
188
In our simulation a room will contain a running PC. Normally some contextual evidence will also be available.
Students' task is to decide the best way to close down the computer, and dismantle it for evidence collection.
We usually allow one student to act as the evidence collection officer, and the others to direct, take notes and
discuss the courses of action available. We have typically included 'surprises' based on reports from our
professional advisors which usually add to students engagement with the simulation.
Students finally image the hard drive that they have extracted during the simulation, and this image is then
used during their assessment.
Setting up this type of scenario can take some time, usually getting a small room that is not obviously a
classroom can be difficult. A computer needs to be set up, a variety of additional elements need to be used to
'dress' the scenario. Various consumables such as latex gloves, evidence bags etc. are needed, and various
pieces of specialist equipment (write blockers etc.) have to be brought in. The costs are small, but even a
simple simulation needs an hour for preparation and an hour for dismantling.
Fig1: Seizure Scenario demonstrating team based work. With one student performing the majority of the
hands on work whilst others direct, discuss, comment and record the process.
2.2 Court room simulations

A second area of simulation is in court room work. Many digital forensic specialists will be required to give
evidence in court at some stage of their career. Again, a number of universities have set up simulated court
rooms, and the University of Portsmouth opened its crown court room simulation in February 2010. The
simulated court room is a room laid out as a crown court, with video recording facilities, and external features
such as jury rooms, interview rooms and facilities to support the giving of video evidence. This year the digital
forensic students were required to give evidence in this court as part of their assessment.
The simulation involved a mock court room, and ancillary rooms (interview rooms, a witness waiting room,
and a jury room) that has been built in the University, with several members of staff playing critical roles. The
simulation was of a Crown Court. A member of staff with some prior experience played the role of judge, the
two unit lecturers played the roles of defence and prosecution council, an unfortunate pair of research
students played the defendant and his guard (largely passive roles), finally another student played the role of
court usher, bring the forensics students playing expert witness to and from the court room. Each student
was allowed to stay in the public gallery to see the following expert witness. The mock court room has been
equipped for video recording, however the new equipment did not work correctly (having just been set up) and
so we didn't have the opportunity to use the footage in review with students.
189
The context, and simulated formality of the environment greatly affected the students, who found the
experience fairly intense. Although we had originally considered using strangers as council, having familiar
lecturers (albeit somewhat oddly dressed) was probably beneficial in moderating the stress of the simulation.
The setup of the court was fairly minimal as it is a ready dressed set. Only some additional props, oath cards,
information typically provided by courts to witnesses, were required. Running the simulation involved
repeatedly questioning and cross examining each student. The performance of students under cross
examination was rated by the two unit lecturers, and formed one element of the assessment mark. The court
report was also submitted for assessment. The main costs of the simulation were staff time, and the time and
availability of the people who played the additional roles.
2.3 Virtual world simulations

Using HCI undergraduate students a number of simulated environments were set up, using the virtual world
Second Life, which is relatively accessible (Linden Research Inc., 2009). These environments were
incomplete but did demonstrate a number of different approaches that could be employed. In such a virtual
world the behaviour of seized devices can be closely simulated, by feeding output from virtual machines
running outside the simulation. Conventional approaches to seizure (as described by ACPO) as well as
contemporary approaches (memory capture) can be attempted. The approach was discussed in (Crellin and
Karatzouni, 2009).
In 2009-2010 we have investigated the use of virtual world simulations for digital seizure, constructing a
number of test examples (with the help of students studying an interaction design unit). One example is
shown in figure 2. These potentially allow a low cost replaying of seizure scenarios, allowing individual
students to experience different scenarios on demand, and replaying them on demand, without a tutor
necessarily being present. We will evaluate these low fidelity simulations with students in 2010-2011, and
contrast the different levels of fidelity involved.
Fig2: One example of a virtual seizure simulation environment in Second Life. The environment includes
learning material in one bay, a simulation exercise in the second bay, and assessment exercises in the third
bay. As a stand alone environment students could use it at any time, presence of a tutor is optional.
2.4 The forensic house

Many universities now have a 'Forensic House' which are simulated crime scenes, that are used to support a
variety of forensic disciplines, and these can provide a suitable context for seizure simulations. The University
190
of Portsmouth has operated a forensic house for about five years. This environment will usually represent a
domestic context with evidence of a crime, for example blood stains, evidence of a fight etc.. The forensic
house can provide a good environment for running a seizure simulation, although digital forensic officers
would not often operate in the scene of crime.
3. IMPACT ON STUDENTS
The use of simulation was extended in 2009-10 with the first use of the mock court room. Contrasted with a
functionally similar assessment viva in 2008-9, the simulated court room appears to be much more engaging
for students. A seizure scenario had been used in 2008-9 successfully and the response of student in 2009-
10 was also very positive. The unit received the highest feedback ratings of any post-graduate unit, and
second highest of any unit taught by the School of Computing. Qualitative feedback suggested that the
seizure simulation had been the most appreciated part of the unit.
Student understanding may not be enhanced greatly by the use of simulation. Performance on the
assessments varied across the class, and some obvious points made in the simulation were missed by some
students.
4. SUMMARY AND DISCUSSION

Simulations do enhance student engagement with the unit. The benefits brought to the unit by simulation
were mainly in terms of increasing enjoyment and motivation among the students. All the staff involved in the
teaching were surprised by the almost ecstatic reaction of many students on the unit to the simulation
experience. Its less clear if they enhance cognitive skills, but they do help replicate some of the emotional
pressures involved in the subject area. The principle disadvantage of simulation is the cost of setting up the
environment. This includes equipment costs, which may be quite low (given that some items will usually be
little more than standard piece of equipment) but will include consumables. The larger cost is the time to set
up environments, even if a custom environment is available it usually still needs some dressing. Computers
need to be re-imaged, set up in the environment. Often quite heavy items moved from one location to
another. Several participants will need to play specific roles, especially for court room simulations. A degree
of acting is necessary. Time for running even fairly small scale simulations is quite long. The main
disadvantages were the costs involved in setting up simulations, which are quite high, especially for the
seizure simulation. Also for individual simulation the time involved was high. For example the court
simulation required each student to give evidence independently, and be cross examined, even with a small
MSc class this took nearly three hours.
Integrated simulations across faculty appear possible, and very interesting, but even more difficult to run.
They need to meet the varied assessment requirements of different disciplines, course timetables, and the
need to co-ordinate a larger number of students and classes. In such a cross faculty simulation law students
may take on court roles, and different forms of forensic evidence might be presented, for example physical
evidence collected in the forensic house, alongside digital forensic evidence, and forensic accountancy may
all be components in a simulated court case.
Using simulation in teaching can be very effective with students, and very rewarding for teachers, but one
should never underestimate how important staff enthusiasm is for a successful simulation.
5. REFERENCES
Association of Chief Police Officers of England (ACPO) (undated) Good Practice Guide for Computer Based
Evidence http://www.7safe.com/electronic_evidence/ACPO_guidelines_computer_evidence.pdf
(accessed 21-May-2010)
Crellin, J., Chandler, J., Duke-Williams, E. and Collinson, T. (2009) Virtual worlds in computing education,
Computer Science Education, Vol. 19, No. 4, December 2009, 315–334
Crellin, J., Karatzouni, S. (2009) Simulation in digital forensic education, at the 3rd International Conference
on Cybercrime Forensic Education and Training (CFET3) (BCS SIG) Conference, Canterbury Christ
Church University, 1st - 2nd September 2009.
L-3 Communications (2009) Link Simulation & Training: Setting the standard for 80 years
http://www.link.com/history.html (accessed 23-August-2009)
Linden Research Inc. (2009). Linden Lab. from http://lindenlab.com/ (Accessed 20-August-2009),
191
Reynolds, L, (2007) HEFCE Evaluation ExPERT Centre 2005-2007, from:
http://www.expert.port.ac.uk/assets/documents/WEBv%20Evaluation%202005-07.pdf (Accessed 15-
August-2009)
192
SHARING PRACTICE FOR COMPUTING EDUCATORS
Janet Finlay Sally Fincher

Leeds Metropolitan University University of Kent
Old School Board Canterbury
Calverley Street, Leeds, LS1 3ED Kent, CT2 7NZ
j.finlay@leedsmet.ac.uk s.a.fincher@cs.kent.ac.uk
http://leedsmet.academia.edu/janetfinlay http://www.cs.kent.ac.uk/people/staff/saf/index.html
ABSTRACT
Appropriate sharing of practice is essential both for educators to improve their own practices and for teaching
and learning projects to disseminate their outcomes effectively, but traditional dissemination is not well suited
to such sharing. This paper reports on a two-year process through which the Centre of Excellence in Learning
and Teaching Active Learning in Computing sought to capture and represent their practice in order to share it
effectively with the wider community. Two representations, patterns and bundles, are considered and the
team’s experience of using them is explored. Key findings are that the representation used must be adapted
to the community using it and that narrative is an essential element in the process of sharing practice.
Keywords
Sharing practice, representations, patterns, bundles, computing education, narrative
1. INTRODUCTION
Appropriate sharing of practice is essential both for educators to improve their own practices and for teaching
and learning projects to disseminate their outcomes effectively. However effective transfer of practice is
recognised as a complex process in higher education [3], which requires representations that will allow
educators to share practice across time and space. Disseminating practice solely through conference and
journal publications is not enough: this representation is suited to presenting objective findings but loses the
direct, experiential, “lived” elements that are critical to enabling other practitioners to adopt new practices [5].
Several projects (e.g. [3, 6, 7]) have explored what might make a useful representation for sharing teaching
practice but the question is still very much open.
The Centre of Excellence in Teaching and Learning Active Learning in Computing (CETL ALiC), the only
computing CETL in the UK, completed its funded work in 2010. Since 2005, CETL ALiC has been exploring a
range of areas of assessment, learning and teaching in Computing, including collaborative work, synoptic
assessment, using podcasting and web 2.0 technologies, peer support and problem-based learning. This
work has been presented through traditional means, such as conference and journal papers, but while these
are valuable as a means of disseminating the outcomes of the project, they do not offer an effective way to
allow other educators to evaluate and adapt practices developed by the project for their own context. The
project team therefore actively explored different means of representing such practice. This paper reports on
the processes undertaken by CETL ALiC to capture and represent the successful practices developed over
the five years of the project.
In the next section we situate this work in ALiC’s overall philosophy of sharing practice. We then introduce the
two representations, patterns and bundles, that have been explored, before discussing the process
undertaken by the ALiC team and its outcomes. We conclude with some observations and recommendations
on the process of representing practice and how effective sharing can be facilitated, based on our experience.
2. SHARING PRACTICE IN ALIC

Effective sharing of practice has been central to CETL ALiC’s philosophy throughout. In the initial phases of
the CETL this focused on transfer of practice between the four collaborating sites: the Universities of Durham,
Leeds, Newcastle and Leeds Metropolitan University. Specific cross-site activities, through which practice at
one site has been adopted at another, have been successfully implemented, focusing on project work and
synoptic assessment (e.g. [2, 8]). However, in order to share practice beyond the consortium, where it was
possible to work closely and share teaching experiences directly, ALiC needed to take a different
193
approach. From the outset, it was recognized that transfer of practice is a complex process and that traditional
dissemination methods, while valuable for sharing some outputs, were not the most appropriate means of
enabling transformation and tailoring of practice between practitioners.
However, deriving appropriate and effective representations of practice is not easy. Practitioners find it difficult
to identify and articulate the processes through which they actually assimilate new practice into their work.
Even where a piece of practice can be identified as successful and worth sharing, it is not trivial to identify the
essential elements of that practice that make it successful and valuable in a different context [6, 7].
ALiC therefore initiated a work package, which ran over two years, with two key aims: to evaluate a number of
representations of practice and to establish a process through which practice could shared appropriately
beyond the consortium. ALiC fellows worked together to represent their practice using two different
approaches. In each case fellows were asked to feedback on their experiences through a questionnaire and
focus groups.
3. REPRESENTATIONS OF PRACTICE
In seeking a representation appropriate for sharing its practice, ALiC identified a number of initial
requirements. Firstly, the representation needed to be aimed at educators and be easy both to produce and to
use in practice. More formal representations, such as Learning Designs, were therefore not selected as they
are recognised as being complex to use [3]. Secondly, the representation needed to be succinct and
accessible “at a glance”, focusing on a single piece of practice. For this reason, portfolio-based
representations were also rejected. Considering these constraints the ALiC team focused on two
representations for sharing its practice: patterns [1] and bundles [6]. Each of these uses a structured natural
language format, is short and concise and focuses on one piece of practice.
3.1 Patterns
The pattern representation was developed by architect Christopher Alexander in an attempt to describe good
practice in architectural design [1]. It has since been used in a number of other disciplines including
education.
A pattern is a structured, natural language representation that describes an effective solution to a recurrent
problem embedded in a specific context and is characterised by being drawn from successful practice rather
than based on theory. A pattern is generally considered to be such if there are at least three distinct examples
of practice where the given solution is successfully applied to the identified problem, the so-called “rule of
three” [7]. The pattern itself abstracts the invariant properties of those distinct solutions to provide a generally
applicable representation of the essence of the practice. The form is concise and intuitive to understand. An
example of a pattern form can be seen at http://www.patternlanguagenetwork.org.
3.2 Bundles
Bundles were influenced by patterns [4, 6] but recognise that there are certain things that teachers need to
know in making the decision to take on a new practice. They need to know what the practice is, why it works
and what pitfalls there may be in its implementation. They also need to know that the practice has worked
successfully elsewhere. Bundles therefore include a specific narrative of a particular piece of practice rather
than an abstraction across several examples.
The bundle representation consciously excludes information that may be less useful to practitioners. The
EPCoS project [6] found that teachers do not need details of the original context because they adapt practice
rather than adopting it unaltered [5]. Bundles also assume that ideas do not need to be packaged and labelled
in order to be reused: practitioners know their own context and what will work there and do not need the
originator of the practice to “second guess” this for them. An example of the bundle form can be seen at
http://www.cs.kent.ac.uk/national/EPCOS/bundles/bundles.html.
4. THE PROCESS OF SHARING

Bundles were the initial representation of practice considered by CETL ALiC in the early part of the project.
However, at the point at which we began to working on representing practice, we had the opportunity to work
with the JISC-funded Planet project team, who were developing a community-based process for capturing
patterns, so we began by exploring the use of patterns as a representation for ALiC practice.
4.1 Patterns and the Planet methodology

ALiC fellows from all four collaborating sites began to meet monthly to explore the details of their practice and
to consider how this could be represented to make it amenable to reuse. A process developed by the Planet
194
project was adopted, which used structured narrative stories to seed the process of exploring and capturing
practice.
A series of workshops was held monthly for about a year to share stories of practice and to try to identify
patterns. These workshops followed Planet’s Participatory Workshop methodology [7], which includes the
following iterative stages:
 Pre-workshop activity: participants submit a case story of their practice to the Planet wiki, using a
structure narrative template, the STARR template. STARR includes the following sections: Situation
(context), Task (what needed to be done), Action (what was done), Results (what happened), Reflection
(what was learned from it). Using a structured narrative has a number of advantages: it encourages a
common level of granularity in the stories, it allows comparison across several stories and it focuses on
the reality of practice rather than abstracted lessons.
 Workshop 1: participants share, question, elaborate and compare their narratives of practice, with a view
to identifying common elements which might be abstracted as patterns. “Candidate” patterns are identified
and entered into the Planet wiki.
 Inter-workshop activity: the Planet team reviews the narratives and candidate patterns to identify any
commonalities between these and those already in the wiki.
 Workshop 2: participants consider the “candidate” patterns from the previous workshop, and any
additional ones proposed by the Planet team, and use structured templates and prompts to refine them, in
particular focusing on identifying the necessary evidence for the candidate pattern to fulfil the “rule of
three”.
 Workshop 3: participants review the patterns and attempt to apply them to new problem scenarios around
the design and delivery of learning experiences. This helps to evaluate and validate the patterns.
The ALiC team primarily iterated around workshops 1 and 2, with some attempts to apply the patterns in
practice (workshop 3). However most of the workshop activity was around pattern production, so this will be
the focus here.
The structured storytelling activity was particularly successful. Participants reported that they had learned
more about the detail of each other’s work through this activity than they had through more traditional
dissemination approaches over the previous years of the project. Preparing narratives of practice and
discussing these helped to clarify the nature of that practice, including elements of significance and areas of
commonality. All participants rated these discussions as the key essential elements of the process. One
participant summarized it as follows:
“Talking about particular case-studies or practices (sometimes my own and sometimes others) was
really helpful in teasing out the similarities (or conversely, the lack of any similarities). … the value
was really in the discussion as it helped me to focus on the significant factors” (CETL ALiC workshop
participant).
However the process of abstracting from these stories of practice to propose patterns proved to be much
more difficult. Participants found the process of abstraction challenging and unfamiliar and some felt the
pattern structure was difficult to understand: “I found the concept of patterns quite difficult to grasp, I just didn’t
‘get it’” (CETL ALiC workshop participant). Also, it became clear over time that the nature of the practice being
described meant that there were often only one or two instances of the practice within the project, making it
very difficult to fulfil the rule of three and develop full patterns. This became frustrating for the team so we
decided to switch to the bundle representation to see if it could address these issues.
4.2 Bundles
The bundle representation focuses more directly on practice than patterns and requires only a single example
of practice. As such it was felt that it might help the team to address the issues of complexity of abstraction
and of having limited examples of practice. The team continued with the Planet process, retaining the
structured narrative element but using the bundle representational form rather than patterns.
This produced a surprising paradox. Even though one of the main issues with patterns had been the difficulty
of abstraction, when working with bundles, which are intended to reflect the detail of real practice, the team
found themselves over-abstracting. They did not initially relate naturally to the structure or fully appreciate the
purpose of each part of the bundle. They had become focused on abstraction and generality, more than their
stated purpose of communicating to an audience of practitioners. They believed that the more generic their
description, the more easily their practice could be transferred, and consequently omitted details that were
critical for sharing to be useful: “I think we got this feeling we had to make it as generic as possible so as
195
many people as possible would use it” (CETL ALiC workshop participant). In one case, there was no actual
practice at the heart of the representation, just a description of a simulation that had been created.
Pay it Forward

ALiC keywords project work, knowledge transfer

Students learn invaluable lessons that are then lost from one cohort to the next.
——o0O0o——
The level‐three undergraduate students undertaking the Project Management module are tasked with managing
a level‐two undergraduate team engaged in the Software Engineering team project. The level‐three students
will have experienced the team project in the previous year. The ‘project managers’ combine their past
experience of team work together with their current study in project management to support the level‐two
students in their team project. As part of the on‐going summative assessment of the level‐three students they
must record the problems they have encountered when managing the level‐two team, the solutions they
applied, and the results of their efforts.
The information is captured in a pattern i.e. in a formal structure used to capture a solution to a common
problem within a specific domain. Collectively the patterns from all level‐three students currently enrolled in
Project Management are then shared with their peers. The first year that this is done a pattern language for
team project management begins to emerge. The collection of patterns are then stored and then shared with
the following year’s Project Management cohort. This next cohort are then tasked with enhancing the collection
of patterns: by provided more examples to existing patterns; fine tuning existing patterns; and adding new
patterns. The pattern language is refined and increased each year thereby passing on lessons learned.
Lecturers are able to take patterns developed by students and integrate them into the course of lectures, which
helps student to see the value of their work as well how to improve it. The mechanism we put in place to
facilitate the capture and sharing of patterns was a blog, which was restricted to the level‐three Project
Management students and associated staff. However, we believe a wiki or asynchronous discussion forum
would be equally effective.
It only works if the patterns are reviewed to ensure the anonymity of contributors and filtered for wrong or
inappropriate content.
It doesn’t work if the captured knowledge is unstructured, voluminous and messy.
It works better if there is a process for refinement and structure when passing the patterns to the next cohort.
It works better if the tool provided to capture lessons learned is intuitive and accessible.
——o0O0o——
So if you want students to share knowledge across cohorts create a mechanism that will allow them to
share, refine, and increase the knowledge from one cohort to the next.
See also:
Hatch A., Burd L., Ashurst C., and Jessop A., (2007) Project management Patterns and the Research‐Teaching
Nexus, 8th Annual Conference of the Subject Centre for Information and Computer Science, University of
Southampton, 28th – 30th August 2007
Figure 1: An example of an ALiC Bundle using the new form
196
To address this issue the bundle form was revised in participation with the ALiC team to reflect their language
and understanding (see Figure 1 for an example of a bundle in the final form). Retaining the essential
elements of the original form, the new representation provided additional, contextualised guidance on what
should be included. The “problem” and “solution” statements were replaced by terms emergent from ALiC
practice: “rationale” and “essence”. The body of the bundle explicitly asked for “description”, making it clear
that this was the narrative of actual practice, rather than an abstraction away from it. This participatory
process of developing the form led the team to a new recognition of the importance of the particular rather
than the general in dissemination.
5. RESULTS
The new bundle form proved to be much more effective in supporting the team in representing their practice,
providing a representation that was meaningful to them: “writing the bundles helped me to capture what
worked and what didn’t in a much more succinct (and useful) manner” (CETL ALiC workshop participant). A
collection of around 30 bundles has been produced and will be published as a project output. The experience
of the team illustrates the need for representations to be meaningful to the community using them and
suggests that local adaptation can reap benefits.
The initial bundles have been shared with a second Centre for Excellence in Teaching and Learning, the
Institute for Enterprise at Leeds Met (http://www.leedsmet.ac.uk/enterprise), who were immediately able to
understand the practice being shared and, as a result, are investigating adopting the form to represent their
own practice. This is promising but further work is still needed to assess whether the bundles produced by the
ALiC team will prove to be meaningful and useful to practitioners beyond the consortium.
6. LESSONS LEARNED
The process undertaken by ALiC in order to share their practice has been informative and a number of
conclusions can be drawn. Firstly, it is important to avoid too much abstraction and generalisation in
representing practice. The detail of the practice is an important element in ensuring that the representation
remains meaningful. Secondly, representations should reflect the community that will use them and cannot be
imposed. It is important to be flexible in the form and language that is used, rather than insisting on a specific
representational approach. It may be that forms will need to be adapted to different disciplinary contexts, an
area that will be investigated further in our work with the Institute for Enterprise. Finally, sharing narratives of
practice proved to be a very effective way of transferring practice: story is important in understanding practice
and its significance should not be underestimated. It was this that was considered to be the most useful part
of the sharing process in ALiC and this was the element that made the bundle representation meaningful.
Indeed it could be argued that it is this narrative, an element that is often missing from formal dissemination,
which is the essence of effective sharing of practice.
7. REFERENCES
[1] Alexander, C., Ishikawa, S., Silverstein, M., Jacobson, M., Fiksdahl-King, I., & Angel, S. (1977) A pattern
language: towns, buildings, construction, New York: Oxford University Press.
[2] Devlin, M., Drummond, S. & Hatch, A. (2009) Using Collaborative Technology in CS Education to facilitate
Cross-Site Software Development Journal of Systemics, Cybernetics and Informatics, Vol.6, No.6, pp 1-6.
[3] Falconer, I., Beetham, H., Oliver, R., Lockyer, L., and Littlejohn, A. (2007) Mod4L Final Report:
Representing Learning Designs, available at http://mod4l.com/tiki-download_file.php?fileId=7
[4] Fincher, S. (1999), ‘Analysis of Design: an Exploration of Patterns and Pattern Languages for Pedagogy,’
Journal of Computers in Mathematics and Science Teaching, Vol.18, No. 3, pp.331-346.
[5] Fincher, S. (2000) From transfer to transformation: towards a framework for successful dissemination of
engineering education, in 30th ASEE/IEEE Frontiers in Education Conference.
[6] Fincher, S., Petre, M. and Clark, M. (2001), Computer science project work: Principles and pragmatics,
London: SpringerVerlag.
[7] Finlay, J., Gray, J., Falconer, I., Hensman, J., Mor, Y., and Warburton, S. (2009) Planet: pattern language
for web2.0 in learning, JISC project final report, available at http://www.patternlanguagenetwork.org
[8] Gorra A, J. Sheridan-Ross, P. Kyaw (2008). Synoptic Learning and Assessment: Case Studies and
Experiences. Proceedings of the 9th Annual Conference of the Subject Centre for Information and
Computer Sciences, Liverpool Hope University, Higher Education Academy, 26th to 28th of August 2008.
197
FACILITATING REUSE OF LEARNING RESOURCES: A TOOL TO
SUPPORT SELF-DEPOSIT AND AUTOMATIC METADATA
GENERATION
Dawn Wood Janet Finlay

Leeds Metropolitan University Leeds Metropolitan University
Old School Board Old School Board
Leeds, UK Leeds, UK
d.a.wood@leedsmet.ac.uk j.finlay@leedsmet.ac.uk
http://streamlinenews.wordpress.com/ http://streamlinenews.wordpress.com/
ABSTRACT
Learning objects are perceived as an essential element of technology enhanced learning. The idea of being
able to create, share, repurpose and re-share learning content across institutions is highly attractive for many.
In view of this, the JISC-funded project Streamline looked at how this process could be simplified and how
content creators could be motivated to describe and deposit their work. To facilitate this, staff development
workshops were observed to see how this process was currently being managed and how current tools were
fitting into the development workflow. From this, several issues were uncovered around the production of
metadata, which is essential to enable the process of searching and finding learning objects in order for them
to be reused. In response to these issues a prototype automatic metadata generator was developed and
tested. This paper presents the issues identified with metadata production, the prototype development and
results of an initial evaluation of this prototype.
Keywords
Metadata, Podcast, Learning Object, Repository, Workflows
1. INTRODUCTION
Reusable learning objects provide the opportunity to share learning resources both across levels [2] and
between institutions, offering benefits in reduced costs, enhanced quality and, ultimately, an improved student
experience. Initiatives such as the JISC Open Educational Resources (OER) programme [8] broaden the
potential impact of these resources, making them freely available to a wide audience. However one serious
barrier to widespread sharing of learning objects has been the lack of quality metadata provided by the
depositor, particularly where that depositor is not a professional cataloguer but is the producer of the material.
The time taken to enter quality metadata is seen by many potential contributors as a barrier to submission and
those that do deposit, provide little of the required information. The effectiveness of OER repositories is
seriously hampered by the poor metadata that results. The JISC-funded project, Streamline, aimed to address
this problem by developing an automatic metadata generation tool, to work with both text-based and multi-
media resources.
This paper presents the results from the design, development and evaluation of an automatic metadata
generator tool and is structured as follows: section two reviews the relevant background to this research;
section three identifies the issues raised by learning object creators; section four discusses the development
of the prototype tool; section five presents the results from the testing and evaluation of the prototype; the final
section, six, concludes this research and puts forward areas for further investigation.
2. BACKGROUND
Metadata is critical to the deposit of learning objects into repositories such as Jorum UK [9]. Based on the
IntraLibrary software [7], like most repositories it offers the depositor a web form for the completion of
metadata. In the case of IntraLibrary, the process of entering metadata at this point can be slow and
cumbersome as the object is not easily viewable during the process. Also, as the system is internet-based,
slow connections can have an impact on the time it takes to enter this information. Most repositories are also
able to extract metadata from packaged learning objects. Packaging tools such as a WimbaCreate [12] and
198
eCat [11] enable users to create or repurpose content, add the metadata and package all into a single zip-
style folder. However, although these tools do not suffer from internet connection problems, they do have their
own set of issues around interface usability and the quantity and quality of metadata required.
The manual process of metadata entry by creators has its obvious drawbacks. The alternative is for the
problem to be passed to professional cataloguers and library staff. This raises a whole set of other issues, not
least of which is resourcing. One other avenue remains open, that of automatic extraction of metadata from
relevant documentation. The AMeGA project report [5] highlights how metadata experts are divided on the
issue of automation, which sources are reliable and which fields can be automated. That project focused more
on Dublin Core [1] and Liberian requirements but many of the issues can be applied to learning objects.
There are very few examples of automatic metadata generators for learning objects. Repp et al. [10] have
devised a means of using presentations data to extract metadata for video lectures. Friesen [4] lists several
metadata editors, some of which were examined during this research. Of these only one has attempted to use
automation, and only in those areas that have already been identified by [5] as being non-intellectual and
context-free.
3. IDENTIFYING THE REQUIREMENTS

To understand the requirements we undertook a short study of how current learning object support tools were
being used across the institution. A series of workshops were being offered to staff to provide hands on
experience of using packaging tools and creating metadata. We observed a number of these workshops and
interviewed the tutor as part of the requirements gathering process. The tools being used included eCat [11]
and Course Genie, now known as WimbaCreate [12]. Both these tools work as a plug-in to MS Word,
enabling the creation of new learning content or the transfer of existing content into a web-format. From the
perspective of metadata, eCat enables the user to fully describe their learning object using the UK LOM [6].
Course Genie’s metadata is less comprehensive focusing on title and keywords.
The process of using these tools to create learning objects, while not intuitive, was generally well received by
workshop attendees. The tutor indicated that most of those attending the sessions found the final web-based
resource very good. The attendees were motivated to use the tools primarily for the transfer of existing
documents into web pages as they perceived this as being more appealing to students. However, when it
came to adding metadata, the tutor indicated that this topic was poorly understood and of little interest to the
attendees. Some attendees also had concerns about sharing their learning resources in an open repository.
This does not provide the motivation to spend time or effort on entering quality metadata.
The metadata entered during these sessions often consisted of phrases or repeated content such as the
document title. There was also a general misunderstanding about how the search process works. For
example, the phrase paper prototype is better entered as paper, prototype so that both words can be
searched individually. The presentation of the metadata fields by the eCat plug-in is over several individual
screens. The structure is the same as that used by LOM and has little help or guidance attached. Participants
found this process confusing and time-consuming when their focus was on the need to create more
interactive learning content for their students.
It is clear that the process of adding metadata needs to be simplified and better supported by creation
development tools. Keywords are of particular concern and learning object creators need to be encouraged to
produce a more extensive and comprehensive list than simply repeating the learning object title. Three basic
requirements were identified: improve the presentation of the LOM metadata standard to users; reduce the
amount of input time where feasible; and provide support for the generation of keywords.
4. PROTOTYPE
A prototype was designed to work directly with the institutions repository, IntraLibrary [7]. Users are able to
continue using their preferred packing tool and use the prototype to generate metadata. The XML file
produced can then be uploaded alongside the learning object into the repository. The three requirements
were met as follows.
To reduce the number of screens that the user is required to complete, the LOM metadata was grouped into
five sections. These are: Collaborators, including both content and metadata collaborators, which were
previously separate; Learning Object Details, this includes copyright information; Educational details,
remained the same as eCat; Technical details, again remained the same as eCat; Keywords and Categories,
originally presented individually. These groupings aimed to offer more logical data entry for the user.
Generally, collaborative effort to produce content occurs within faculties or departments between the same
individuals. Using this assumption the same content and metadata collaborators, if they exist at all, are likely
to be entered repeatedly for different learning objects. To reduce the amount of input time required, the
199
prototype stores collections of user details from which the user can select. In an ideal situation this information
could be generated from university systems, further reducing the input burden. In the case of this prototype, a
simple flat file storage system was used and loaded at startup. Users then select from the list, shown on the
right side of Figure 1, changing the role of the contributor if required, or add details of a new collaborator.
Figure 1 Using collections to reduce input burden

The same list of collaborators is available for use with both content and metadata fields as shown in Figure 1.
This method of creating collections is also used for the input of technical details, with the assumption that the
majority of users will be using the same methods and technical aspects for the creation of learning objects .
The keyword issue is more challenging and became the main focus of this development. From observing the
workshops and interviewing the tutor it was evident that there is a quality issue with text-based content. With
multi-media objects the problem is even greater and less input effort was applied by attendees. One possible
solution to this problem is to generate the keywords automatically using the content of the learning object.
However, in the case of multi-media content this is not possible, unless there are textual elements.
A B C
Figure 2 Keyword creation, editing and view panels

A review of the work processes involved in the production of both text and multimedia learning objects was
then undertaken. Staff who regularly produced learning content were asked to describe the processes and
resources they used. From this review several alternative sources of textual content were identified. In the
200
case of all learning content, course and module specifications were often referenced. In the case of
multimedia productions, many produced detailed scripts to guide the creative process.
To take advantage of the content in these documents, an extraction and analysis algorithm was developed.
The prototype presents panel A of Figure 2 to the user and they are able to enter these documents and label
them as to the type of content. The extraction process then presents the most relevant terms to the user in
panel B of Figure 2. Here the user can add new terms or remove terms from this list. The final panel in Figure
2, panel C, shows the currently complete metadata in human readable form, as opposed to the XML that is
output for upload to the repository. For a complete description of the metadata tool see [3].
5. EVALUATION
To evaluate the prototype two methods were used. First, the producers of a set of podcasts were asked to
describe their works using keywords. These terms were then compared to the list generated by the prototype
from the scripts associated with each podcast. The results are presented in 5.1. The second method was an
evaluation by users of the keywords produced for their learning objects from a variety of documents. This is
presented in 5.2.
5.1 HCI Podcasts keyword comparison

The podcast data was provided by a former HEA ICS project [2] and consisted of four individual episodes.
Two of the authors agreed to produce keywords for these and also supplied digital scripts that were used with
the prototype. Prior to comparison all the keywords were cleaned to remove duplicates and similar terms. This
process involved: splitting any phrases into individual terms; removal of small terms such as it and the;
removal of any duplicates; grouping of terms with the same root for examples prototype and prototyping would
be classed as part of the same group and replaced with term prototype.
Original Number of
Count after data
word/phrase prototype terms
cleaning
count matching authors
A-1 A-2 P A-1 A-2 P A-1 A-2 total

Episode 1 11 3 20 12 5 17 2 3 3
Episode 2 10 3 13 13 5 12 5 3 5
Episode 3 8 4 16 10 6 14 5 3 5
Episode 4 12 4 17 11 6 15 4 3 5
58 18
Figure 3 Keyword Comparison
Author one, the first column of the triplets in Figure 3, assigned a greater number of terms than author two, the
second column. This number of keywords is not usual for content creators. The number of terms assigned by
author two is on a par with previous observations. Figure 3 shows the original number of terms assigned
followed by the final count after data cleaning. The last set of columns show the number of terms generated
by the prototype that matched those of the individual participants and the number that matched those of any
participant. This shows that about 31% of predicted terms were assigned by one or more authors across all
the episodes.
5.2 Prototype evaluation

Three participants took part in a pilot evaluation of the tool with non-multimedia learning objects. This process
involved the users interacting with the prototype and uploading the resulting XML file and their learning object
into the institutional IntraLibrary repository. This process was screen-captured and a post-study questionnaire
presented for evaluation of the experience.
The results of the evaluation questionnaire showed that two participants found the collections function useful.
The process of generating the metadata was perceived as moderately easy across the group. All three
participants indicated that they would use the tool again and would recommend it to a colleague. There were
mixed feelings regarding accuracy of the keywords produced, but two participants indicated that they did find
the list of generated keywords useful. Figure 4 shows the comparison of user generated words to those
generated by the prototype. Prior to interacting with the prototype, participants were asked to assign keywords
to the learning objects they had brought for testing. The number suggested is shown in the first column. The
second column shows the number of generated words that matched those initially suggested by the
participants. The number of matched words was about 22% of the total number words assigned by the
201
participants. This is lower than those matched in the previous study. The middle column shows the number of
suggested keywords that were removed by participants. Column four shows the number added by participants
and the final number of keywords assigned to their learning objects is presented in the final column.
Final Number
Added by the
Removed by
of keywords
Generated =
Suggested
the user
User
user
user
SLP01 4 1 6 0 14
SLP02 8 3 0 4 24
SLP03 6 0 15 2 8
18 4 21 6
Figure 4 Keyword comparison for pilot group
6. CONCLUSIONS
This research is in its preliminary stages with the results from the two pilot evaluations showing some
promise. The prototype development met all the requirements stated in section 3. The comparison for
podcasts shows that a greater number of keywords were generated by the prototype than was generated by
either user, over 50% of which could be deemed useful in describing the content. The prototype also picked
out the obvious keyword for half learning objects, podcast, which neither user suggested. The second part of
the evaluation again shows a greater number of keywords being assigned to the learning objects than had
been previously observed. This suggests that the system does encourage an increase in keyword production.
However, given that all participants selected only a subset of these additional generated keywords, it is
important that such tools offer suggestions only and that the user is given ultimate control over which
keywords are relevant to include. Further evaluation is needed to fully assess the contribution of the prototype
to improving the quality of metadata provision.
7. REFERENCES
[1] Dublin Core, (2008), Dublin Core Metadata Initiative, Dublin Core, http://dublincore.org/documents/dces/,
(30/08/2008).
[2] Finlay, J., Soosay, M., Thomson, S., Chawawa, M., Moore, D., Renshaw, T. , Gorra, A., and Sheridan-
Ross, J. (2008), Video-based learning objects for teaching Human-Computer Interaction at different
levels, Proceedings of HEA-ICS conference, Liverpool Hope, pp. 196-200.
[3] Finlay, J., Gray, J. and Wood, D. (2009), Streamline, Integrating Repository Function with Work Practice:
Tools to Facilitate Personal E-administration, Final Report. Available at
http://streamlinenews.wordpress.com/ (18/5/10)
[4] Friesen, N., (2005), Learning object Metadata Editors, Cancore,
http://cancore.athabascau.ca/editors.html, (18/05/2010).
[5] Greenberg, J., Spurgin, K., and Crystal, A., (2005), Final Report for the AMeGA (Automatic Metadata
Generation Applications) Project.
[6] IEEE, (2010), Learning Technology Standards Committee, IEEE, http://ieeeltsc.org/ (30/08/2008).
[7] Intrallect, (2010), IntraLibrary, Intrallect, http://www.intrallect.com/ (18/05/2010)
[8] JISC, (2009), Open Education Resources programme, JISC,
http://www.jisc.ac.uk/whatwedo/programmes/elearning/oer.aspx, (18/05/2010).
[9] Jorum (2010) Jorum UK, Jorum, http://www.jorum.ac.uk/, (18/05/2010).
[10] Repp, S., Linckels, S. and Meinel, C., (2007), Towards to an Automatic Semantic Annotation for
Multimedia Learning Objects. Proceedings of EMME’07 conference, Augsburg, Germany, pp. 19-26.
[11] Soosay, M. (2007), Beginners’ guide to eCat, Leedsmet, www.leedsmet.ac.uk/inn/streamline/.../e-cat-
user-guide-streamline_version.pdf, (30/08/2007).
[12] Wimba, (2010) WimbaCreate Create course content, Wimba,
http://www.wimba.com/products/wimba_create/, (18/05/2010).
202
ENHANCING INTERNATIONAL STUDENT ENGAGEMENT THROUGH
MAKING AND SHARING FILMS
Andrew Cox Nashrawan Taha

Department of Information Studies Department of Information Studies
University of Sheffield University of Sheffield
Sheffield Sheffield
a.m.cox@sheffield.ac.uk lip07nnt@sheffield.ac.uk
http://www.sheffield.ac.uk/is/staff/cox.html
ABSTRACT
Supporting international students to adjust to the requirements of studying in British universities and also to
bring their diverse knowledge and perspectives actively into the classroom is the challenge of
internationalisation. Film making offers a promising route to both engage students through an intense group
work task, reduce emphasis on English language skills and create reusable content articulating issues in new
ways. A project redesigning an information society module with film making as part of the assessment is
explained, focussing on how the design offers strategies of hospitality, empowerment and criticality to engage
internationalised students. Outcomes that made further issues around internationalisation more concrete are
discussed. Future plans include running two parallel classes to promote inter-professional and inter-cultural
learning.
Keywords
Internationalisation, student engagement, film making, assessment methods
1. INTRODUCTION
Although the primary impetus for British universities seeking international students is economic (de Vita &
Case, 2003), their presence in the classroom opens up exciting opportunities to diversify, indeed transform
the curriculum (Morey 2000). This paper reports on a curriculum development project, supported by a
University of Sheffield Learning and Teaching grant, to realize the potential of internationalisation in a post
graduate information systems module with students from 12 different countries. Central to this process was
moving to assessment of students partly through film making, rather than relying on traditional assessment
tasks such as exams and essay writing. The student groups were also required to share their film online,
seeking to get substantial input on their topic from wider communities beyond the classroom. The project thus
explored the potential for film making both to engage international students more fully, help them adjust to
British university life and to effectively enrich the curriculum.
The paper starts by exploring the literature around the concept of internationalisation, seeking to establish the
complexity of this concept and the differing responses authors have suggested to engaging international
students. The potential of a film making or digital storytelling assignment to address some of these issues is
established from the literature. The curriculum redesign project that arose from this analysis is then described,
with an in-depth discussion of how the curriculum design relates to international student engagement. The
project seems to have been a success, and evidence for this is offered. The discussion explores further the
internationalisation issue, especially in the disciplinary context, through reflecting on the outcomes of the
project. The paper concludes by looking at particular areas where future development work will be focussed.
2. CONTEXT
2.1 Internationalisation and engagement
It is commonly understood that international students have a number of issues of “adjustment” to British
university life, in addition to those experienced by any student entering university (eg Zhou & Todman 2008).
One set of barriers is the culture shock of being in an unfamiliar country and language barriers, including
203
understanding regional accents. There is also academic culture shock, in terms of different expectations
about academic work (such as being analytic, synthetic and critical), about what types of thing to read,
definitions of plagiarism, differing marking schemes, expectations about teaching style. Such factors operate
in addition to obstacles to engagement such as the “mass” character of higher education institutions, as well
as the many practical issues that all students face such as finding accommodation and financial problems.
International students may have particular issues around making friends and maintaining networks of
information and support, far from home.
It is easy to fall into simplistic, “large culture” stereotypes (Clark & Gieve 2006) about the problems
international students may encounter, and it is argued that opening up a debate between the tutor and
students in a class about past learning experience and current expectations is the most effective way to come
to a common understanding about how learning will happen (Skelton 2005). This is unlikely to be easy since it
is a discomforting debate about our basic values and daily habits (Boler and Zemblyas 2003). A mass system
has problems personalizing the educational experience. But there is a need for adjustment, both on the side
of the institution as well as the student.
Further, much of the educational literature argues for a more transformative form of internationalisation,
recognising the profound benefits and opportunities in terms of bringing new perspectives and knowledge into
the classroom, and countering “hegemonic” knowledge (Schoorman 2000). Potentially the richness of learning
on UK HE courses could also be enhanced through encounters with other cultures, languages and
perspectives. Some outcomes are defined by Leask (2004) such as the ability to: Think globally and consider
issues from multiple points of view; Recognise inter-cultural issues relevant to professional practice;
Appreciate the complex and interacting factors that contribute to notions of culture and cultural relationships;
Value diversity of language and culture. The argument for transformation calls for a decentering of the
teacher’s power and recognizing students as co-creators of knowledge, in a way familiar to educationalists
from critical pedagogy. Yet this is a critical tradition that is itself rather unfamiliar, possibly unattractive (Choo
2007), to many international students themselves. This might be particularly the case in the field of information
and computing science, where international students often come with expectations of learning about “more
advanced” systems in more advanced countries, concepts of progress which a critical approach may itself
query.
Thus there is a complex set of arguments about the character of internationalisation and different approaches
to be taken. In analyzing the approach taken here we follow Mann’s (2001) typology of approaches for
countering structural causes of student dis-engagement, through strategies of solidarity (empathy), hospitality,
safety (providing safe spaces to learn), empowerment and criticality.
2.2 Digital storytelling and film making

Techniques such as digital storytelling and film making have been advocated as potentially enhancing student
engagement, as well as offering new skills for employability and reflection (Ivers & Barron 2006, Sadik 2008).
Film making as an exercise seems to offers some affordances that could address international student
engagement, in particular. By stressing communication through images and sounds as well as text, the
international students’ potential disadvantage in terms of command of English is lessened. Visual literacies
may equally be culturally specific; we cannot assume equal understanding of media conventions,
nevertheless by recognising other skills than writing the exercise may be seen to enhance the range of skills
being assessed making it generally fairer. By setting a non-standard assessment challenge, issues around
academic standards and criteria can be immediately opened up for discussion. Sharing and discussing the
final content, gives a further opportunity for discussing evaluation criteria. All these features address
“academic culture shock”. As a rather intense group work exercise, students who have just come to the UK
can be given an opportunity to work closely with other students, reducing the risk of isolation and creating a
context for talking closely with each other about all aspects of learning. Furthermore the films that have been
made constitute a resource for future teaching that effectively decentre the academic’s voice potentially
transforming the curriculum with novel perspectives.
With these considerations in mind the authors developed a project to introduce film making into one module,
which was funded by University of Sheffield learning and teaching development grant system.
3. THE PROJECT
The unit being redesigned was a core module in Semester one for taught postgraduate studying Information
Systems and Information Systems Management. It is also optional for some students on other programmes
and some Department of Journalism students. During the project described here, there were 40 students on
the course, representing 12 countries, with a rather even spread between China, the Indian sub-continent and
204
the Middle East. The module covers issues in the information society. The following detailed discussion of the
module design, is developed in relation to addressing of the needs of international students.
The module consisted of two strands: lectures and practicals. The practical sessions began by focussing on
statistics about UK internet use, based on the latest Oxford Internet Survey (Dutton et al. 2009). This was
seen as a strategy of hospitality (Mann 2001) towards students offering them an insight into British life, via a
topic of interest to them. Students were also shown a short film made by the project team which explored
stories of Internet use all from Sheffield, with some undergraduates talking about facebook, some Flickr users
discussing their involvement in the Sheffield group, interviews in a library about social inclusion through the
people’s network and a retired woman talking about her internet use. This was both again telling students
something about Britain and offering a model for their own film. Later practicals and lectures opened up the
perspective from the UK, to think about contrasting uses of the Internet globally. Clearly the topic of
comparative global internet use invited students to bring in their own experiences, and to open up the benefits
that international students offer.
The lectures established a theoretical base for the content of the film, reviewing literature around three foci:
namely, web2.0 and the concept of community on the Internet; the digital divide concept and issues around
censorship and surveillance. The topics themselves invited “counter hegemonic”, critical treatment, e.g. in
exploring the literature which shows that the Western gold standard of internet access, broadband, is
inappropriate in developing countries. Three research students gave guest lectures on Internet access in their
own countries (the Philippines, Jordan and Mexico), again reinforcing the message that non UK perspectives
were to be valued. Critical viewpoints on surveillance in Britain were given priority over Internet censorship
issues, which might have led to the poor track records of certain students’ countries to feel victimized. So
there was an attempt to explicitly open up critical perspectives and class room debate, where students were
encouraged to explore their own experiences and not to privilege “Western” perspectives. Potentially at least
critical perspectives could be helpful in encouraging students to engage (Mann 2000). By looking at the global
variety of experience of the Internet, technocentric or technological deterministic accounts (often quite
prevalent in student thinking) were implicitly undercut. Castells (2001) work was used to establish this
perspective more theoretically.
Later practicals covered a justification for the group work exercise; discussion about group organization and
roles; storyboarding and film planning; and then technical skills around video camera operation and editing. It
was stressed that it was the ideas in the film rather than technical quality that would be evaluated. Explicit
discussions attempted to come to a consensus around evaluation criteria for the films. The original remit
below was even renegotiated to allow slightly longer films.
“Working as a group of four produce a multimedia project (e.g. video) reflecting on similarities and differences
between two countries in their Internet use, focusing on one of the Information Society key themes discussed in
the module: Web2.0/community, digital divides and surveillance/ censorship. The multimedia should be a
maximum of 3 minutes long. Imagine a multi-national, multi-cultural audience.”
By asking students to compare national experiences and because of the audience to be imagined, issues
around diversity and valuing different experiences were actively promoted. Group membership was defined by
the lecturer, principally on the basis of mixing different nationalities, again to promote international working.
In addition to making the film, students also had to create a group bibliography and write a short individual
essay about the theoretical area addressed by the films, they were also asked to post their film on the open
Internet, and were assessed on their strategy to gain useful comments. We thought that the exercise would
give students a deeper insight into the ways needed to operate and risks of Web2.0. thus it was an
opportunity to think about how to link to wider communities to gain content, but the need to manage privacy,
the risk of vandalism or negative comments. The idea here was also to explore the value of opening up the
class room to networks beyond it. This could affect engagement, because students knew they were going to
share their work with friends and family, so were keen to use it to show off how their studies were
progressing.
4. EVALUATION
4.1 Engagement and benefits of the film making assignment
The quality of the films produced speaks in itself for the impact on engagement
(http://www.good.group.shef.ac.uk/wiki/index.php/Opened_up_learning:_generating_content_and_learning_o
pportunities_with_communities_beyond_the_classroom). Anecdotal evidence suggests that students spent a
lot of time on the filming work. A long evaluation questionnaire was completed by 39 out of the 41 students
attending the class. On nearly every count the feedback was positive. This was doubtless influenced by the
added excitement around an explicitly innovative class and also the effect of having a research student
205
studying the class (Taha & Cox 2010). Most students agreed that they had learned film making skills, about
group work skills and about the substantive issues around the Information Society. There were indications
that attendance improved.
The benefits to the students of the film making and sharing assignment could be summarized as follows. They
had gained experience of film making. These themselves are arguably increasingly relevant skills, given the
increasing use of films and multimedia to explain and promote information services in accessible ways, and
could directly effect employability. Students had gained experience of a complex group work task. Because
we had asked for the films to be shared on the open Internet, students had to address copyright issues. Thus
there was much classroom discussion was around IPR, and student feedback indicated that this was a key
area of learning, one highly relevant to these students as information professionals. In the long run student
generated teaching material is a valuable way to diversify and enhance teaching in the module.
4.2 Further issues

The project made concrete some of the conceptual challenges around internationalisation. This section
therefore develops a discussion around a key aspect that emerged during the project, as well describing
some more practical issues.
Asking students to make a film could produce work in many different genres, familiar from television,
documentary making or even specific to a particular discipline or area of practice. Thus Fletcher and Cambre
(2009) write about drawing on traditions native to the discipline, in their case ethnographic film making.
Information Studies does not really have such native genres. During the practicals a variety of models were
offered, including a comic film explaining open access using puppets
(http://www.youtube.com/user/llordllama), as well as more obvious styles, such as the model of interviews in
our own film. We wanted the film making task to be quite open ended and invite unexpected and creative
approaches. In reality looking at the final films the influence of our own model and of some of the practical
work using statistics is very evident. Most films mixed some interviews, usually with students and
representations of basic statistics of net access in the countries explored (a focus of early practical work).
Most films were about the digital divide, exploring how it differs in different countries. Some films were a little
different, e.g. one film introduced the concept of accessibility based on interviews. Theoretical engagement
and criticality were not very evident, though the essays brought this out better. The introduction to the film was
using the JAWS screen reader, and part of the film was looking over the shoulder of a blind person accessing
e-journals using the same reader. There was some excellent use of graphic software. But stylistically they
were rather similar. It is clear therefore, that the models offered to students need to be carefully chosen,
particularly if one wishes to encourage the production of “counter hegemonic” viewpoints. There is room to
expand the range of models offered, e.g. by giving storytelling models, which would invite more personal
responses.
Potentially film making opens up some interesting questions about the nature of knowledge in the discipline.
Because a film privileges non textual, visual, audio perhaps emotive representation, this asks some questions
about the nature of knowledge we teach. The student films created this time did not go far in this direction.
On the whole, student success in gaining substantive feedback on their films through sharing them online was
a little disappointing. They got a lot of positive feedback on the quality of their films, but little response to the
content. Most took the simple strategy of surfacing their film on youtube and facebook, without thinking about
specialist online communities who might have had a more in-depth interest in the content they had created.
Nor did they perceive the need to build up relations with others online (through commenting on the work of
others) before trying to get comments on their own work. We may need to connect to specific classes in our
own department or elsewhere in the university or other universities, to realize these benefits effectively, or
certainly offer clearer models of how to get comments.
On a more practical level resourcing is clearly an issue. The creation of our own example film was very time
consuming. The project allowed the university to buy cameras, but there were only ten, thus effectively
determining group size. Explaining camera usage, editing, storyboarding, copyright and group work was a lot
of effort on top of the intellectual content of the module. It was difficult to find time for all these topics to be
sufficiently covered. The learning and teaching (LeTS) team offered input on technical aspects of filming and
editing. A member of library staff gave input on copyright. Nevertheless, it took a lot of effort on behalf of the
lecturer, e.g. in supporting storyboarding techniques. The final assessment was rather complex, and
demanded a lot of work by the lecturer. There were potential issues around accessibility, in that we did not
have any visually impaired students, but if we had we would have found it difficult to easily enable them to
fully participate. Arguably the student films needed subtitles or transcripts.
206
5. REFLECTIONS AND FUTURE PLANS
Reviewing the success of the module redesign after the first year of its presentation, the film making exercise
seems to have been demonstrated to be an effective, exciting and challenging learning experience. It will be
particularly interesting to have the students’ films as a resource for teaching the module next year. This
should effectively engage students through giving them a greater sense of power over the learning process
(Mann 2000). As indicated above the range of models offered to students could be effectively expanded.
Although there are issues of scale and time, having more direct discussion about students’ previous learning
experiences as a way into discussing how the class works would be beneficial both for students and lecturers
in building common understanding. More discussion about how the groups should function would also have
been useful. This would have ensured that the groups offer the feeling of safety suggested by Mann (2000).
We are currently planning an exercise in which MA librarianship students, who are predominantly British, will
work in parallel with the Information Systems students on a similar but lower effort digital storytelling project.
Groups from both classes will be loosely twinned, so that they view each other’s work in progress. All
students’ work will be shared at the end of the course. This could create richer support networks across the
programme groups and allow librarian students to share their knowledge of the UK academic system. Equally
it should help MA librarian students see the big picture of information values in a global context. Both sets of
students can further develop their skills in multi-cultural working, but because they have rather different
professional identities, also gain valuable experience of inter-professional understanding and collaboration.
6. REFERENCES
Boler, M. & Zemblyas, M. (2003). Discomforting truths: the emotional terrain of understanding difference. In
Tryfonas P. (Eds.), Pedagogies of Difference: Rethinking Education for Social Change (pp.110–136). New
York: Routledge Falmer.
Castells, M. (2001) The Internet Galaxy, Oxford, Oxford University Press.
Choo, K.L. (2007) the implications of introducing critical management education to Chinese students studying
in UK business schools: some empirical evidence, Journal of Further and Higher Education, 31 (2), pp 145-
158.
Clark, R. & Gieve, S.N. (2006) On the discursive construction of ‘The Chinese Learner’, Language, culture
and curriculum, 19 (1), pp 54-73.
De Vita, G., & Case, P. (2003). Rethinking the internationalisation agenda in UK higher education. Journal of
Further and Higher Education, 27(4), pp383-398.
Dutton, W, Helsper, E. & Gerber, M. (2009) The Internet in Britain 2009, Oxford Internet Institute, [Online]
http://www.oii.ox.ac.uk/research/oxis/OxIS2009_Report.pdf [Accessed 9 May 2010].
Fletcher, C. & Cambre, C. (2009) Digital storytelling and implicated scholarship in the classroom, Journal of
Canadian Studies, 43 (1), pp 109-130.
Ivers, K.S. & Barron, A.E. (2006) Multimedia projects in education, third edition, London, Libraries unlimited.
Leask, B. (2004). Internationalisation outcomes for all students using information and communication
technologies (ICTs). Journal of Studies in International Education, 8 (4), pp336-351.
Mann, S. (2001) Alternative perspectives on the student experience: alienation and engagement, Studies in
Higher Education, 26 (1), pp7-19.
Morey, A. I. (2000) “Changing higher education curricula for a global and multicultural world‟, Higher
education in Europe, 25 (1), pp25-39.
Sadik, A. (2008) Digital storytelling: a meaningful technology-integrated approach for engaged student
learning, Education Technology Research and Development, 56 (4), pp487-506
Schoorman, D. (2000) What do we really mean by ‘Internationalization’? Contemporary Education, 71 (4), pp
5-11.
Skelton, A. (2005) Internationalization and intercultural learning, In A. Skelton, A. (Eds.), Understanding
teaching excellence in Higher Education (102-115). Oxford: Routledge.
Taha, N. & Cox, A.M. (2010) Social network dynamics in international students’ learning, Networked learning
conference, Aalborg, Denmark, May.
Zhou, Y. & Todman, J. (2008) Patterns of adaptation of Chinese postgraduate students in the United
Kingdom, Journal of Studies in International Education, 13 (4), pp 467-486.
207
YOU CAN’T SOLVE A JIGSAW WITH ONLY ONE PIECE – WHY
CROSS-TOPIC LTA IS KEY TO STUDENT ENGAGEMENT
Siobhan Devlin
University of Sunderland
Dept of Computing, Engineering &
Technology
St Peter’s Campus, Sunderland
siobhan.devlin@sunderland.ac.uk
http://osiris.sunderland.ac.uk/~cs0sdv
ABSTRACT
The paper describes a new approach taken in designing, delivering and assessing a level one curriculum in
computing. It focuses on the major part of our level one provision: a 100 credit module that is common to all
programmes in order to equip students with the fundamental knowledge and skills to progress to any level 2
computing degree. The paper begins from the viewpoint that the modular curriculum leads to a
compartmentalising of learning that hinders many students from properly contextualising their subject and that
by, in effect, demodularising what we teach at level one we may engage the students better as they see and
understand the relevance and impact of the subtopics that comprise the larger computing picture. Importantly,
this demodularisation has been introduced not only into the structure and content of the curriculum, but also to
its delivery and assessment. As well as teaching ‘cross-topic’ we assess ‘cross-topic’, with every assessment
becoming a ‘mini project’ – a complete jigsaw puzzle to solve rather than a single piece to puzzle over. This is
our first year of delivery, and this paper presents our rationale, discussions of the making and initial roll-out of
the course, and how both our students and staff have responded to it.
Keywords
Curriculum design, demodularisation, first year experience, student engagement, assessment.
1. INTRODUCTION
Following a recent review of our undergraduate provision in computing, we took the decision to make bold
changes to the structure, content, staffing, learning, teaching and assessment (LTA) of our first year.
Preliminary ideas were drawn up and posted to an all staff blog for comment. Feedback was collated and
analysed, and a small team was put together to further develop the course. What emerged was a largely
common first year (100 credits) with a 20 credit flavour module representing the specific chosen course of
study. This paper presents our motivations for developing such a structure and gives an overview of our
experience in running it for the first time.
2. MOTIVATION
The problems with modularisation are well known (Gass et al, 2004; Goodhew, 2002; Rust, 2000) and were
recently well articulated in a Readers’ Comments contribution in the Times Higher (Don Quixote, 2009): “I find
a problem with modularity in that it's very, very difficult for academics to actually know the content of all
modules. Consequently, sometimes modules overlap and students complain about doing the same old stuff,
sometimes it can turn out that some basic prerequisite knowledge (I know, it's all supposed to be defined in
pathways, but...) is missing. What does seem to come out of it is increased workload for students, increased
assessment loads, and the left hand doesn't know what the right hand is doing. Further, at the end of each
stage, it's actually difficult to fully grasp how the student's experiences stack up. I seem to notice that students
seem to approach learning in a modular way as well, so that knowledge gained from one module is not
deployed to solve a problem in another. So, in a sense, they've taken a large number of mini-courses in a
topic area, but these may add up to less than the sum of the parts”. In Sunderland, further factors motivated
our desire to make radical changes: current day students often seem to be attracted to computing because
208
they have grown up with games, mobiles, social networking sites, music downloading etc., meaning that they
see the bright and sparkly end-products and connect those with exciting career prospects, but they have little
concept of and patience for the hard work behind the scenes. Where the modular system meant these
students took a diet of modules, each delivered and assessed in a distinct and discrete manner, this
oftentimes left them wondering at the value or relevance of certain topics on their degree, or the connection
between these elements, and becoming demotivated with their studies. Why, they’d ask, do I need to know
the binary number system, or 48 ways to elicit client requirements, when all I want to do is build first-person-
shooter games? Conversely, themed programmes (for example Multimedia Computing, Forensic Computing
as opposed to straight Computer Science) could potentially limit learning to designated areas, omitting
valuable fundamentals to fit in modules conducive to the course’s theme. Thus programme ownership of
modules was another area we wished to address.
3. DESIGNING THE COURSE

3.1 Structure and Content
We made a common 100 credit module of core fundamentals which we believe are necessary for progression
to year 2 of any computing degree. This has the added attraction to students that they may change course at
the end of the first year. The content of the course would not surprise any academic in any computing
department in the country and in fact is not different to what it would have been had we revalidated our
programmes with the same old modular scheme. However, the way it is structured – the way our students
perceive the content through our manner of delivery and assessment - is key to how we are attempting to
better engage them.
3.2 Learning, Teaching and Assessment

The module is delivered by a small team of eleven staff, who teach across the topics thereby ensuring
relevance and continuity of interaction. These staff had to be chosen carefully for a number of reasons: they
needed to be able to teach a diversity of topics; several of them would be personal tutors to the students as
well as academic tutors; and the commitment to the first year course would be time consuming and potentially
take them away from other academic activities. This approach allows us to have a much closer relationship
with the students than previously and to keep a keener eye on engagement and retention.
In order to address the issue of students failing to grasp the relevance of everything they study, we decided to
introduce context sessions. This weekly tutorial with the module leader is the lynchpin of the module and
provides for tying together the learning, teaching and assessment, and the reflection and recording of
achievement through the students’ e-portfolios. All of our assessments are cross-topic, in order to underline
the importance we attach in the students’ learning of the relevance of all the topics we teach. This is
problematic to achieve when topics are taught and assessed discretely. We use the context sessions to issue
and discuss assessments, and to return assessments and feedback, again emphasising the cross-topic
importance, because if an assessment is issued by the tutor who teaches a specific topic then it is seen as a
discrete assessment in that topic only i.e. the learning is compartmentalised.
In addition, the students receive two matrices detailing the teaching and assessment on the module. These
enable them to see where the topics interlink and how the learning outcomes of the module apply across the
different topics. Students often complain of topics such as Software Engineering being dry or boring, because
they want to be doing the hands-on development all the time. The cross topic approach allows us to relate
topics like this to everything practical they are doing. Every assessment becomes a ‘mini project’ – a complete
jigsaw puzzle to solve rather than a single piece to puzzle over.
Our approach also recognises the importance of scaling the risk in assessment. In the first term there are 4
portfolio pieces which are very low stake at just 5% each. This allows for the students to ‘find their feet’ while
actively contributing to their portfolios. In term 2, the four assessed pieces rise to a worth of 10% each,
reflecting the students’ developing skill set and confidence. The year ends with a showcase project (worth
40%) which consolidates their learning from the whole year and allows them to specialise and contextualise
this to their specific programme of study.
4. EVALUATION
The academic year is still in progress and so at the time of writing a complete evaluation is not feasible.
However, regular feedback has been taken throughout the year both formally and informally, and a discussion
of some of this, where it links directly to our initial motivations for change, is possible.
209
4.1 Contextualising the year
As indicated previously, the context session is seen as vital to the smooth integration of the students’ learning.
We asked the students “How useful has the context session been during the year? Can you suggest ways in
which it could be improved?” While we anticipated they might say they found them useful primarily as a
source of assignment information, their responses were much more expansive and positive. Their comments
included:
• The context sessions seemed to have brought the course together, both students and work related
• Have more than one context session per week.
• Another fantastic idea as it keeps the path through the year clearly marked.
• Has given me more information about assignments so it has been very useful.
4.2 Cross-topic Assessment

We are confident that the cross-topic assessment is forcing the students to think more carefully and more
roundly about the assignments, but in a hectic first year of running this course we do feel that we have not
been altogether successful in always making an assessment regime that is as balanced and cross-topic as
we would like. We posed this question to the students: “Our aim was to offer you assessments that brought
together multiple strands of the computing curriculum. How far do you think we have achieved this?” Again,
the answers that the students gave us were actually much more encouraging than we had anticipated:
• 70% achieved;
• Very successfully – I can see how I have to pool together my knowledge from all strands to achieve
better marks;
• Very well – each portfolio covered a number of strands and brought them together nicely;
• While the early portfolios claimed to be cross topic they did not necessarily feel like that. This has
changed on the later portfolios;
• Pretty good, some links more obvious than others, weighted too heavily on programming and not
enough on others.
4.3 Scaled Risk in Assessment

The regime of having low risk assessment initially, gradually building to a high worth project at the end of the
year has been well received, but there is a small issue with some students feeling they should only put in as
much effort as befits the amount of marks they’ll receive. So, the responses to our question on this scaled
importance included:
• Good – because at the beginning everything was going over my head and in the second term it
clicked a lot more.
• I really like this idea. Eased us in.
• interesting way to build the year with little fear of messing up at the start.
• I think it’s good as we can gain skills and rectify weaknesses
• Some assignments were low % but were hard.
• Seemed people put more effort in when more marks were available
4.4 Common Course and Varied Skills Development

We do feel that the students have been really pushed on the course this year and that this has had postitive
effects. Clearly there are always going to be some students who don’t like certain subjects and who intend to
drop such subjects at the first opportunity, but they have in fact seen the value of studying them at least at a
fundamental level. Moreover, there is great camaraderie on the first year with all the students readily mixing
with each other regardless of programme. The peer support is far better than we have seen previously without
any formal mechanism being put in place, and this is an important factor when there are students studying
topics they would not normally choose and with which they might struggle.
One of our motivations in developing the course concerned the fact that many students think they know it all
when they arrive at university, and that this is sometimes a barrier to engagement. When we asked them
210
“How knowledgeable about computing concepts did you think you were at the start of the year – and has this
view changed as the year has gone on?” they responded:
• I thought I was fairly clued up but soon realised that I knew a lot less than I thought!
• Think I had a basic view but it has changed hugely through the year;
• Yes (my view has changed) – I feel that all areas of computing are linked;
• I thought I was around intermediate. I know I feel I know more but I’m aware there is much to learn.
Finally, the students are very positive about the ability to change programme because of the common
foundation we give them. They say:
• I find that knowing parts of all programmes helps to gain understanding of everything within
computing. Making your course decision effective to get the best out of it;
• Gave me the opportunity to see what I enjoyed and what I didn’t. Good indication for year 2 choices.
5. CONCLUSION
In this paper a brief overview of a new approach to curriculum development and delivery has been presented.
Because we are not yet through the academic year it is impossible to give a full evaluation as final results and
progression rates are as yet unavailable. However, it has been possible to present our motivations and
methods for effecting change and to report some preliminary quantitative data on the response to it. In terms
of student experience - and indeed of staff experience - this is perhaps of greater importance.
The development and running of the course has not been without problems. It has been time-consuming and
tiring for many on the teaching team and we have been acutely aware of the times when we have fallen short
of our own wishes, for example in effecting the best possible assessment regime, and in successfully
negotiating the division of marking between ourselves. However, these are teething problems to be expected
and we remain upbeat and convinced of the value of this new approach. We believe the standard of the work
this year’s cohort of students is producing is higher than that of previous cohorts and that this has been
effected largely through this LTA approach together with the peer support that has developed and increased
competition among the students for the highest marks. We are looking forward to the challenge that level 2
brings with these students moving forward. Meanwhile, we will spend time at the end of the year fully
reviewing and evaluating this year’s experience and planning for an improved level 1 roll out next year. The
student voice in this process will continue to be invaluable.
REFERENCES
Don Quixote (2009) Readers’ Comments, in response to Tysome, T. (1996) Very model of a modular
approach. Times Higher Education.
http://www.timeshighereducation.co.uk/story.asp?storyCode=93861&sectioncode=26 (accessed 21/05/10)
Gass, J., Banks, D., & Wilson, A. J. (2004) Modularisation – flexible or restrictive professional education.
Nurse Education Today (2004) 24, 337–343
Goodhew, P. (2002) Modularisation & Sliced Bread. Higher Education Academy Discussion Paper.
http://www.materials.ac.uk/discuss/modularisation.asp (accessed 21/05/10)
Rust, C. (2000) An opinion piece: a possible student-centred assessment solution to some of the current
problems of modular degree programmes. Active Learning in Higher Education 1 (2), 126–131
211
AN INVESTIGATION OF THE SUCCESSFUL APPLICATION AND
DEVELOPMENT OF A QUICK, EASY TO USE, CROSS-DISCIPLINARY
INFORMATION EVALUATION MATRIX
Mike Leigh Lucy Mathers Kaye Towlson

Department of Informatics Department of Media Technology Library Services
De Montfort University De Montfort University De Montfort University
Leicester Leicester Leicester
ml@dmu.ac.uk lmathers@dmu.ac.uk kbt@dmu.ac.uk
ABSTRACT
This paper reports a study that undertook the further development of an Information Source Evaluation Matrix
(ISEM) using a series of field tests with students from a range of academic disciplines including technical
computing and non-technical subjects. The aim was to produce an evaluation tool that has a generic
application; can be used to methodically evaluate different kinds of source materials for a range of different
academic tasks; is straightforward in its application; and can raise students’ awareness of the necessary skills
required for information evaluation.
This study also identifies the potential to adapt the idea of an evaluation matrix in other areas of computing.
In particular prototype matrices have been developed for evaluating code from software libraries and also for
the selection of appropriate software development patterns for a given problem domain.
Keywords
Information evaluation, skills enhancement, academic referencing
1. INTRODUCTION
A research project investigating the development of learning communities in a Virtual Learning Environment
(Mathers and Leigh, 2008) highlighted the need to enhance the students’ ability to evaluate the academic
worth and relevance of materials that were posted. Although a range of free and widely available on-line
tutorials already exist to help students evaluate information, for example, QUICK (Heath Development
Agency, 2000), Intute’s Internet Detective (Place et al, 2006) and the Open University’s Safari (Skills in
Accessing, Finding and Reviewing Information); when links to these were provided it was found that less than
5% of students used them. This led to the initial development of an Information Source Evaluation Matrix
(ISEM), which is a simple, and easy to use tool facilitating the appropriate selection of source materials for
academic work (Leigh et al, 2009). The design of the matrix was based around the following five Ws
(questions) that form the foundation of information evaluation:
 Who? – is the author
 What? – is the relevance of the points being made in the source
 Where? – the context for the points being made in the source
 When? – was the source published
 Why? – the author’s apparent purpose / reason for writing the article.
It is believed that this tool is a significant pedagogic and learning aid as it facilitates a deep understanding of
information evaluation of academic and other sources and encourages appropriate academic citation and
referencing practices. This study aimed to validate this belief by trialling its use within several faculties in the
University and also as part of a transition project helping students in their move from school/college study to
studying at university.
212
2. METHODOLOGY
This work was carried out using an action research approach (e.g. Denscombe, 2005; Cohen et al., 2000) to
investigate the likely utilisation of the ISEM by students from within different academic disciplines and with
diverse needs. This strategy was chosen, as the project is the continuation of earlier work carried out using
this methodology. Appendix 1 outlines the processes and cycles undertaken during the previous study.
Figure 1 outlines the processes undertaken during the current study.
Figure 1. Action research approach undertaken in this study - adapted from “the action research spiral” (UCD Centre for
Teaching and Learning, date unknown) and “the iterative nature of action research” (Allen, 2001).
3. DATA COLLECTION
Interest in, and take-up of, the ISEM has been much greater than initially anticipated. In order to understand
the effectiveness of the ISEM usage by students with diverse needs and within different academic disciplines,
the collection of data has been summarised into the following categories:
 A deeper study of the use of the ISEM within the Faculty of Technology.
 A broader study of the use of the ISEM across the University in a wider range of disciplines, and from
different levels of study.
 The use of the ISEM within transition project packs provided to sixth-form students beginning their
first year of study at De Montfort University.
 Ad-hoc usage of the ISEM.
 External interest in the ISEM.
3.1 Faculty data

Within the Faculty of Technology, further field-testing of the ISEM has taken place. A final year (level 6)
Computing, Ethics and Professional Practice module, studied by technically orientated students, used the
213
ISEM to develop their information source evaluation skills required to write a research paper. Students were
prompted for feedback as follows:
1. Did you like using the Information Source Evaluation Matrix? Yes/No
2. Did you find it useful? Yes/No
3. Would you use it again? Yes/No
4. Write two statements outlining what you found useful in the use of the Information Source Evaluation
Matrix.
5. Write two ways in which the Information Source Evaluation Matrix could be improved.
Additionally, the ISEM has been embedded into the same level-5 Multimedia module participating in the
previous study (Leigh, et al., 2009). The new cohort of students completed a short (on-line) questionnaire to
ascertain their understanding of, and attitude towards information source evaluation. This was administered
both before and after their use of the ISEM.
3.2 University-wide data

To date the matrix has been applied within the following subject areas (not including the Faculty of
Technology, see section 3.1) and levels of study from undergraduate (UG) to postgraduate (PG):
 Education
 Architecture (UG & PG)
 Textiles (UG: level 4 and level 6)
 Interior design (UG)
 Institute of Energy and Sustainable development (PG)
 Group of students of mixed subject disciplines enrolled on Transition project.
 Business and Law via study support office
 Design innovation (PG)
 Masters level (7) study support group attended by students from all Faculties / disciplines.
Feedback information, as specified in section 3.1 above, was collected.
3.3 Transition project

The use of the ISEM within transition project packs for use by a select student sample from a local sixth-form
college during their first year of a University programme, again covering a variety of disciplines. The pack
was provided to students during their final year at the local college and introduced, with the contents
explained by University staff. Regular focus group meetings have taken place with the students throughout
their first year of university study and students use of the pack content explored. Feedback on the use of the
ISEM is being collected by the Transitions project team and will be available for this study during July.
3.4 Ad-hoc usage

The ISEM is now an official De Montfort University library publication and has been promoted across all
Faculties. A .pdf version of the matrix is available for staff and student usage via the library web page
http://www.library.dmu.ac.uk/Images/Selfstudy/ISEMLeaflet.pdf. The leaflet contains a feedback page that
requesting the same information as found in section 3.1 above. Users of the leaflet can place the feedback
page in collection boxes at information desks across the library, or send via email. As yet, no feedback has
been collected in this way. However, statistics have been collected relating to the number of times that the
ISEM has been accessed via the web page.
3.5 External Interest

Tutors at Leicester University Library, Herriot-Watt University Library and Cranfield University Library have
requested to use the ISEM within their teaching sessions and have been asked to feedback into the project.
Leicester University will be applying it to critical analysis training for medics.
It has been used within teaching of library use, and a feedback statement provided by library staff at Lake
Washington Technical College at Kirkland, Washington, USA and has also been embedded into Headingly
Library of Open Education Resources at Leeds Metropolitan University.
Furthermore, a school librarian requested to use the ISEM with Sixth-form students about to embark on an
Extended Project Qualification.
214
4. FEEDBACK ON ISEM USE
This section reports on the feedback received during the project so far. It should be noted that cycle 5 of the
research (Figure 1) is only just beginning. The project is not due to be completed until the end of July 2010.
Feedback received from use of the ISEM within the Faculty is provided in section 4.1. University-wide
feedback from students is summarised in section 4.2, and all other feedback, for example from external
sources, is outlined in section 4.3.
4.1 Faculty feedback

This section reviews the feedback that has been collected from students undertaking two modules within the
Faculty of Technology at De Montfort University.
4.1.1 Computing, Ethics and Professional Practice module

The ISEM was introduced to the students at a session in which advice was given on how to write academic
paper. The importance of identifying appropriate journal papers from the literature was covered and use of
the matrix in achieving this was explained. Students were asked to complete a feedback form, as outlined in
section 3.1 above, and to submit it with their written paper. A total of 38 feedback forms were received. It was
possible to check which students had submitted feedback against the attendance list of the introductory
session at which 41 students were present. A comparison showed that more than 90% of the students who
submitted feedback also attended the introduction.
Of the 38 students providing feedback 29 stated that they liked using the ISEM with the other nine reporting
that they did not. A total of 30 students found the matrix useful in evaluating their source materials with 8 not
finding it useful. When asked to indicate whether or not they would use the matrix in the future 28 said yes, 9
said no and one student said maybe. Examination of the feedback forms revealed that opinions on the use of
the matrix were polarised with students either having positive views on its usage or else the negative views.
There was little evidence of students’ mixed views of its usefulness. In general, the more positive responses
on the use of the ISEM were from students that had attended the introductory session.
4.1.2 Multimedia module

In a change to the previous study (Leigh et al., 2009), the ISEM was introduced in laboratory session within
this module, rather than a lecture. Additional support and information on the use of the ISEM was provided
electronically as well as face-to-face, which informed the guidance information provided with the ISEM Library
Leaflet. The feedback collected differs from that obtained during the remainder of this study (see section 3.1);
the questionnaire used is a shortened version of the one presented in Leigh et al. (2009) and is provided in
Appendix 2 along with the statistics of the student answers for the two surveys.
The findings in the current surveys bear little resemblance to those provided in Leigh et al. (2009). However,
similarities were not necessarily expected due to the differences in cohorts and the number of students who
volunteered to complete the surveys, information provided on ISEM use, and timing of the session within the
academic year. The current surveys do indicate a move away from indifferent answers, and it is suggested
that use of the ISEM has given the students more confidence in their own opinions relating to evaluation of
information sources. This finding at least is similar to the work presented in the previous study (Leigh et al.,
2009).
4.2 University-wide feedback

Across the University, the ISEM was introduced and used in face-to-face sessions. Students were asked to
complete a feedback form as outlined in section 3.1 above. A total of 32 forms were received. 21 students
stated they liked using the ISEM and 11 did not. 23 students found it useful and 9 stated they did not. 15
students said they would use it again, 15 would not and 2 students were unsure.
Additionally, out of a subset of 24 of the 32 students, 21 stated that they found the matrix easy to use (2 did
not, one did not answer) and 23 agreed that they thought it was important to evaluate information before using
it for an assignment (1 student was not sure). The postgraduate students were the least positive about the
use of the matrix, however as a collective group, those studying at level 6 found the matrix to be most useful.
This could be related to the immediacy of these sessions to them working on their dissertations or final year
projects.
4.3 Other feedback

Feedback gathered from those outside the university that expressed an interest using the ISEM suggests that
students found it easy to use, useful and would use it again; the additional information provided to explain the
five Ws (Who? What? Where? When? and Why?) had helped students focus on all the required criteria for
215
evaluating sources; and the grid is laid out cleanly and allows a student to quickly evaluate their source. It
was also suggested that the source and task information be moved to the top of the grid in order to prompt
students to fill this in before beginning the evaluation of the source. Support for an on-line version of the
document was also provided.
Rank (in the top 100 of
Month Number of Hits
library URLs based on hits)
February 50 332
March 63 152
April 51 295
May (to 26th) 81 95
Table 1. Usage statistics for access to the PDF file from the DMU library web site were collected on 26th May 2010.
Usage statistics for hits (downloads) of the PDF version of the DMU ISEM Library publication have been
collected (Table 1). It should be noted that these statistics will include the accesses by the authors and others
related to the project during promotion and use of the ISEM. For example, when embedding a link to the URL
from within a VLE module shell most tutors would test that the link works correctly, thus creating a hit statistic.
There is no way to remove these types of hits from the statistics.
4.4 Summary
Preliminary analysis undertaken on the feedback results in the following:
1. Where students were asked to write two statements outlining what they found useful in the use of the
ISEM, statements were identified as belonging to the following categories:
 Facilitating the quick evaluation of sources
 The ease of use of the ISEM
 The effectiveness of how sources maybe evaluated
 The raising of awareness of evaluation criteria
2. ISEM improvement suggestions fell into the following groups:
 Disagreement of the given criteria
 The need for more guidance on the use of the matrix
 Improvements that could be made to the matrix layout
 Issues concerning the access mechanism
A selection of comments from the feedback forms, representing as many different viewpoints as possible, has
been provided in Appendix 3. The Information Source Evaluation Matrix has been modified as a result of the
feedback so far collected from various sources. This can be found in Appendix 4.
5. FUTURE WORK
This project has resulted in the ISEM undergoing further field-testing and modification. It is now in a position
where an online version can be created which will be able to calculate scores, store comments and task
descriptions and provide a keyword search of the information held. The demand for such a widget would need
to be investigated since out of all the participants involved in the study so far only a very small number have
suggested that an online automated version of the matrix would be useful.
One area of future work that is considered promising is to use the idea of the ISEM in other areas of computer
science where evaluation is an important process. In particular such a tool might prove useful when using
software libraries to promote reuse and also when selecting software development patterns for specific
problem domains. Initial prototype evaluation matrices have been developed for these purposes and can be
found in Appendix 4. These are far from complete but do give some insight into the possible usefulness of
such tools.
6. CONCLUSIONS
It is important that students are guided in the use of the ISEM. As was evidenced in the feedback from the
technical computing students (section 4.1.1) those that gave the most positive feedback on he usefulness of
the tool where the ones that attended face-to-face introductory sessions. It is, therefore, recommended that
the ISEM be introduced in this manner. If that is not possible, then clear guidance notes are needed to help
students understand the descriptors of each of the criteria.
216
This project has validated the findings of the previous study in an interdisciplinary context. Students found the
matrix quick and easy to use; a useful comparison tool when investigating different sources; and its visual
structure and presentation facilitating the inclusion of all necessary evaluation criteria. Also confirmed from the
previous study was that the ISEM helps students to better understand the process of information evaluation.
However, it is apparent that more work needs to be done on making the ISEM more suitable for evaluating
visual sources such as images, video and multimedia.
It was suggested that the inclusion of an example of reference would be useful. This, though, would require
tutors to amend the reference to suit the referencing system adopted in that subject area.
The development of an electronic version created as a widget would allow the user to add / edit criteria and
adjust weightings to suit their specific needs. It would also allow different versions of the ISEM to be
accessed, for example, selecting the matrix type via a drop-down list. Versions developed could be
specifically targeted based on the type of source to be evaluated and/or for specific tasks such as selecting
from software libraries and choosing appropriate software development patterns.
REFERENCES
Allen, W.J. (2001) The role of action research in environmental management. In: Working together for
environmental management: the role of information sharing and collaborative learning. Ph.D. (Development
Studies), Massey University. Available from:
http://learningforsustainability.net/research/thesis/thesis_ch3.html, (accessed 28 May 2010)
Cohen, L., L. Manion, and K. Morrison (2000). Research methods in education, 5th Ed., Routledge Falmer,
pp 226 - 241.
Denscombe, M. (2005). The good research guide for small-scale social research, 2nd Ed., Open University
Press, pp 73 - 83.
Heath Development Agency, Centre for Health Information Quality and Showme Multimedia Ltd (2000). The
QUality Information ChecKlist (QUICK). Available from: http://www.quick.org.uk (accessed 9 October 2008).
N.B. Link no longer available, but details from:
http://www.nice.org.uk/aboutnice/whoweare/aboutthehda/hdapublications/quick_website_pdf.jsp (accessed
28 May 2010)
Leigh, M., E.L. Mathers, and K.B. Towlson, (2009) Using face-to-face sessions and focus groups to develop
online support to enhance student content evaluation skills in VLE learning communities, SOLSTICE
conference ‘It’s all in the blend?’, Edge Hill University, 4 June 2009, Available from:
http://www.edgehill.ac.uk/solstice/Conference2009/docs/session22.pdf (accessed 28 May 2010)
Mathers, E.L., and M. Leigh, (2008) Facilitators and barriers to developing learning communities, Higher
Education Academy Annual Conference ‘Transforming the student experience’, Harrogate, 1–3 July 2008,
Available from: http://aces.shu.ac.uk/support/staff/HEA_Harrogate/ (accessed 28 May 2010)
Place, E., M. Kendall, D. Hiom, H. Booth, P. Ayres, A. Manuel, and P. Smith (2006). Internet Detective: Wise
up to the Web, 3rd edition, Intute Virtual Training Suite. Available from: http://www.vts.intute.ac.uk/detective/
(accessed 28 May 2010)
The Open University (date unknown) Skills in Accessing, Finding and Reviewing Information (Safari).
Available from: http://www.open.ac.uk/safari/index.php (accessed 28 May 2010)
UCD Centre for Teaching and Learning, (date unknown). Problem solving research/action research,
Available from: http://www.ucd.ie/teaching/t&lresearchMethodologies_sub/problem_research.html (accessed
28 May 2010)
217
APPENDIX 1
The action research cycle undertaken in Leigh et al. (2009).
APPENDIX 2
An on-line likert-scale questionnaire distributed to students undertaking the Multimedia Module during 2009-
2010.
The questionnaire was completed voluntarily before and after their use of the ISEM. The questions were
introduced to the students in random order.
Information Evaluation Survey Questions

1. I am confident that I can rank articles in terms of their reliability for my work
2. I usually have a gut feeling about whether a resource will be useful for my work
3. Material is out of date if published before 2000
4. Older material (for example from books or journals) can still be relevant for my work
5. In general you can trust information posted on (we)blogs or wikis (e.g. wikipedia)
6. I can trust academic journal articles to be of higher quality than articles found on websites
7. I would only choose to read a source if the title closely matched my keywords
8. I am confident that I can understand the reasons behind the author’s work and its application to my
own work
9. If I find an article difficult to understand I would never use it in my studies
10. If I do not agree with the information in an article then I won’t use it in my studies
218
Survey Results
APPENDIX 3
Example feedback statements selected to illustrate a wide range of feedback and grouped into themes.
Write two statements outlining what you found useful in the use of the Information Source Evaluation Matrix.
Theme: Facilitating the quick evaluation of sources
It's quickly showed what papers to use
It is very clear; in a few minutes we can know which papers are better for an essay
Using the score system we can quickly find and review which journal papers are sufficient to use in the assignment
Theme: The ease of use of the ISEM
It is easy to follow the guidelines to find the papers
The matrix made the assessment easy and simple
A numbering system to consider whether the piece of writing is useful
Theme: The effectiveness of how sources may be evaluated
The additional levels 1 – 5 and associated questions helped me to focus my thoughts when assessing the source
The fact that there were marks from one to five to each question was very useful because I was aiming for the best mark so I tried to
find journals which were up to date and their authors were well-known in the area
It made me look deeper to find journal papers because I didn't really know that there could be so many
Theme: The raising of awareness of evaluation criteria
Leads me to notice the author and the publication year, to see if the reference is sufficiently professional and up-to-date which may be
more appropriate for present research and project
Knowing what to consider and knowing which is a good choice of reference.
It helped to give me an idea of how referencing would be assessed
Write two ways in which the information source evaluation matrix could be improved
Theme: Disagreement of given criteria
I don't agree with the way marks are allocated, for example, under ‘when’ sometimes an old article is better as it was written straight
after new developments and new articles cannot reflect on the impact such developments had on the environment at that time
Fewer criteria
Why section does not relate to images/websites
Theme: The need for more guidance on use of the matrix
Can have a more specific contexts side, the when, why etc are a bit vague, have example statements
Some questions in matrix not easy to mark
Include example of reference, would be useful
Theme: Improvements that could be made to the matrix layout
A better layout, i.e. is there a need for the score row under every category
Allow for more individual feedback e.g. being able to write in own comments
Should include other information resources eg. Type of resources, name of resources etc
Theme: Issues concerning the access mechanism
The more useful if it was online
219
APPENDIX 4 – MODIFIED INFORMATION SOURCE EVALUATION MATRIX
Source/Reference:
Task/Question:
Evaluation 1 2 3 4 5 Score
Criteria
Who? - is the Author background Some evidence Evidence of some Author has several Author is a known
author is unknown author works in this publications in this published works in authority in this area
area but few articles area by author this area
Score
What ? - is the Content and Only of peripheral Some of the content Several points Content and
relevance of arguments of little or /little relevance to is relevant to task made are of arguments closely
points made no relevance to the task being requirements relevance to task match your needs
task undertaken
Score
Where? – Situation to which Minimal similarity Author’s situation Reasonable Author’s context and
context for points author applies between author’s and that of the task similarity between that of the task very
made points is different to context & the task have some similarity author’s and task similar
that of the task context context
Score
When? – was the Date is unknown or Old reference – Reference is Recent reference is Up-to-date source –
source published older than 20 years between 10 and 20 between 5 to 10 2 to 5 years old or published in last two
old years old years old known important years or seminal
paper paper
Score
Why? – author’s No apparent News paper (or Trade magazine / Book source / Academic journal
reason/purpose motivation seen in online) article commercial paper – conference paper or paper – peer
for writing the article opinion – not might have some subject interest reviewed
article evidenced bias forum/blog
Score
Notes:
Total score
220
APPENDIX 5
Software Libraries Evaluation Matrix - Prototype Software Development Pattern Evaluation Matrix - Prototype
1 2 3 4 5 Score 1 2 3 4 5 Score
Who? - provenance of Author background is Code is from a reputable Who? - wrote the Author(s) Pattern is from a reputable
software code (source – unknown - code ... ... ... source – vendor/author
pattern – what is its background is .. ... .. source –
author) possibly ‘hacked’
provenance? unknown - pattern . . organisation/author
Score possibly ‘flawed’
What? - is the functionality Does not provide Provides an exact match Score
provided by the software required functionality to required functionality
but is adaptable ... ... ... What? - is the Does not directly Provides an exact match
design problem solve the problem to the problem being
Score .. ... ..
addressed by the but is adaptable with resolved
. .
What? – is the Interface? Not coded in target Written in required pattern effort
How easy to integrate? language, can be language, can be
adapted with effort ... ... ... plugged-in Score
Score
Where? – context Problem situation Context and that of the
What? – is the license Open source code Strict licensing in which the pattern for which the pattern pattern highly similar -
.. ... ..
agreement? Can code be can be used for agreements to be is a proven solution is needed is proven solution for the
used freely? whatever purpose ... ... ... adhered to . .
different to that of domain
the pattern
Score
Score
Where? – context for use of Situation for which the Context and that of the
code -is the software proven code is needed is requirement highly What? – are the Some of the forces Forces seen in the
for required situation different to that of the ... ... ... similar - proven code for Forces that need to will be resolved but problem will be resolved by
requirement the domain .. ... ..
be resolved? Will not others – some the pattern with few
Score . .
the pattern resolve conflict seen difficulties
When? –how established is Beta version - not yet Well used, reliable and them?
the software fully developed robust software
... ... ... Score
Score When? –how Pattern has been Widely applied and proven
established is the used in a small in many different domains
Why? – reason/ purpose for Written for single use Written to be a reusable .. ... ..
in a specific component - well
software pattern? number of cases – – well known solution
writing the code . .
environment –sparse ... ... ... documented and not yet established
documentation reviewed Score
Score
Why? – reason/ Written for use in Written to be widely
purpose for writing specific specialist applicable - well
Total score .. ... ..
Software Source: the pattern environment – documented and reviewed
. .
transferability
Required Functionality:
uncertain
Score
Total score
Pattern name:
Problem requiring resolution:
221
FOSTERING SOCIAL COMPETENCE, COMMUNITY
AWARENESS AND EMPLOYABILITY SKILLS THROUGH
VOLUNTEERING AND EMOTIONAL INTELLIGENCE (EI)
SKILLS
Dr Lynne Humphries
Dept of Computing, Engineering and
Technology
University of Sunderland
lynne.humphries@sunderland.ac.uk
http://osiris.sunderland.ac.uk/~cs0lh
u/Lynne/hea_tasc.htm
ABSTRACT
Success in computing at higher education level is not just about acquiring a set of technical and academic
skills but also developing a set of social competencies to be able to work in teams and act professionally, all
skills that will enhance employability. This paper reports on the outcomes of the ICS Development project
entitled Transition to Academic and Social Confidence through an Education for Sustainable Development
(ESD) Volunteering Activity. 62 of an initial cohort of 100 level 1 computing students at the University of
Sunderland undertook an online emotional intelligence test (the Mayer-Salovey-Caruso ability-based test
(MSCEIT) in their first term at the university. The ESD part of the project was volunteering activities with local
community groups to provide a website, training and publicity material. Only 3 level 1 students volunteered, all
3 are non-English speaking students. The test and volunteering was also offered to new MSc students within
the department and 12 students were recruited from this cohort. This group comprised home and non-EU
students. All were given initial training in emotional intelligence (EI) awareness and feedback on their results.
There was a significant correlation between their EI score and performance on the four assessments of a
large, common module. By contrast, there was no relationship found for the masters' students between their
performance on a common research module and their EI score. Analyses of the results for social competence
in this transition period highlight two groups in possible need of intervention. The lowest EI scores were
obtained by a small group of young (18 year-old) male undergraduates, some of whom had been reported by
staff as having problematic behaviour in class, most having a problem with attention and a lack of empathy/
perspective taking. The other group were some of the level 1, non-EU students and the results suggest not
only a comprehension problem with the test but possibly also a conceptual problem with course material. The
low take-up of volunteering amongst the youngest students suggests that Level 1 is not the best time to offer
these activities. The more mature MSc students have gained from interaction with the community with
improved transferable skills of communication and negotiation. Suggestions are made in the paper about how
to apply free, validated, self-report versions of the EI test to new students and which interventions can be
used by departments.
Keywords
Transition to Higher Education, emotional intelligence, volunteering
222
1. INTRODUCTION
Retention of new undergraduate students in all disciplines is important and with the emphasis being placed on
widening participation it is important to investigate factors that help or hinder students to succeed in this
transition. Students who are the first generation to attend University and from lower socio-economic groups
are most likely to leave the course possibly due to the lack of support networks (NAO, 2007). Tinto’s (2000)
model of student departure identifies pre-entry internal and external factors in which the internal factors
include individual attributes. The corresponding post-entry factors include social integration as the main non-
academic factor. The attention to secondary school marks and cognitive factors as predictors of success were
earlier reviewed in Tinto (1993) but the predictive power proved limited (Randsell, 2010) and attention turned
to other predictor variables. More recently the link has been made between emotional intelligence (EI) and
academic achievement in the transition into higher education. Parker, Summerfeldt, Hogan and Majeski
(2004) used the EQi short test (Bar-On, 2007) to test students at their first week in classes at a Canadian
university. This test is a self-report, trait-based test. The results showed no difference in academic
achievement between those with high EI or low EI scores, but most of the successful students scored more
highly on the the emotional and social competency variables. Lowe and Cooke (2003) describe how the move
away from home requires social, emotional and psychological adjustments. In their study of the transition of
elite athletes McNamara and Collins (2010) identify useful mentalising skills used by successful students such
as imagery and goal-setting. The metacognitive skill of mentalising and reflecting on the process of
adjustment is part of the development of social competencies including emotional intelligence. Emotional
intelligence has been described as the capacity for recognising ones own feelings and those of others, for
motivating ourselves, and for managing emotions well in ourselves and in our relationships (Goldman, 1998)
or the ability to effectively reason about emotions and use emotions to aid cognitive processes and decision-
making (Mayer, Salovey and Caruso, 2002). In this study the focus is on the metarepresentational skills
required by students to adjust to the transition to self-directed learning. Identifying mental states is contingent
upon attending to others and identifying correctly their emotions. This is one of the four components of the
ability-based Mayer-Salovey- Caruso emotional intelligence model that comprises the MSCEIT test:
perceiving emotions, using emotions, understanding emotions and managing emotions. This is the test (in its
online version) that is used in this research for measuring emotional intelligence.
The main contribution of this study is to investigate the link between EI and social competencies that may
encourage students to volunteer. The main research question was how does emotional intelligence help with
the transition to Higher Education and can interventions, such as volunteering help EI skills development,
social competency and success at University. This leads to 2 hypotheses: Hypothesis 1 – there is a
correlation between EI skills and success in the first year of a university course; Hypothesis 2 volunteering
activities aids social and emotional development.
2. METHODS
In order to test these hypotheses all Level 1 computing students were given an induction into emotional
intelligence research and the work done in the department on previous community projects (such as
volunteers building a Country of origin website for the North of England Refugee Service in Sunderland) and
how they would be given an opportunity to both take part in the research by participating in the EI tests using
the ability-based, online MSCEIT test and to take part in one of a selection of volunteering projects under the
auspices of the HEA TASC-ESD project. We had a launch of the project in November 2009 where community
representatives requiring work and interested students could meet and talk. This resulted initially in 12
volunteers and two community projects. Unfortunately only 1 student from Level 1 volunteered (a non-EU
student) and the rest were MSc students. However, 62 of the Level 1 students completed the MSCEIT as well
as 11 MSc students. It was decided to collect data from the Level 1 students to test Hypothesis 1. All the
students on Level 1 were following a substantially common syllabus so their achievement on the large 100
credit common module could be collected as data for the dependent variable with their EI scores being the
independent variable. Qualitative data was collected from Level 1 and 2 students from interventions that took
place during the year. These interventions were in the form of EI awareness-raising and targeted counselling
for a small number of students (1 Level1, 1 MSc student). Additional qualitative data was collected from the
volunteers. This took the form of interim feedback on progress and team relationships and feedback at the
end of the project. An assessment was made of the success of each project in terms of deliverables and client
feedback. Data was collected from the MSc students for the volunteering activity (Hypothesis 2).
223
The first of these projects was the St Mark’s Community Association Website project. The students were
asked to develop a sustainable website that was a support resource that could be easily maintained by the
community. Six students volunteered for this project initially, 1 level 1 student and 5 MSc students.
Unfortunately the Level 1 student wanted to repair computers and as this was not needed he decided not to
continue. Support was given to the students in the form of regular meetings with me and also training in the
use of Camtasia software so that they could record a video for the client on how to administer and update the
website. The second project was for The Washington Riding Centre for the Disabled. The centre provides
stabling facilities and lessons for disabled riders. The Centre Volunteer Coordinator at the centre required a
durable and attractive handbook for the volunteers they recruit regularly. The students were given the
opportunity to help with riding for a day so that they could understand the context of use. Support was given to
the students again with regular meetings, transport arrangements and printing of the final handbook. Each
group elected a project leader and tasks were assigned: St Marks - website developer, layout developer,
Joomla (a Content Management System) implementer and photographer; Washington riding centre – project
leader, designer, author).
3. RESULTS
3.1 Volunteering
3.1.1 St Mark’s Community Group.
The five students (4 male and 1 female) who completed the project comprised 2 EU students and 3 non-EU
student. At the interim feedback stage the team had produced Joomla templates for the “look and feel of the
site”, the facility to add new content easily, an added live feed from “Twitter”, allowing the website
administrator to add the latest news using his mobile phone with a Twitter account and a form for users to fill
in with any queries. They identified milestone meetings for the addition of a location map, video tutorials,
photo gallery tool and delivery and training of the community administrator. The project was very successful
and the feedback from the client was very enthusiastic- the administrator praised the students and thought
that both sides had gained from the volunteering. The students took pictures for a virtual gallery and designed
the site using Joomla- a free Content Management System which allows the site to be maintained by non-
specialists. They also produced an instruction walkthrough of the site using Camtasia. Links to the website
and the Camtasia video are on the website listed on the first page. The students were asked what they gained
from the project. They appreciated learning the new skill of using Camtasia to provide training material.
However, there was more engagement from the two English students who had very forceful personalities.
Although very efficient at organizing everything the project leader tended to take over. His EI score was in the
normal range overall but a low score for Understanding Emotions. He had a high score for perceiving
emotions (reading people) but a relatively low score for Using Emotions (empathy). He may well be a natural
leader but misses some of the subtler cues or does not care that others may be not entirely happy with his
leadership. The EI scores of the ten MSc students who volunteered for the two projects showed no correlation
with scores on a common module research skills taken by all MSc students. On re-testing his score had
dropped in all 4 sub-areas over the period of volunteering. This was unexpected but on the evidence of just
one student no conclusions can be drawn. However, Dainty, Mo and Price’s (2006) found a decline in
emotional intelligence during the engineering course taught at Loughborough that they ascribed to a course
design that was not conducive to developing EI.
3.1.2.Washington Riding Centre for the Disabled

The number of students who started the project soon diminished to 3 very keen volunteers. These MSc
students (1 female, 2 male) are from Malaysia, Iran and Nigeria. Their MSCEIT scores were all good, in the
competent range. The Riding Centre is situated in the countryside some 12 miles from Sunderland and the
visits offered them a chance to see the locale in the extremes of weather ( Jan/ Feb 2010) and also to mix
with the local community. All three commented on how much they had gained and the volunteer coordinator
was very pleased with their application and the quality and usefulness of the handbooks. The Centre have all
the electronic files to enable them to print more in the future. The handover photographs at the Centre can be
seen on the website link.
3.2.Qualitative results and analysis: Student interventions
224
Having recently completed the MSCEIT training and also having gained an MSc in Psychology I am in a
position as a computer lecturer to try to apply some socio-affective analysis of my students’ results obtained
during this research. There is rarely time to interact with new students outside teaching to gain some insight
into what helps them settle or want to leave. The opportunity afforded by the volunteering activities and the
intervention activities allowed time to do this, albeit on a small scale.
One Level 1 student was identified as one who could benefit from intervention. He had disciplinary problems
and a very low EI score. He was offered and accepted a counseling session to discuss his EI results. An MSc
student was referred as having an interest in EI and in having help with his studies as he is dyslexic. They
both gave permission for their cases to be reported (without identification). The analysis of their scores and
MSCEIT feedback gives interesting case studies in to the use of socio-affective skills used by students, one
as a widening access student at risk of dropping out and the second a self-referred dyslexic student wanting
some insight into how a knowledge of EI skills could help his academic success. The full 24 page MSCEIT
Resource report was obtained for each student and in an hour-long session I asked them (in a one-to-one
session) to reflect on their scores and their behaviour at university.
Widening access- Student S

Student S (age 20, Level 1 Computing) started the session by saying how unhappy he was at University and
how he had distanced himself from some of his friends who were not as committed to the course as he
wanted to gain a first class degree. His overall score for emotional intelligence is very low (64)- a score of less
than 70 equates to lowest 2.5% percentile of the population. His best score was on the Perceiving Emotions
sub-score and the lowest was Using Emotions. This latter sub-score is linked to empathy (Salovey, Caruso
and Mayer, 2002). He was surprised by this and recounted the tale of where he had volunteered to be
sacked from a firm before he started University as he knew that he didn’t need the job as much as others with
children. This was surprisingly empathetic given his Using Emotions score of 66. We discussed the possibility
that in some situations he was empathetic but in others (where he thought he’d got away with inappropriate
social responses) he perhaps was misreading the impact of his actions on others, he was not able to put
himself in their shoes. He agreed with that scenario. He had a competent score for Perceiving Emotions
(reading people) -interestingly he mentioned that he uses this skill in poker games.
Emotional intelligence and dyslexia-Student N

Student N (age 22, MSc Software Engineering) volunteered to take the MSCEIT as he wanted some insight
into emotions and his academic behavior and whether this would help specifically with his dyslexia. Student N
scored competently across all four of the MSCEIT EI sub-scales. Student N has dyslexia and mentioned that
his performance (like speed of reading text) is very variable but believes that his performance is not stress
related. One intervention is to try to link positive moods with positive behaviours. I suggested that he might
like to keep a log of what moods he is in when he performs well or badly so that he can try to reproduce the
good mood when reading all the time. His lowest MSCEIT score was for Using Emotions (empathy). We
teased out examples of where he thought this may have caused him problems in the past. He initially
disagreed as he thought he was very empathetic but he also mentioned that he got too involved in another
person’s problems. One possible interpretation suggested is that he may be getting himself too involved but
he may not be picking up on subtle cues that the people he’s helping may be a little frustrated by his attention
if it’s perhaps over the top. He agreed with this when he reflected on it.
3.3 Academic success and emotional intelligence

The data from the MSCEIT scores (emotional intelligence) and the academic scores from the common
module were inspected and the data for the academic scores was found to be non-normally distributed
(Skewness,-.718, .304). A correlation analysis was conducted using the non-parametric correlation test
Spearman’s rho using SPSS v 16. The results (Figure 1) showed a significant correlation between academic
scores obtained at the end of the year and emotional intelligence scores at the beginning of the year (rS (62) =
.567, p <.01).
225
FIGURE 1 THERE IS A SIGNIFICANT CORRELATION BETWEEN EI SCORE AT THE BEGINNING OF THE ACADEMIC YEAR AND
ACHIEVEMENT BY THE END OF THE YEAR.
This supports Hypothesis 1 that there is a significant relationship between emotional intelligence and
academic achievement. Further analysis of those students with the lowest EI scores show that of the students
with the lowest scores (below 70 - the 2.5% percentile), the non-EU students are disproportionately
represented (58% despite only comprising 15% of the cohort who took part in the test). The low scores could
reflect difficulty with the language used (although the test has been validated using non-European/ American
users). Of the rest of the 20 English students with below average EI scores, there was one female and 11
males. This set of students comprise 19% of the tested cohort.
4. DISCUSSION
Results from the EI tests suggest that those with competent social skills as measured by the four components
of the MSCEIT test (perceiving emotions, using emotions, understanding emotions and managing emotions)
when they arrive at university achieve greater academic success. This supports Parker et. al.‘s (2004) study
with new students at University. The two students who volunteered for counselling both reported a positive
impact on their view of other people and life at University and both reacted positively to the way in which they
were encouraged to reflect on their behavior. Many of the students with low scores seem to have deficits in
the Perceiving Emotions sub-scale. They seem not to be aware of how their behavior impacts on people- they
are just not good at reading people. This may partly be inexperience and they will pick up the social skills in
time. It would be wrong to use any test as a label and the aim of this work has been to try to help students.
There is a gap here in University support for students in that there is a distinction to be made as a personal
tutor when a student may be advised to see a trained counsellor. However, very often there is a grey area of
lack of social skills or interaction that does not necessarily require the counselling service but perhaps what
the Open University advocates (or did when I worked for the OU as a tutor-counsellor Sumer school
counsellor). They recruit tutor-counsellors for new students, people who are specially recruited to offer skills
of empathy and the ability to read people as well as offer academic guidance.
There was not enough data to quantitatively test Hypothesis 2 that volunteering as an intervention aids the
development of social competencies, but the qualitative feedback indicates that those that volunteer gain
skills of teamwork, time-management and project-planning outside the constraints of their courses but often
competing for time with academic work. Not one of the volunteers complained that it had affected their
studies. For the overseas students they had a chance to interact directly with the community and the
community with them. It was a shame that the younger students were too busy coping with the adjustments to
academic life to come forward and volunteer.
226
5. CONCLUSION
The project was a success in that it allowed time to gather rich qualitative data on specific examples of where
students may struggle. The volunteering opportunities were eagerly taken up by the slightly more mature MSc
students but it is noteworthy that most of the volunteers were overseas students. They were a pleasure to
work with and it was not only the communities or the students that gained from this work. I had an insight into
the world and motivations of these volunteers that was invaluable. The University has a centralised
volunteering facility that matches students and community groups but valuable as this is it is not course-
based. Although time constraints preclude many academic staff from considering such community
involvement the students that do volunteer gain a great deal beyond the curriculum. A previous student from
Bangladesh succeeded at an interview for a part-time job because his volunteering work with older people
had shown him that he did not always get his point across and he had to change his approach. He reported
that this experience helped in interview situations.
Further work in the department will focus on possibly using a freely available self-report measure for new
students such as the Schutte Self Report Emotional Intelligence Test (SSEIT). This is a 33 item self-report
measure of emotional intelligence developed by Schutte et al. (1998). The SREIS has been designed to map
onto the Salovey and Mayer (1990) model of EI with items of the test measuring appraisal and expression of
emotion, regulation of emotion and utilisation of emotion. Whilst there may not be the funds or the time to
carry out lengthy evaluations and counselling, we will continue to use group activities to raise awareness and
skills in emotional intelligence.
References
Bar-On, R. (2007). The Bar-On model of emotional intelligence: A valid, robust and applicable EI model.
Organisations & People, 14, 27-34.
DAINTY, A, MO, Y. AND PRICE, E. (2006) EMOTIONAL INTELLIGENCE AMONGST ENGINEERING UNDERGRADUATES AT
LOUGHBOROUGH UNIVERSITY [HTTP://WWW.ENGCETL.AC.UK/RESEARCH/EI/, ACCESSED 20/5/2010]
Faye, C. and Sharpe, D. (2008) Academic Motivation in University: The Role of Basic Psychological Needs
and Identity Formation, Canadian Journal of Behavioural Science, Volume 40, Issue 4, October 2008, Pages
189-199
Goleman, D (1995) Emotional intelligence. Bloomsbury Publishing, London.
Lowe, H., and Cook, A. (2003) 'Mind the Gap: are students prepared for higher education?', Journal of Further
and Higher Education, 27(1): 53–76.
MacNamara, A. and Collins, D. (2010) The role of psychological characteristics in managing the transition to
university. Psychology of Sport and Exercise. (In Press) 1-10.
NAO (2007) Staying the course: the retention of students in higher education, Report by the Comptroller and
Auditor General | HC 616 Session 2006-2007| 26 July 2007., London: The Stationery Office
Parker, J D A; Summerfeldt, L J, Hogan, M J, and Majeski, S A (2004) Emotional intelligence and academic
success: examining the transition from high school to university, Personality and Individual Differences, 36,
pp. 163–172.
227
Randsell, S. (2001) Predicting college success: the importance of ability and non-cognitive variables.
International Journal of Educational Research, 35,357-364
Salovey, P., & Mayer, J.D. (1990). Emotional intelligence. Imagination, Cognition, and Personality, 9, 185-211.
Salovey, P and Caruso, D (2002) Mayer-Salovey-Caruso Emotional Intelligence Test (MSCEIT), Version 2.0.
Toronto, Canada: Multi-Health Systems.
Schutte, N.S., Malouff, J.M., Hall, L.E., Haggerty, D.J., Cooper, J.T., Golden, C.J., et al. (1998). Development
and validation of a measure of emotional intelligence. Personality and Individual Differences, 25, 167-177.
Tinto, V. (1993) Leaving college: Rethinking the causes and cures of student attrition (2nd ed.). Chicago:
University of Chicago Press.
Tinto, V, (2000) Learning better together: the impact of learning communities on student success in higher
education. Journal of Institutional Research; 9(1): 48-53.
228
PLAYING WITH CLOUDS: MAKING WEB APPLICATION
ASSESSMENTS MORE REALISTIC
Bruce Scharlau
University of Aberdeen
Computing Science
University of Aberdeen, Aberdeen AB24 3UE
b.scharlau@abdn.ac.uk
http://www.csd.abdn.ac.uk~/bcharla
ABSTRACT
This paper describes the reasoning behind introducing more of a problem-based learning approach to the
delivery of an advanced computing module, and the lessons learned from this change. In addition a key
component of the change was the use of Amazon Web Services and their cloud computing tools, which allow
for more adventurous applications to be developed by the students during the term. The basic lesson is open
up the assessment to let the students imagination guide their work, and to have a ‘plan b’ ready for when ‘plan
a’ does not go as expected due to technology issues, or other external factors.
Keywords
Assessment, cloud computing, problem-based-learning.
1. INTRODUCTION
CS5302: Enterprise Programming at the University of Aberdeen is the core module of an advanced MSc
computing science degree, which all students on the programme take. This module aims to provide them with
the basic knowledge of enterprise level Java web applications that will see them through the programme
including what they will need for their group and individual projects. The module includes the ‘nuts and bolts’
of Java web applications, plus the theoretical underpinnings for building such applications. This is reflected in
the 50/50 weighting between the coursework and the exam.
Originally the module had two assessments of the ‘build a web application’ and the ‘build additional
components to support or use the web application’ variety. This produced lots of work to be marked, and
mostly unexciting web sites, although some were quite creative and inventive. Nor did these assessments
address the deeper issues of building such sites, which would make the students think more fully about the
issues around large complex web applications.
Three years ago the approach to the coursework was changed to present more of a challenge to the students.
Now the coursework harnesses the imagination of the students. It also more efficiently pulls together the
theory and the nuts and bolts of larger web application development. The course is now more ‘professional’,
and aids development of graduate skills.
2. THE OLD APPROACH WAS PROBLEMATIC

The coursework in CS5302 is worth 50% of the total course mark, which reflects its emphasis on
programming. This was formerly broken down into two submissions. The first assessment was a ‘build a web
site’, which was due half-way through the term. The second assessment was ‘build a mobile front-end using
Java ME for the site plus integrate appropriate web services for the site’, which was due at the end of term.
Both components had required technical details, but were generally open to interpretation to cover a wide
variety of sites. However, the results were generally unsatisfactory. This worked fine for a number of years as
the students worked to understand the concepts and learn from the examples, but had become less
challenging as an increasing number of students started the programme knowing this material already.
229
First, students would put minimal effort into the content of their sites. This meant e-commerce sites with ten
items for sale and not much effort was required for the organization of the content on the web pages. Small
sites mean most of the work was trivial. Search, browsing, navigation and pagination are easy when there is
not much content to manage. Lists of items were retrieved from the database with ‘select * from tablename’
statements. This type of SQL statement was safe with no repercussions as so few items were in the database
that the returned result would not break the application. Similarly, most students managed to code up the
Java classes managing the models, controllers and views without too much trouble, and made good use of
the basic design pattern.
Second, the students had a hard time seeing any benefit in using more professional software development
approaches that will be expected of them after graduation such as testing, iteration planning, source control,
and similar processes. According to them it was all too short of a timescale for these tasks. The level of work
required for the assessment meant that there was no immediate payoff for the students to deploy professional
approaches.
Third, there was a short window in which the first assessment needed to be marked and returned in order to
be helpful as an input into the second assessment. The students already had a good idea of how well they did
on the coursework, so the value to them was less than the effort involved in the marking. Yes, they liked to
have the official comments and feedback on their coursework, but also realized that informal feedback, which
they had already received during practical sessions was just as useful for them to continue on with their
second assessment.
3. ENTER THE NEW APPROACH

Three years ago the approach to the coursework was changed. The coursework became more unified in an
effort to introduce more of the skills students will need to use as software developers after graduation. Instead
of two assessments, there is now one large assessment due at the end of term. Now the students put more
effort into their web sites, they see the benefit of source control, iteration planning, and some limited testing
while doing the work. Instead of doing this individually, the students also have the option of submitting the
coursework as a pair of students. If they do this as a pair, then both of them sign a form agreeing on how
much each of them contributed to the whole. Usually they claim to have done equal amounts of work. Most
students now do this as a pair, while some prefer to work on their own. Whether someone works on a team or
individually does not seem to have a bearing on their coursework mark, and both teams and individuals have
submitted exceptional pieces of work.
The coursework pulls together the theory and the practicalities of larger web application development so that
it is more ‘professional’, and aids development of graduate skills. The aim is to have the students do what they
would be doing if they were building this application in a small development company. This means
familiarizing them with the ‘nuts and bolts’ of everyday development. With the changed coursework the
students need to deal with navigation, pagination, search strategies and content planning if the site is to work
correctly. The students need to start this assessment early and continue to develop it as they progress
through the term.
The new coursework is moving towards a problem-based learning (PBL), or enquiry-based learning approach,
as it is also known. (Schanck, 2005; O’Kelly, 2005; Kahn and O’Rourke, 2004) PBL aims to let the students
choose how to tackle the problem and learn what is needed for the solution. This assignment differs from a
pure PBL approach in that the course still has most of the content set. However, the course does not
proscribe a set solution for the students. The students can choose the solution they prefer to resolve the
problems of their particular scenario. Indeed, there are little, but useful things, which are not covered in detail,
such as shopping carts, and pagination, which are mentioned, but for which they need to find a solution that
works with their application themselves.
This goal was to remove as much of the proscriptive details of the assignment as possible. This allows more
room for freedom of choice by the students to build something that they find personally challenging within a
larger framework of options. They should feel free to let their imagination roam and create imaginative
applications. If they are inspired by their application, then they will put more effort into it too.
4. PLAYING WITH CLOUDS

A centrepiece of this change has been using the tools and components of Amazon’s web services at
http://aws.amazon.com/ to build a finished application required to have 10,000 or more items in the database.
The scale of the application forces students to contend with issues required in real-world applications. Many
230
tend to find novel and useful solutions to the needs of their application. The students are guided in supervision
meetings during practical sessions where their plans and problems are discussed and evaluated.
Use of Amazon’s web services (AWS) computing cloud is only possible through grants which they make
available to academics for use in teaching. AWS has no free sandbox for developers to use. Yes, you can
make calls using REST and SOAP to retrieve data from Amazon’s product catalogue for free, but if you want
to start up your own instances running in the cloud, and deploy your application there, then you need to pay;
and the best way for academics to do this is to obtain a grant through the Amazon Web Service education
office (http://aws.amazon.com/education/). The prices for the hourly rates are low for each of the services, but
the grant provides cushion for the time a student forgets to shut down an instance and the fates prevent
anyone from checking that all instances have been stopped.
AWS provides storage for objects in its Simples Storage Service (S3) service, remote MySQL instances with
Relational Database Service (RDS), message queue systems with Simple Queue Service (SQS), and a basic
server instance running Windows, Linux, or Unix, and which you can install what you need, with its Elastic
Compute Cloud (EC2). You can just run your web server and database inside an EC2 instance, or separate
them out to use EC2 with an RDS instance for more stability, and then back up your application to S3 so that
you can replicate it again next time you need it, or if you wanted to start up multiple instances.
The new assessment allows the students to run their application inside of Amazon’s cloud. While they are
encouraged to do so, and a range of tools can make it trivial or not so trivial to do so, they do not have to
deploy their application into AWS. Setting up the tools and fighting with the proxy, as well as possible
problems with other components of the assessment mean that some students do not deploy their application
into the cloud. Most students take advantage of this chance to use it and show how they pulled the
components together for their application.
5. THE NEW ASSESSMENT

The assessment is conceived of as a mini-project that the students can do either individually or in pairs. They
are told about the details in the first class and that it is due on the last day of class. As noted earlier there are
three parts to complete prior to the final submission. Each of the three stages has a part describing what is
intended for that stage, which is due a week later, and the application components detailed in the description,
which are due about a fortnight later. The students are not obliged to follow the description of what they intend
to do blindly. They are free to submit something different if they find that their intended approach would not
work, but that they needed to pursue an alternative instead.
The new assessment is flexible in interpretation for most parts and comes down to a few simple required
parts: an application providing a transaction for users, who can either register for a service, or purchase a
product from a choice of at least 10,000 items, which they can find through browsing or searching through the
site. In addition there are a few ‘extras’, which would be desirable such as using the Spring Framework,
deploying the application in the AWS cloud, and enabling and consuming web services, or enabling suitable
mobile browsing of your application. Most students achieve at least several of the extras, while two or three
achieve them all. As these are not design students, we do not require an aesthetically pleasing site, only that
it provides complete and consistent navigation.
6. ITERATIVE AND INCREMENTAL DELIVERY

The students use iterative delivery during the term. This ties in with three stages during the term to ensure
that the students know where they should be for each stage of development. These are to (a) gather data and
build a prototype, (b) deploy to Amazon web services, and (c) optimise the application for mobile users, and
resolve potential bottlenecks in scalability. The final deadline is a fortnight after the third stage is due.
Building the site using multiple iterations has a number of benefits for both staff and students. The process
ensures that the students have something to submit for their final deadline. Each additional iteration adds new
functionality. Therefore the students will always have something that works which can be submitted for
assessment. In addition, this is similar to the working patterns in software houses, so understanding the
process adds to their professional skills. The staff have benefited from this approach by having a number of
places where they can intervene to point out problems that might occur if the team follows their proposed
path, and suggest alternative solutioins. It is also useful for staff to see the application develop over time, as
this provides a number of teaching points about the student’s design decisions, and the implication that arise
from their design. The result is that the students are more likely to have a successful application. In addition,
instructors are more familiar with the application at marking time, which means that the whole application will
be seen more clearly and better understood.
231
7. PROVIDE GUIDES AND ROUTES
The lectures and practical exercises align along the functionality required for the coursework assessment. The
lectures provide the context for the work, while the practicals provide details for a specific scenario. Between
the lectures and the practicals the students need to see how they can accomplish the ‘what’ of the
assessment. They need to see how the ’10,000 records in the database’ requirement can be achieved using
RESTful web services, and why this is not as difficult as it might appear. Similarly, using frameworks to build
applications is shown to be useful and less difficult than it appears.
The lectures cover the distributed web application concepts and discuss solutions while pointing out the
background. Alternative solutions are illustrated too, and differences between them discussed. The lectures
provide the context for their coursework and guide them to why something is useful, and what they need to
know in order to implement the solution.
The practicals allow the students to practice solutions in isolation, before integrating the components into their
own application. Implementation details of alternatives can be provided here too, so that they are easy for the
students to use and try as needed.
Alternative solutions are needed because the preferred solution might be unworkable due to lack of
documentation on a combination of frameworks or APIs. Sometimes the solution planned for inclusion this
term is discovered to be impractical in the time you have available. When this happens students need to be
pointed to an alternative solution that does work, and provided with a working example for them to use.
Working with technology such as AWS means that you might have new possibilities each week. This means
the practical used last year will need updating, and might even be redundant as new solutions are possible.
For example, last year RDS was available the week we were to explore configuring MySQL in an EC2
instance. Instead of doing that then, we could use RDS. This required a re-write of the practical and lectures.
Yes, it is fun and exciting to use the bleeding edge, but be aware that it takes time and you will need to know
the work arounds available to use proxy servers in the labs, which are usually only found in the fine print of
the documentation.
8. MONITOR PROGRESS
The deadlines for the iterations are all soft deadlines. It does not matter if the students miss them. The final
deadline at the end of term is the one that matters. Instead the deadlines provide a way for students to gauge
whether or not they are on schedule to meet the final deadline. In addition staff discuss progress with each
team or student during the weekly practical sessions. This provides another chance to uncover issues the
students might have encountered, and recommend alternatives if required. Sometimes it is necessary to
suggest alternative solutions as the path the students have chosen will not work as they expect. This needs to
be done so that the students can argue their position carefully and critically, while the alternatives are laid out
clearly, and they are left to find that they made a poor decision. They should always have the final choice, but
be made aware of fallback positions, which can mean they still submit suitable coursework.
Monitoring progress also means that those students who are reluctant to ask for help, still receive help. There
is at least one student like this each year, who is behind schedule, but does not ask for help, until spoken to
by someone else about their progress. It is important not to let these students be overlooked. They need to be
directed to workable solutions that will mean they submit something for their coursework.
Problems faced by one group might also be indicative of what the other teams will face, as usually one team
is ahead of the others, so this also provides an early warning system. Alternatives can then be found and
brought to the attention of the other students. Therefore, a ‘plan b’ is always necessary for this approach as
sometimes the tools and processes do not work as expected. The fun new tool may not work as expected so
you need to have a fallback of what was done the previous year available and working as an alternative.
The other possibility is that while your course is progressing, other courses are causing difficulties for the
students. The symptom appears when everyone is having difficulty achieving the desired functionality
required for better assessments by the deadline, because they are busy working on competing assessments
for other courses. There are several solutions to this problem.
First, where possible, reduce this risk at the start of term by agreeing with the other course organisers that the
students can use the same basic assessment for both courses. While we have not introduced synoptic
assessments (SA) and synoptic learning per se, we do allow for a loosely linked version of SA. (Kyaw and
Drummond, 2007) For example if one course is about web applications, and another covers security for web
applications, then the security components can be assessed in that course, while the general web application
is assessed in the other. This allows students to spend their time working on ‘both’ simultaneously to a large
extent. They can also use what they learn in one course in another course quite openly and not have to build
232
similar applications twice. This is something, which we should look at more closely in the future as a means to
reduce the work required of the students.
Second, if aligning the assessment with what is needed in other courses does not work, then the rubric, or
marking scheme may need to be changed. This could be due to students having problems completing parts of
the assessment due to issues with the components, or due to work in other courses causing problems.
In any case, be aware that the rubric may need to be changed due to external events beyond your control.
This happens and you need to have plans in place to accommodate these eventualities.
9. CONCLUSION
This approach works and will be continued in the future. This approach pulls together a real-world approach to
coursework, while pushing the students to stretch the boundaries of what they have done previously. They
leave as changed students more aware of what they can do with their abilities, and how to resolve technical
and design problems that will be useful in other courses as well as after they graduate. All students find
CS5302: Enterprise Programming difficult and challenging, but they also think it is one of the best courses on
the programme.
10. REFERENCES
[1] Kahn, P and O’Rourke, K (2004) Guide to Enquiry-Based Learning, University of Manchester)
[2] Kyaw P., and Drummond S., (2007) Synoptic learning and assessment: An experience report, in the
8th Annual Conference of the Higher Education Academy, University of Southampton, 28th - 30th August
2007.
[3] O’Kelley (2005) Designing A Hybrid Problem-Based Learning(Pbl) Course: A Case Study Of First
Year Computer Science in Nui Maynooth, NUI, Maynooth
[4] Schank, Roger (2005) Lessons in Learning, e-Learning and Training. Pfeiffer.
233
INVESTIGATING THE USE OF WEB 2.0 TOOLS AND OPEN
EDUCATIONAL RESOURCES FOR ICS
Tina Wilson Giselle Ferreira

The Open University The Open University
Walton Hall Walton Hall
Milton Keynes Milton Keynes
Martina.Wilson@open.ac.uk g.m.d.s.ferreira @open.ac.uk
http://iet.open.ac.uk/people/view- http://www.cands.org/Home/people/gisel
profile.cfm?staff_id=martina.wilson le-ferreira
ABSTRACT
The availability of Web 2.0 tools together with associated Open Educational Resources (OER) enables the
creation of new collaborative online learning spaces. This paper discusses initial findings from the Higher
Education Academy (HEA) Information and Computer Sciences (ICS) funded project: ‘Using Open
Educational Resources and Web 2.0 Tools to support Ethical Reasoning in ICS Project-Based Learning’. The
project started officially in December 2009 (students joined in February 2010) with the aim of investigating the
potential of Web 2.0 tools and OER to support students engaged in final year project-based learning at the
Open University. Capitalising on existing online environments and tools as well as available learning
resources in the area of ethical reasoning in ICS, this project explores the potential of openly and freely
accessible Web 2.0 tools to provide a space where students can interact with each other and their tutor to
discuss concerns that arise within their project work including those related to ethical issues. This intervention
was planned as existing arrangements that support communication between tutor and distance learning
students (the course website, separate closed tutor group forums and email) appeared to be insufficient to
facilitate the necessarily intense episodes of interaction required for productive supervision. This paper
discusses students’ views of social networking tools and seeks to shed some light on which tools they would
prefer to use for formal learning.
Keywords
Open Educational Resources, Ethical Reasoning, Web 2.0 tools, Project-based Learning
1. INTRODUCTION
The project discussed in this paper investigates ways of using Web 2.0 tools to provide further support to
distance learning students engaged in project-based work at the Open University. In terms of OER the focus
is on the area of ethical reasoning. The project is using the community-oriented facilities available in
OpenLearn [1] the United Kingdom (UK) Open University OER repository. These facilities include the
OpenLearn twin sites LearningSpace (http://openlearn.open.ac.uk) and LabSpace
(http://labspace.open.ac.uk), which provide social networking tools that can be 'described as integrating rather
than integrated' [2].
The community building features discussed in this paper include video conferencing, online journals and mind
mapping tools. The association of collaborative tools with OER has been advocated [3]. In addition to these
integrating tools, the sites also offer ‘learning clubs’, an integrated feature designed as part of the evolving
OpenLearn environment. A learning club starts from a premise of a space where social interaction is
encouraged through Web 2.0 tools and the OER content can be drawn or pulled into that space. This project
also fits within the Open Learning network (OLnet olnet.org), which is undertaking research into design, use
and reuse OER in order to share the findings with a worldwide audience.
234
The project was proposed as a pilot study involving volunteer students drawn from one tutor group in the 2010
cohort, hence constituting a piece of action research that should inform future developments in a course that
has counted on a total cohort of 185 students in 2009 alone. This project also draws upon student feedback
gathered in 2008 and 2009. Furthermore, given the continued deployment and increased use of OER and
online environments, as well as the rapidly growing student engagement with Web 2.0 tools in contexts not
necessarily considered ‘educational’, it is expected that the findings will be potentially relevant to both
campus-based and distance teaching institutions of project students in ICS.
In addition, the project capitalises on the OpenLearn unit Introducing Ethics in Information and Computer
Sciences, published in 2009 (http://openlearn.open.ac.uk/course/view.php?id=3990). Appropriate elements
from this unit that suited the requirements of M450 were identified for reuse in this project.
2. THE COMPUTING PROJECT

The Open University course M450 The Computing Project
http://www3.open.ac.uk/study/undergraduate/course/m450.htm) is a level 3, 60 CATs points course taught at
a distance over a period of 9 months. Before undertaking their project students need to have studied at least
one level-3 course. The students involved in this project have (in most cases) previously taken the third level
course in Human Computer Interaction M364 Fundamentals of interaction design
(http://www3.open.ac.uk/study/undergraduate/course/m364.htm) and base their project in the main on that
course. M450 provides opportunities for students to put into practice the knowledge gained in previous
courses as well as their professional experience, which equates the course to a final year project in a face-to-
face setting.
M450 is centred on a piece of project work in which students are expected to gain ‘practical experience of
independent learning and reflective practice (…) [and] apply advanced principles and techniques to solve
problems’ [4]. The course materials are all presented online on a dedicated, password-protected Web site.
Tutorial support is also provided entirely online to groups of 6-8 students, each working in their own chosen
topic or area. In addition to prompting and moderating discussions and providing individual, tailor-made
support to students, tutors also mark and provide feedback on the 3 pieces of summative assignment
completed throughout the course (Tutor-Marked Assignments or TMAs) as well as their examinable
component, the project report (End-of-Course Assignment or ECA).
3. RESEARCH APPROACH
This is a small scale investigation that aims to draw upon students’ views as an essential element to inform
teaching practice and course development. The project is a pilot study adopting an action research approach
[5]. The tutor is acting as a participant observer and evaluator, working along side a collaborator and external
evaluator. Data gathering methods include semi-structured interviews, informal surveys, observational
techniques and data analysis adopts an approach based on grounded theory [6].
At the start of the project (December 2009), approval was sought from relevant Open University committees
so that students could be approached to take part in the project (based on their previous recent involvement
in research). Five students (out of eight in the tutorial group from the 2010 cohort) agreed to participate in the
project. The number of students involved from the 2008 cohort was five students and the number involved
from the 2009 cohort was four.
Participants were surveyed regarding their preferences and attitude to using Web 2.0 social networking tools
for their course work. The discussion draws on data collected from three presentations (student cohorts in
2008, 2009 and 2010). The data was collected from a tutor’s own students through a short email survey sent
after the students had submitted their examinable component in October 2008 and 2009 respectively and
from the present cohort when they started their projects in February 2010.
Data collection and analysis regarding the students’ actual usage of Web 2.0 tools and the ethics OER is
ongoing at the time of writing.
4. PRELIMINARY FINDINGS
The aim was to allow geographically spread distance learners, in particular, who do not have face-to-face
interaction, the opportunity to choose communication media and methods that suited the types of interaction
they prefer to engage in.
235
After submitting their examinable component, students in the 2008/09 cohorts were asked (by their tutor) to
report their preferences in relation to the types of Web 2.0 tools they would like to have used to help them
collaborate with their tutor and other students on their course. They were asked to indicate their preferences
by typing in 'yes' or 'no' next to each of the descriptive options below (presented to them in a table) and asked
to make additional comments.
Might any of the suggested facilities below help you with your study of M450:
 Use of some sort of learning club for students;
 Use of video conferencing between yourself and your tutor;
 Use of video conferencing between yourself and other students;
 Use of video conferencing between yourself, other students and the tutor together;
 Use of instant messaging;
 Use of a private online diary or blog;
 Use of a public online diary or blog;
 Use of mind mapping tools.
Initial analysis suggests that the results for 2008 and 2009 cohorts were fairly similar across the eight options
offered. The one area where there was a difference of opinion was on ‘Use of video conferencing between
the student, other students and the tutor together’. In 2008 only one student out of 5 made this selection
whereas in 2009 three out of four students made this choice. Taking the results from the 2008 and 2009
cohorts of students together the overall results suggest that the majority of the students would prefer to use
the following Web 2.0 tools to assist them with their final year project based work:
 mind mapping tools (seven students out of nine);
 some sort of learning club for students (six students out of nine);
 one to one video conferencing with their tutor (five students out of nine);
 a public online diary or blog (five students out of nine).
The slightly less popular Web 2.0 tools to assist students with their final year project based work were:
 video conferencing between a student, other students and the tutor (four students out of nine);
 instant messaging (four students out of nine);
 a private online diary or blog (three students out of nine).
The Web 2.0 tool, which appeared to be fairly unpopular (one out of nine students) in terms of assisting
students with their final year project based work, was use of video conferencing between students and other
students (without tutor input).
The 2010 cohort of students were sent the same survey though at the start of the year which differed from the
2008/09 cohorts who were approached at the end of their course. In 2010 the majority of students would
prefer to use the following Web 2.0 tools to assist them with their final year project based work:
 one to one video conferencing with their tutor (four students out of five);
 instant messaging (four students out of five);
 video conferencing between a student, other students and the tutor (three students out of five);
 a private online diary or blog (three students out of five).
The Web 2.0 tools that were less popular were:
 mind mapping tools (two students out of five);
 video conferencing between a student and other students (one student out of five);
 a public online diary or blog (one student out of five).
The one area of common agreement in relation to student preferences for tools to help them with their study
of M450 (between the 2008/09 and 2010 cohorts) is in terms of one to one video conferencing with their tutor.
Indeed this was a much more important option for students in 2010. Perhaps this is an unsurprising result as
distance learners would want to increase their contact with their tutor. However video conferencing between
the student, other students and the tutor was also important to both the 2009 and 2010 cohorts. The students
(across the three cohorts) indicated a lack of interest in collaborating with other students if the tutor was not
236
involved. Unexpectedly, the 2010 students did not appear to be interested in using either a learning club or
mind mapping tools which had been important selections for the 2008/09 students. The 2010 students
preferred to use a private online diary or blog as opposed to a public diary or blog when compared to the
2008/09 cohorts of students. These findings indicate that a wide variety of Web 2.0 tools are required to suit
the needs of different student cohorts and that students select a small number of Web 2.0 tools for formal
learning. This finding is in agreement with other authors [7] who report that the use of social networking tools
for undergraduate course work as compared to personal usage (age groups 18 to 40 plus) is much more
limited. The majority of respondents in this earlier survey [7] though from the Net Generation, report that they
prefer only a modest amount of technology use in their courses.
The M450 course itself started in February 2010. As indicated above students were keen to engage with their
tutor in one to one video conferencing which started almost immediately. A learning club was then set up as a
space with a selection of alternative tools to encourage social interaction around the re-purposed ethics
materials. The club forum was used to indicate types of activity to students and students could choose tools
appropriate to their needs.
Students gradually became familiar with the features and tools that they decided to use. Video conferencing
has been adopted for intense episodes of interaction between student and tutor on a one to one basis (in line
with the students’ preference indications in 2010). Students did not find it difficult to access and use the video
conferencing system with minimal guidance. Certainly too authors [8] propose that technological barriers are
not necessarily presenting a problem; rather, they are encouraging interaction. The learning club (though not
of interest to the students in terms of preference) has been used by at least four of the five students to access
the additional re-purposed materials on ethical issues which they have used for part of the work they were
nd
required to do for their 2 assignment for the M450 project course. Usage of the ethics OER has been on an
individual basis initially though this echoes usage of OER by the single learner and for self-regulated learning
[9]. The 2010 cohort of students have shown an interest in working with other students as long as the lecturer
is present. ‘Social networks arise around common (learning) interests ...' [10] and students may find generic
aspects of their projects that they would like to discuss with other students.
5. PRELIMINARY CONCLUSIONS
This is still an early stage in the project and the findings are tentative and need more in-depth analysis. There
are interesting differences in students’ expectations/preferences across cohorts. The initial overall result from
both cohorts in 2008 and 2009 suggests that the majority of the students would prefer to use: mind-mapping
tools, learning clubs, one to one video conferencing with their tutor, and a public online journal or blog. In
2010 students’ preferences are for video conferencing with their tutor and other students, a private online
journal or blog and an instant messaging facility. Preferences for video conferencing either between the tutor
and individual students or between the tutor and a number of students varied between the cohorts. The 2010
cohort of students did not display a particular interest in using a learning club.
In exploring, in particular, the learners’ perspective of using OER within Web 2.0 online learning spaces, the
project findings should be of special value as this is an area of research that is only in its infancy. This
research (as it develops) will influence future deployment of Web 2.0 tools associated with OER content for
project based learning at The Open University.
Acknowledgements
Thanks to Dr Patrick McAndrew for commenting on the original bid.
The work discussed in this paper is supported by a number of projects:

Using Open Educational Resources and Web 2.0 Tools to support Ethical Reasoning in Information and
Computer Sciences Project-Based Learning is supported by a grant from the Higher Education Academy -
Information and Computer Sciences subject group in the UK.
The OpenLearn unit Introducing Ethics in Information and Computer Sciences, was developed by Giselle
Ferreira and Professor John Monk with the support of a grant from the Higher Education Academy -
Information and Computer Sciences subject group in the UK in 2008-2009
(http://openlearn.open.ac.uk/course/view.php?id=3990).
OpenLearn and OLnet have both received funding from the William and Flora Hewlett foundation.
237
6. REFERENCES
[1] Open Content Initiative (2006). Application to the William and Flora Hewlett Foundation. [accessed
03/08/2007] http://www.open.ac.uk/openlearn/__assets/06sngpqpwminsmwxov.pdf
[2] Culwin, F. & Lancaster, T. (2004) An integration suite of tools to assist investigation of non-originality. In
Proceedings of the Plagiarism: Prevention, Practice & Policy Conference. St James Park, Newcastle upon
Tyne 28-30 June. Newcastle: Northumbria University Press, pp 49-55. Available online at
http://www.jiscpas.ac.uk/documents/papers/2004Papers06.pdf [Accessed 20 August 2009]
[3] Smith, M. S. and C. M. Casserly (2006). The Promise of Open Educational Resources Change 38(5): pp.
8-17.
[4] Open University, (2009). Course information for M450 - The Computing Project. Available from the Open
University website: http://www3.open.ac.uk/study/undergraduate/course/m450.htm
[5] Somekh, B. (2006) Action Research: a methodology for change and development. Maidenhead: Open
University Press
[6] Strauss, A. & Corbin, J. (eds.) (1997) Grounded Theory in Practice. Thousand Oaks, California: Sage
Publications
[7] Smith, Shannon D., Salaway, Gail and Borreson Caruso, Judith with an Introduction by Katz, Richard N.
(2009). 'The ECAR Study of Undergraduate Students and Information Technology, 2009. (Research Study,
Vol. 6). Boulder, CO: EDUCAUSE Center for Applied Research, 2009. Accessed 13th November, 2009 from
http://www.educause.edu/Resources/TheECARStudyofUndergraduateStu/187215
[8] Tallent-Runnels, M.K., (2005). The relationship between problems with technology and graduate students'
evaluations of online teaching. Internet and Higher Education, 8, 167-174. Accessed 5/12/09 from
http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6W4X-4G9R2G5-
1&_user=126980&_rdoc=1&_fmt=&_orig=search&_sort=d&_docanchor=&view=c&_searchStrId=1127376395
&_rerunOrigin=scholar.google&_acct=C000010439&_version=1&_urlVersion=0&_userid=126980&md5=9c8f
acf1b0389b837bff7525e0d3e88a
[9] Remmele, Bernd and Seeber Günther (2009). From Teaching to Testing – A Business Model for OER. In
proceedings of the EDEN conference - Innovation in learning Communities. 2009 EDEN Annual conference,
Gdansk, 10-13th June 2009
[10] Redecker, Christine (2009). ‘Review of learning 2.0 Practices: Study on the Impact of Web 2.0
Innovations on Education and Training in Europe’. Accessed 20th July 2009 and available from
http://www.twine.com/item/127zdk2m5-37w/review-of-learning-2-0-practices-study-on-the-impact-of-web-2-0-
innovations-on-education-and-training-in-europe
238
VIRTUAL WORLDS FOR COMPUTER SCIENCE EDUCATION
Alan Miller, Colin Allison, John McCaffery, Thomas Sturgeon, Ross Nicoll,
Kris Getchell, Indika Perera, Iain Oliver
School of Computer Science

University of St Andrews
KY16 9SX
{alan, ca, jm726, kg, tjs, jrn, indika, iao}@cs.st-andrews.ac.uk
ABSTRACT
Virtual Worlds offer the promise of both widening and enriching the educational scenarios within which
exploratory learning may be practiced. They empower students to explore virtual contexts that would be
inaccessible to them in the real world. Learners achieve presence through the proxies of avatars, and
consequently are aware of and may interact with fellow learners within the virtual environment. The
geography and architecture of virtual worlds may be defined and programmed by users, thereby allowing
learners to participate in a creative process online. This paper reports on experiences of working with virtual
worlds for Computer Science education. These include the areas of human computer interaction, wireless
networking and Internet routing. The work was conducted over three years and involves both the popular
virtual world Second Life and its open source alternative, OpenSim. We find that virtual worlds are engaging,
popular and motivating. Students enjoy using them and find them educationally valuable. Furthermore, virtual
worlds provide a rich environment, through which content, presented on media such as text, images, video
and animations, may be collaboratively explored. Interactivity ranges from simple exhibits as might be found in
a museum to configurable, complex simulations visualised in a 3D space. This paper identifies a number of
challenges that need to be met for the educational potential of virtual worlds to be unlocked. These reside in
the areas of system development, content creation and service provision. We argue that on balance the
service model adopted by Second Life acts as a barrier to these challenges being met and that the greater
flexibility provided by OpenSim offers a more potential.
Keywords: Computer Science Education, Virtual Worlds, Virtual Laboratory, Exploratory Learning
1. INTRODUCTION
This paper discusses our experiences in designing, developing and using educational resources to support
exploratory learning (Allison 2007) within Computer Science. The use of virtual worlds in educational contexts
(Thomas 2006) is likely to continue to expand as experience in the area grows and as commodity computer
systems improve in their handling of 3D immersive graphics.
The paper starts by identifying the key characteristics that make virtual worlds interesting from an educational
perspective. Our practical work with virtual worlds has involved Second Life, the most popular virtual world,
which is run as a commercial service by Linden Labs, and OpenSim, a freely distributed open source virtual
world, which may be accessed using Second Life clients.
Human Computer Interaction (HCI), routing and wireless networking are the educational contexts reported in
this paper. Virtual worlds were used by modules in Human Computer Interaction (Fig. 1) and Data
Communications and Computer Networks (Fig. 2), as part of the BSc. Computer Science degree program at
the University of St Andrews. The main contribution of this paper lies in the identification of the benefits that
accrue from virtual worlds and the challenges that need to be met for these benefits to be widely realised.
These were arrived at through reflection on our use of Second Life and OpenSim. Three challenges are
identified:
 System Development: to facilitate using virtual worlds as a platform for application development;
 Content Creation: to enable users to easily populate a virtual world with domain specific material;
 Service Provision: to allow institutions to efficiently support scalable provision of virtual worlds.
239
© 2010 Higher Education Academy Subject Centre for Information and Computer Science
Figure 2: A Simulation of Dijkstras shortest path algorithm used for Internet routing and a view of WiFi Island
2. VIRTUAL WORLDS
Virtual Worlds offer new possibilities for supporting exploratory learning. As a programmable multimedia
platform - supporting video, sound, pictures and animation - virtual worlds make access to a rich set of
educational resources possible. They provide an intuitive 3D environment where users are represented by
avatars which are navigated through the 3D space, much as a group of students might explore a park, art
gallery or museum. The sense of presence is engaging as users are able to interact with the surrounding
environment and each other. This in turn provides natural support for group work and collaboration.
Virtual Worlds differ from multi-player online games in that there are no predefined goals that need to be met
for progress to be made. Users are free to think up there own goals and to create their own content. This
flexibility lends them to being utilised by educationalists who can define goals that are appropriate for the
discipline and activity in question. The generality that virtual worlds offer comes with the need to use more ICT
resources. For example virtual worlds require ten times the network bandwidth used by online multi-player
games.
Within Second Life and OpenSim, “residents” are able to dream up and create content. This extends beyond
the ability to design and build objects, to shaping terrain, constructing buildings and designing clothes.
Through the Linden Scripting Language, LSL, (Linden 2009), users are able to program the world making it a
truly interactive experience. Intelligent tutors may react to stimuli; learners may use interactive exhibits in an
in-world museum or run full blown virtual experiments in virtual laboratories.
The same family of software clients can be used to access both Second Life (Linden 2003) and OpenSim
(OpenSim 2010). Avatar movement, clothing adjustment, building construction and scripting can all be
achieved using the same gestures and commands in both environments.
There are important differences in the way that the Second Life and OpenSim services are provided. Second
Life is run as a commercial service by Linden Labs. In order to create landscape or construct buildings real
estate must be rented on an annual basis. In addition, an initial down payment (set-up charge) must be paid.
Within Second Life there is an internal market where buildings, clothing and other objects may be bought and
sold using Linden Dollars, which in turn can be purchased using real money. This economic model brings
restrictions on copying, loading and saving assets outside of the virtual world. Options for running private
Second Life servers locally have been offered recently by Linden Labs, but these are prohibitively expensive
for educational use.
OpenSim is open source, freely distributed, supported by the Mac, Linux and Windows operating systems and
will run on of-the-shelf commodity hardware. An OpenSim server may be run as a stand-alone area of “land”,
as part of a grid linking together a collection of private servers, or as a server which is connected to a wider
external grid. In each case it is possible for an administrator to load and save entire “Islands” as oar
(OpenSim archive) files. Buildings, images and other objects may be freely uploaded or saved to disk, by
ordinary users through the file menu. Currently, the cost of renting a single Second Life Island for a year
would purchase a server capable of running a dozen OpenSim Islands.
240
3. EDUCATIONAL CONTEXT
We have used Second Life and OpenSim virtual worlds in three parts of the curriculum: HCI programming
assignments; 802.11 simulation; and Internet routing. As part of their HCI coursework students designed
systems in Second Life in AY 2008-9 and OpenSim in 2009-10. In order to support exploratory learning about
wireless networking a WiFi virtual laboratory was developed in Second Life in AY 2008-9 and then expanded
into OpenSim in AY 2009-10. The most recent ongoing development is an Internet routing simulator and
visualiser. We briefly describe these three case studies.
3.1 HCI Assignment

In a BSc. final year HCI module students were asked to design interactive machines for learning about
Dijkstra’s shunting algorithm in Second Life (Perera 2009). The following year the next cohort of HCI students
were asked to design a structure with at least two interactive doors in OpenSim. In both the Dijkstra practical
in academic year 2008-9 and the doors practical in 2009-10 impressive originality was shown and a significant
amount of work went well beyond that asked for in the practical.
In OpenSim each student was assigned a parcel of land within a dedicated ‘HCI’ region which gave them a
space within which they could build and script with minimal restrictions imposed on them. Each parcel was
2500 sq. m (50m x 50m), with exclusive ownership for the assigned student. This was approximately 10
times more space than it was possible to give them in Second Life. In both Second Life and OpenSim the
ability of users to build on their land and to program their creations was central to the success of their
coursework. The extra space available in OpenSim gave the students more resource to express themselves.
In addition, OpenSim offers administrators much greater control over permissions than Second Life. This
allowed the assessment process to run much more smoothly than in the Second Life case.
3.2 WiFi Virtual Laboratory

The WiFi Virtual Laboratory (Sturgeon 2006/2009) uses ns2 (ns2 1995) for generating discrete event
simulations. The original version provided web-based interfaces for specifying a scenario of protocol
interactions and then watching the resulting simulation played back in 2D graphics via Flash; the virtual world
versions implemented set-up and playback interfaces in Second Life (WiFiSL) and OpenSim (WiFiOS). In the
first virtual world version, based in Second Life, the limitations of a small parcel of land on the shared, rented
Island, effectively made the WiFi virtual lab seem like a sideshow. However, in the case of WiFiOS OpenSim
really came into its own. The removal of the land-space/cost limitations of Second Life allowed WiFiOS to be
expanded into a dedicated region: Wireless Island. The island acts as a portal to open courseware, standards
documents and research papers related to wireless networking. A movie theatre allows students to view
streamed videos, a lecture theatre shows presentations from an archive, a museum illustrates the history of
wireless networking and interactive animations allow learners to explore wireless networking challenges such
as the hidden node problem. WiFiOS has been evaluated with a cohort of final year undergraduate students.
The set-up encouraged students to take control of their own learning. They could navigate between resources
as they pleased, rather than being expected to follow a linear path. The students found it highly engaging and
educationally valuable.
3.2 Routing Algorithms

The third Computer Science resource is an interactive simulation of routing algorithms. As with the WiFi Island
the virtual world acts as a 3D portal providing access to videos, sound clips, animations, presentations,
lecture notes, research papers and standards, all about routing. At the centre of this learning experience is a
3D interactive routing simulation. This allows learners to construct network topologies, and observe how
network routing algorithms work in determining shortest paths and routing tables. The flow of packets across
the network can be observed and the effect of topological changes experimented with.
3.4 Summary
In each of the above cases the potential for enhanced presence, the support for multimedia and the
programmability of the virtual world, enriched the specific subject matter. In effect, the virtual world is used as
a portal into a collection of learning resources and as such the approach is generic. Many subjects, both
within Computer Science and more widely, would benefit from such portals. This is relevant to the growth of
open courseware repositories as there are many multimedia learning resources available for deployment
within such portals.
241
4. TECHNICAL CHALLENGES
Whilst educational virtual worlds are educationally valuable there are technical challenges that need to be met
if their use is to move from the experimental edges of education into the mainstream of college life. In order to
facilitate the routine use of virtual worlds in education certain conditions should be met:
 relatively complex software projects can be set as coursework and realistically be achieved;
 educationalists can create content, and easily move content into and from the virtual world;
 the service can be provided efficiently and affordably.
In the case of the routing learning resource a routing simulator had to be written and integrated within the
virtual world. In achieving this two challenges had to be met. Firstly, the programming environment provided
by Second Life protects the system from the programmer. LSL (LSL 2009) scripts are interpreted by a
DotNetEngine and the resources the script can use are limited. Secondly, the software development support
is very limited, making it difficult to develop programs of any size or complexity.
Figure 3: Architecture for Routing Simulation and Spectrum of Programming Paradigms

.
These challenges were not resolved within Second Life, however a solution was achieved using OpenSim.
OpenSim contains the LSL scripting facilities. It also has Mini Region Modules (MRMs) and Region Modules
as illustrated in Fig. 3. MRMs provide an API which unlocks the power of the virtual world for the programmer,
whilst providing an interface which will be respected in future OpenSim releases. Fig. 3 shows how MRMs
can be linked to an in-world script. All but the in-world stubs can be developed using off world tools, including
IDEs, debuggers and source control systems. Thus software development is much better supported in
OpenSim compared to Second Life.
Second Life restricts the saving and loading of images and other objects, severely limits the amount of data
that can be sent or received as one protocol interaction from external locations and prevents the wholesale
saving of regions to back-up storage. There are no such restrictions in OpenSim. The possibility of bringing
OpenSim and open courseware together into a toolkit which allows educationalists to easily create spaces
and populate them with content, holds out promise to unlock the potential of virtual worlds.
The ability to run OpenSim locally, within a departmental or institutional firewall opens the possibility of easily
installable sandboxes being created and distributed. Educationalists will be able to click on an icon, fire up
their local copy of a virtual world, load in a template island, do some tweaking and then populate it with
educational content. A lab session may be prepared in an afternoon and uploaded to a server, or a 3D
presentation made on a laptop and used during a lecture session.
Virtual worlds are quite hungry in their resource requirements; in addition usage patterns are bursty within an
educational setting. When a subject is being studied, there are likely to be high peaks in usage. For example,
during a laboratory session which involves using the Island, or in the run up to a coursework deadline where
the Island is to be used. On these occasions over 50% of the cohort would want to be online at the same time.
At other times there may be zero, one or two people using the resource consecutively. Given this bursty
usage pattern the Second Life model of resource provision does not lend itself to efficient utilisation. A space
of land is “rented” for the duration of a year, and restrictions on saving and loading resources in and out of
242
Second Life make re-purposing land cumbersome. Consequently the resources that could be utilised at peak
periods were constrained to what could be afforded for long term rent. Running local copies of virtual worlds,
facilitates efficient resource utilisation. A novel challenge is how to efficiently provision virtual worlds in the
Cloud whilst meeting system and user Quality of Service (Shelden 2003) requirements.
5. CONCLUSION
Virtual worlds continue to attract considerable interest as an innovative means of supporting exploratory
learning and promoting student engagement through the use of immersive, 3D, collaborative environments.
They provide a shared environment that allows for the dynamic creation of content and for that content to be
programmed. They can provide an interactive learning environment for students to explore, thereby
supporting many aspects of socio-constructivist learning including collaboration, exploration and
communication. A wide variety of learning contexts including virtual laboratories, networking test beds,
interaction design and team projects can be provided for using both blended and remote modes of teaching
and learning across a range of Computer Science topics.
This paper has briefly reported on our experiences in designing, building, deploying and evaluating virtual
world scenarios for Computer Science education. Three specific contexts were described; a virtual laboratory
which supports exploratory learning about wireless network protocols; a tool which enables students to
investigate routing algorithms; and the use of virtual worlds as a milieu for programming coursework in Human
Computer Interaction. In all three cases the OpenSim based virtual world has allowed for much greater
expressiveness and creativity by the students.
Three significant challenges which need to be met if the potential of educational virtual worlds for Computer
Science are to be realised have been identified. These relate specifically to: support for the development of
complex software and systems, support for content creation and management without the virtual world, and
effective and affordable service provision. In all three of those challenges our experiences to date have
shown OpenSim to be a superior virtual world system.
SL is a ready-made service provided as a commercial product to end users. In contrast OpenSim is a free,
open source alternative, which enables users to run their own virtual worlds. We believe that the connection of
open source virtual worlds with open coursework offers the prospect of unlocking the educational potential of
virtual worlds.
Finally, many educational activities are similar across subject domains. Consequently, “template” spaces
designed to meet generic educational needs, which can be loaded into a virtual world, can save much
duplication of effort. The templates might include museums, parks, theatres, cinemas and tutorial rooms
arranged in various “island” configurations. These islands could then be edited to meet specific needs.
Having defined a space it remains for it to be filled with educational content. Interestingly, much content
already exists, for example from various open courseware initiatives and repositories. This content could
conceivably be loaded into OpenSim virtual worlds, and while enriching a 3D lmmersive learning environment,
can also offer a new means of structuring open source learning object repositories.
Acknowledgements
The work reported in this paper has been supported by the HEA-ICS Development Fund, and by FILTA, the
University of St Andrews Fund for Initiatives in Learning, Teaching and Assessment. The authors gratefully
acknowledge this support.
References
C. Allison, A. Miller, K. Getchell, and T. Sturgeon, "Exploratory Learning for Computer Networking," in ICWL 2007,
Lecture Notes in Computer Science, vol. 4823/2008, pp. 331-342, 2008.
Linden_Labs, "Linden Scripting Langauge, http://secondlife.com/whatis/scripting.php." accessed 2009.
Linden_Labs, "Second Life, www.secondlife.com." accessed 2010, 2003
ns2, "The Network Simulator, version 2.", http://www.isi.edu/nsnam/ns/index.html, accessed 2009, 1995.
Oliver, A. Miller, and C. Allison, "Virtual Worlds, Real Traffic: Interaction and Adaptation," in ACM Multimedia Systems
2010, Scottsdale, Arizona, 2010.
The_Open_SImulator_Project, "OpenSim: http://opensimulator.org/wiki/Main_Page," accessed 2010.
G. I. U. S. Perera, C. Allison, T. Sturgeon, and J. R. Nicoll, "Towards Successful 3D Virtual Learning - A Case Study on
Teaching Human Computer Interaction," in 4th International Conference for Internet Technology and Secured
Transactions, London, 2009.
243
N. Sheldon, E. Girard, S. Borg, M. Claypool, and E. Agu, "The effect of latency on user performance in Warcraft III.," in
NetGames ’03: Proceedings of the 2nd workshop on Network and system support for games 2003, pp. 3-14.
T. Sturgeon, C. Allison, and A. Miller, "802.11 Wireless Experiments in a Virtual World," ACM SIGCSE Bull., vol. 41, pp.
85-89, 2009 2009.
T. Sturgeon, C. Allison, and A. Miller, "Exploring 802.11: Real Learning in a Virtual World," in 39th IEEE Frontiers in
Education Conference, 2009. San Antonio, Texas, pp. T1A-1, T1A-6.
T. Sturgeon, C. Allison, and A. Miller, "A WiFi Virtual Laboratory," in 7th Annual Conference of the Higher Education
Academy Subject Centre for Information and Computer Sciences, Dublin, Ireland, 2006, pp. 207-211.
L. D. Thomas and P. Mead, "Implementation of Second Life in an Electromagnetic Theory Course," in 38th IEEE Annual
Frontiers in Education Conference, 2008. Saratoga Springs, NY, pp. F3D-3-F3D-4
244

Proceedings 11th Annual Conference

Enviado por

Dados do documento

Descrição original:

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

Proceedings 11th Annual Conference

Enviado por

Direitos autorais:

Formatos disponíveis

11th Annual Conference of the Subject Centre for

Information and Computer Sciences

Conference Organizing Committe

Local Organising Committee

Janet Lavery and Liz Burd, Durham University.

The Problems Of Teaching Programming: Do They Change With Time? 6

Using The Principles Of Variation To Create Code Writing Problem Sets 11

A Mad Approach To Java Concepts 17

Supporting The Learning Of Object Oriented Design With A Lightweight Web-

Towards an Interactive Multimedia-based Environment for Teaching Academic

On Second Life As An E-Learning Environment For Teaching Academic Writing 32

Education in Library Management Systems (ELMS) 38

Inquiry-Based Learning And Information Literacy With International Masters

Enhancing Student Engagement Through Pair Programming 47

Teaching Media And Information Literacy To Postgraduate Researchers 52

Anytime-Anywhere Student Support Through Virtual Mentors 58

Enhancing Student Engagement And Support With Digital Video 73

Tags As A Feedback Mechanism In Programming Courses: Analysis To Support

Turnitoff – Defeating Plagiarism Detection Systems 84

Transition, Engagement And Retention Of First Year Computing Students 90

Re-thinking Student Induction 96

Widening Access, Cpd And Work-Based Learning Using Asynchronous Dialogic

Engaging Schools With Computer Programming 106

Listening Step By Step 112

Sustaining Sustainability: Developing Material For Higher Education 117

Developing Sustainable International Partnerships: A Case Study Of The Indian

Cutting Wedge Or Cutting Edge? - Embedding E-Learning For Engagement 128

Introducing A.L.L. To Computing 134

Engaging The Attention Of Technical Students In Learning Business Skills 147

Technology Enhanced Learning: Systems To Increase Engagement &

Investigating ‘Voice Email’ Technology Efficacy In Information Management

The Impact Of Peer Assessment On Academic Success 167

Staff Led Individualised Assessment – A Case Study 173

The Influence Of Presentation Upon Examination Marks 178

An Evaluation Of The Durham Tec Mobile Educational Software System In The

The Use Of Simulation In Digital Forensics Teaching 187

Sharing Practice For Computing Educators 193

Facilitating Reuse Of Learning Resources: A Tool To Support Self-Deposit And

Fostering Social Competence, Community Awareness And Employability Skills

Virtual Worlds For Computer Science Education 239

Janet Carter Tony Jenkins

3.1 Start From Where They are at

3.2 Differentiation in the Classroom

3.4 Language Wars?

 Ben-Ari M, (2001), Constructivism in Computer Science Education, In Journal of Computers in

 Birch M, McCormick F and Haddo J, (2005), Improving Student Progression by a combination of

 Fuller U, Johnson C, Ahoniemi T, Cukierman D, Hernan-Losada I, Jackova J, LahtinenE, Lewis T,

 Hamer J, (2004), An Approach to Teaching Design Patterns using Musical Composition, In

 Jenkins T, (2001), Teaching Programming – A Journey from Teacher to Motivator, In proceedings of

 Jenkins T, (2003), Pragmatic Possibilities of Pre-Programming with Python, In proceedings of 3rd

 Keenan F and Coleman G, (2003), Extreme Programming: Results of an Educational Experiment, In

 Lawhead P, Bland C, Barnes D, Duncan M, Goldweber M, Hollingsworth R and Schep M, (2003), A

Qualitative Extended Requires knowledge of the Used constructs and

Quantitative Multistructural A line by line description is Requires some Represents a translation

Unistructural Provides a description for Requires a direct translation Represents a direct

Prestructural Substantially lacks Substantially lacks

3.1.1 Unistructural questions and answers

3.1.1.1 Problem: Calculating area of a circle

public double areaOfCircle(double radius) {

return 3.14 * radius * radius;

3.1.2 Multistructural questions

3.1.2.1 Problem: Calculating area of a ring

public double areaOfRing(double outerRadius, double innerRadius) {