Students’ Perceptions of Learning Management Systems

Abstract

This paper reports on the first results of a study to investigate how the usability of

a Learning Management Systems is influenced by the design of the interface and

technical competence of the students. The study was originally designed as a

small-scale study applying a Grounded Theory approach (Glaser and Strauss,

1967) working with students who had experience of two different systems. The

outcomes, even at the open coding stage, were so unexpected that the aim of the

study has significantly changed. Both the usability of an interface and technical

competence of the students appear subservient to more fundamental and systemic

issues that determine a student’s choice of how and where to study. The value of a

grounded approach to research appears to be established.

The conclusions, even at this stage, have already influenced the way in which the

University uses its own LMS. The focus of the study has now changed to a wider

consideration of how students selectively choose between remote learning and

studying at University. The current model suggests that many remote access

learning activities are too different from conventional models for the students to

make an effective comparison. Rather than completing a small-scale model based

on the limited data set that is available, a second phase of data collection has now

been started with sampling directed towards this new issue.

Introduction

This research was designed to review a strategic decision to select Blackboard™

(Blackboard Inc., 2001), rather than WebCT™ (WebCT Inc., 2002), as a learning

management system (LMS) at Kingston University. The study with existing users

was intended to examine issues that would be significant in planning the full-scale

implementation of the LMS across the University. Two key aspects of usability

were identified: whether the interface could be improved through minor changes in

the design and whether a student’s ability to use this LMS successfully was

determined by particular competences developed either before or at University.

Two groups of students who were studying Geographical Information Systems had

experience of using WebCT™ in addition to using Blackboard™. A study that

focused on these particular students offered the potential for a comparative

assessment of these two interfaces by students with a relatively high degree of

technical competence. Some of these students had also had experience of a third,

intranet system.

page 2 of
Each of the systems provided similar functions as a learning environment accessed

through a web portal. This portal offered links to a number of integrated functions

supporting document management, both synchronous and asynchronous

communication, and integration with email services. From a management point of

view an LMS can provide integration with other management functions (e.g.

student records and finance) but this aspect offers little direct educational

advantage to the student. For a student an LMS would appear to offer few technical

advantages beyond those that are available free through other Internet providers

offering support for more generic communities. The educational advantages for a

student are essentially mediated by the design of the interface. The arrangement

and labeling of icons and links to the various facilities control the students’

expectations of what activities they could be engaged in. If the learning

management systems are substantially the same, then any differences will only be

realized if the students succeed in navigating to those services and can recognize

the potential of the additional benefits that are provided. As the educational

effectiveness is controlled through the interface, simple changes in the usability of

the interface could provide significant educational gains. Early experience with

Blackboard™ suggests that there is considerable potential for redesign. Of the

various active icons that are available as standard to a student for a course, more

page 3 of
than half of them can provide identical functionality and are only differentiated by

the word used on the icon.

The Students as the Subjects

This research focused on students studying Geographical Information Systems

(GIS) in their final year of two courses, either a BSc. or HND. These students had

all used Blackboard™ during 2001, WebCT™ during 1999-2000 and, in some

cases, an intranet system (Sci-Net) during 1998-1999. Although they were

technically supported by an LMS at each stage, the nature of the course was still

essentially a conventional undergraduate course with attendance required for

lectures and some workshop activities etc.

Of the systems that were used, Sci-Net primarily provided access to lecture notes

and instructions for practical work. WebCT™, like Blackboard™ offered their

normal suite of services. WebCT™ was only accessible from within the

University. Blackboard™, however, was made available through remote access

with password protection.

Selecting a Research Methodology

The first stage in the research was the selection of a research methodology

appropriate to the outcomes that were required and the constraints of the particular

page 4 of
study. The first requirement was the importance of capturing the students' own

perceptions and understanding of the systems that they were using.

Many research methodologies expect the research to be conducted within an

objective, conceptual model that is framed by those external to the subject group.

Several groups other than the students have an interest in the performance of the

students: the course lecturers, system designers, university managers and even

ourselves as researchers. Such research seldom provides insight into why a system

is not used in the way that is intended. Research that is based on an external

framework may well provide evidence that a system is not working – but will

seldom provide answers to why the system is not working. This requirement

implies that a qualitative approach would be required.

The second requirement was that the methodology had to support the potential of

building a theory about what was being studied. This would enable the results of

the research to inform the next stage of the development in the use of the LMS.

This requirement would exclude any methodologies that did not conform to this

requirement, e.g. ethnography.

A final class of methodologies were excluded by the constraints of the study. Some

qualitative approaches use an initial subject-centered qualitative phase as grounds

to generate relevant questions for a larger statistical analysis. The timing of the

research in two phases would not produce any results over the time scale available
page 5 of
as all the students with experience of both systems would have left by the end of

the year and the group would not have been large enough for a statistical analysis.

In addition, we were not interested in determining an overall general pattern. Our

interest lay in providing a fuller description of the range and variety of student

perceptions and the factors that could be altered to produce different behaviours.

The overall research model selected was Grounded Theory (Glaser and Strauss,

1967; Glaser 1978; Strauss and Corbin, 1990; Miles and Huberman 1994). Alsop

and Tompsett (2002) provides a more detailed discussion of this selection process.

Grounded Theory

The Grounded Theory method requires that the research process is grounded in the

data that is collected. As with most qualitative methods it accepts that the

researcher cannot remain external to the collection and interpretation of data but is

inherently bound up in the process of research. The process of collection and

interpretation is cyclical, with low-level simple analysis of data providing grounds

for simple levels of interpretation, which then provide the basis for reinterpretation

of the data and selection of more data. It is an iterative process that focuses on the

subject's perspective while accepting the researcher's bias and viewpoints. The

implication that the results will be dependent on the researcher’s implicit

involvement is managed through carefully recording of the process of

interpretation, not just as information external to the analysis, but as data within the

page 6 of
project itself. The suggestion that different researchers could produce different

results is openly recognized, not as a problem but as consistency with the

underlying epistemology. The validity of any resulting ‘theory’ is established by

the ability of other researchers to trace the sequence of interpretation.

The development of a theory is therefore cyclical, using existing data to build a

theory that increases the depth of interpretation and the coverage of the theory

across subjects. The first stage is to generate a catalogue of the terms and concepts

used by the subjects (this is called ‘open coding’). This catalogue is then used to

generate a time-sequenced framework (an ‘axial model’) that establishes the

interrelationships between the concepts within the open coding. This develops the

interpretation of the subjects’ perception of the problems that were being solved,

for example the conditions controlling what was possible, the choices that were

open to them and their assessment of what was successful and what was not. When

this appears to be effective in structuring the data that has been collected this

model is then summarized by the identification of a single bounding concept,

termed the core category. This core category possesses the framework within

which all of the subject’s perceptions can be included. In reality there is a constant

cycle between the last two stages as the intention is to develop a theory for which

the core category provides a summary that successfully allows all the data to be

interpreted within the axial model and hence the core category. The axial model

page 7 of
and core category are validated as a theory through a recursive process termed

‘theoretical sampling’. In this phase concepts and issues that are already relevant in

the current axial model are used recursively to search for new cases to be

integrated into the axial model. The new data that is collected may well introduce

new concepts or variables that enrich the open coding stage, provide changes and

complexity to the axial model or even require a reconsideration of the core

category. When this process stops, that is when all the data that has been collected

has been accounted for within the axial model and can be summarized within the

core category, the research is considered to be complete and the model is said to

have reached ‘theoretical saturation’.

This paper covers the first stages of the analysis: the open coding, early axial

coding and considerations for the core category. It is unusual to report at this stage

of analysis. However, the interim results were so surprising that key issues are

significant and have been used to influence the way in which the University uses

its own LMS.

Data Collection

The particular method for generating data was derived from an earlier qualitative

study based on semi-structured interviews (Bliss & Ogborn, 1977) which used

page 8 of
systemic networks (Halliday, 1976) to direct and code semi-structured interviews

with first-year physics undergraduates.

Data was collected from students within a normal lecture session. In each session,

following an introduction that provided the context and assured the students of

anonymity, students were asked to write an account, with as much detail as they

could remember, of the occasion on which the use of one particular LMS had been

‘the most rewarding experience’. As they completed their individual account they

were put into groups of four or five and completed the first phase by summarizing

their own account with a single word or short phrase, reviewing without discussion

the accounts of other students in their group, summarizing each in turn, and then

discussing the accounts collectively between themselves and agreeing a common

word or phrase for the set of accounts within the group. Although this is less

constrained than a semi-structured interview, earlier experiments with a different

topic by one of the researchers have been shown to produce remarkable

consistency between individuals and groups despite starting with individual,

personal accounts. Each of the groups then repeated the cycle with a different

starter question – they were asked to write an account, with as much detail as they

could remember, of the occasion on which the use of one particular LMS had been

‘the worst educational experience’.

page 9 of
If time allowed, students were then asked to contribute in two different ways to

building the model. The first was to ask for a specific comparison of the different

interfaces. The second was to ask them to identify any issues which they or their

group had failed to refer to but which were key issues as far as they were

concerned and which should be included in the model.

This relatively open structure to the collection of qualitative descriptions was

designed to focus the responses on a description that would be framed within

language to be understood by each other. Descriptions written directly for us would

increase the likelihood that their personal accounts would be contaminated by a

number of psychological processes: filtering issues that should not be passed on to

lecturers, second guessing the issues that were ‘what we wanted them to say’ and

re-expressing their ideas in a way that they believed would be understood by

lecturers. Requesting a narrative account provided a description that would link

closely to an axial model. The request for individual summaries provides a check

that the accounts make sense to the students and, with a collective discussion,

provides a starting point for developing a core category that is firmly rooted in the

subject’s perception.

These results focus on the language used by the students and their decisions,

framed within that language, that were relevant when using the different interfaces.

The nature of these results provides clear evidence to validate the original decision

page 10 of
to select ‘Grounded Theory’ as the appropriate research methodology for this

context, even if the answers are not what was anticipated!

Open Coding

The analysis of the full sample collected so far (49 accounts) was conducted using

the software Nudist 4™. This provided effective facilities for recording coding

notes and for tracking progressive sorting and resorting of coding groups, though

considerable effort was needed to compensate for the rigid document structure.

Some of the students followed the request to closely describe a single event with

some detail whilst others provided wider but shallower accounts. A typical account

(transcribed with minor changes for legibility in this article1) is presented below:

When using blackboard, groups can be set up so you can communicate

with people in group work. This involves talking to the people directly

using chat, or emailing them, but also being able to exchange files. This

is very useful especially as people work in different places and times to

others. It also enables people to communicate easily without having to

find the ID numbers. It also means people can continue to develop work

without actually having to meet up to exchange files, which is very useful

for people who don’t need to come into the building.

1
Original spellings and expressions were recorded in Nudist 4™.

page 11 of
 Personal summary: communication with others

Summaries of this account by other members of group:

communication facilities, easy communication, communication tool

Group summary (combining all four accounts): communication

The consistency within the summaries this account was repeated in almost all cases

and gave a clear indication that the individual accounts were well understood

within the student community. It also repeated the finding from the earlier study

that it is not necessarily difficult to achieve consistency in such studies. The

consistency of language and terminology used by the students in their individual

accounts also provided a strong indication that the particular model of data

collection was working effectively.

The expectation had been to provide a more detailed understanding two issues of

usability, and to apply these findings to the future development of the LMS.

The expectation that the accounts would reveal differences between the designs of

interfaces was confounded by the near complete omission of terms or issues

relating to human-computer interface design. One student summarized the account

of another in their group as ‘navigation’. One further account, triggered by the

‘worst experience’ question, raised the problems that occurred when no

page 12 of
confirmation was given after an assignment was delivered electronically through

Blackboard™: a lack of closure. There are many examples of successful use of the

interfaces but interface design itself (or the interface design as perceived by the

students) is, at most, a minor consideration in the use of an LMS. It could be

suggested that this may just reflect ‘bad interface design’ but arguing for this

position would be to devalue what the students choose to say, requiring students to

guess the meaning of the questions that are asked.

The second aspect of usability, understanding the relationship between the

students’ technical abilities and their use of an LMS was also reflected to a minor

extent in the accounts and is not yet sufficient to feature in our models. If anything

there is evidence that an LMS creates a framing effect, limiting the students

awareness of a system’s potential to what the words on icons mean. Though there

may be issues that are transparent to the data collection process, such as overall

confidence in using technology, there seems to be little evidence that students

exploit their technical abilities in a way that could not be achieved by most

students.

To view these results as negative would be to misunderstand the intention of the

Grounded Theory methodology. The expectations of the researchers should not

direct the information that is collected. The lack of discussion of interface design

indicated that other issues were considered as more critical in using an LMS.

page 13 of
The ‘open coding’ process reveals instead a terminology and language focused on

being able to (or alternatively failing to) work in a remote-access mode when the

students needed to or wanted to. This single factor, supported by both WebCT™

and Blackboard™ systems, was only implemented as policy by Kingston

University for Blackboard™. This issue overrode any other consideration of

implementation or design. Within this common theme, the different resources

available seem equally significant (e.g. lecture notes, assignment details, etc.), and

the ‘reasons’ for access, were varied.

Axial Model

A story line is constructed to provide an account of how the individual in each case

perceives the problems that they encounter. This includes the causal conditions that

led to the problem, the particular context of each problem, the intervening events

and behaviours that provide a dynamic sense to the case, the strategies applied by

the individuals themselves and the eventual outcomes (Strauss and Corbin, 1990,

pp. 61-2). One possible formulation of the account from the box above, represented

here without a strict typology of concepts for simplicity, could be as in Figure 1.

Figure 1

page 14 of
With further analysis of similar accounts a model is developed that provides more

subtlety to the analysis and an understanding of the consequences that result when

options fail. Figure 2, for example includes an assessment to consider the

possibility that a virtual meeting is not sufficient.

Figure 2

Even without a more rigorous approach there are evident gaps in the network that

need to be explored (theoretical sampling). What, for example, determines the

choice of mode for a virtual meeting? Or what could be the consequence, to match

with ‘avoid coming into university’, when a real meeting cannot be organized?

Completing an axial model to integrate all student accounts in the same, single

model is the key step in ‘understanding’ the students’ behaviour. There will be

contexts in which students have little opportunity to make decisions that affect the

consequences for themselves. Equally there will be contexts and circumstances

when students can make choices that they anticipate will produce greater benefits

in their own terms. Understanding the relationships between these circumstances,

actions and benefits helps us to understand why students perceive some patterns of

behaviour as the rational pattern to choose. Understanding these relationships can,

page 15 of
ideally, allow the University to make changes to the context or circumstances in

which the students find themselves, so that they change their decision.

Core Category

A core category should provide an overriding conceptual framework within which

the decisions taken in each case can appear rational to the individuals (Strauss and

Corbin 1990, p116).

The terms that were identified by students as summaries in the positive accounts

were intended to provide a catalogue of concepts that could be considered as the

candidate phenomena for a core category, with reinforcing evidence taken from the

summaries of the negative accounts. This has not directly applied to the accounts

that have been collected.

Within the positive accounts the concept of ‘accessibility’ (including variations

such as working from home or remote access) was most consistently used and

almost all the negative accounts reflected failures of the system to support this. The

summary terms used by the students appear to be chosen in terms of their

instrumental value, identifying the significant change in circumstances that has

most influence on outcome (as also with ‘communication’ in the example used

above).

page 16 of
Remote access did not confer direct benefits to the students. It was the way in

which that access is exploited that was of benefit. This suggested that the

summaries should primarily be treated as intervening conditions rather than as

possible core categories. Where the system works as students expect, the benefits

cover a wide range of issues: avoiding journeys into University (as in the

example), the direct saving of time, starting academic work sooner, catching up on

study when behind, being aware of new and relevant information, etc. It was these

stated benefits that became potential core categories.

The benefits themselves, as taken from the accounts of the students, were too

varied to provide a single ‘core phenomenon’ and generally could not be expected

to show such consistency. A stated benefit, such as ‘avoiding travel to the

University’, could be stated in a more general or specific form or be seen to have

direct consequences that occur in other accounts, such as ‘saving time’ or ‘having

more time to work’.

The development of an axial model and core category is not yet complete. It is

clear that the relationships between the benefits can be constructed into a limited

set of possible lattice structures which closely limit the range of what could act as a

core category (see Figure 3, for example). Strauss and Corbin (1990) suggest that it

is more effective to produce alternative models where the balance between more

than one type of benefit occurs, but this does not appear to be the case in this study.

page 17 of
Please note that some of the connections may not be obvious – this is a

representation of the students’ perceptions and not our own interpretation.

Figure 3

page 18 of
As would be predicted from the minimal impact of usability on any scale in open

coding, the network above makes little reference to either technical skills or to

aspects of interface design. Even in those cases the benefits that were gained or not

gained by the individuals are represented directly within this network.

An initial reaction to the students’ use (or non-use) of the LMS might be to suggest

that students are merely using it as a passive filing or messaging system, but this

would fail to reflect that the students are actively engaged in using (or attempting

to use) the different systems to manage their learning. Remote access, rather than

computer access, provides them with a choice of how and where they can study.

It is relevant to note that these students and this analysis, have provided a more

critical assessment than most students who work within a full distance learning

framework. These students choose to meet their objectives by working remotely,

by following a traditional model of full attendance at the University, or by

managing a combination of both. From the evidence in the accounts there is no

single solution because costs, benefits and reliability have to be weighed against

each other. The analysis of the accounts from these students suggests that students

do not currently perceive these costs and benefits to be realized in direct

educational terms.

page 19 of
Implications for the Design of an LMS

This research indicates that students consider the potential to work in distance

learning mode as the overriding benefit of using an LMS, and selectively choose to

do so to integrate their work at University with the rest of their lives. They actively

exploit the flexibility that remote access offers to manage their normal learning

efficiently, even if the benefits do not directly affect the quality of work that they

produce.

It was noted, at the start of the paper, that the technical facilities within an LMS

offered little that was not already available elsewhere on the Internet. It was

suggested that this might offer some scope for gaining significant advantage by

making minor changes to the interface. The evidence from this research suggests

that the scope for improvement is far wider than anticipated.

As students are provided with options as to how to study, they need to be able to

assess in comparable terms, the options that are available. Students can exploit the

facilities in an LMS that fit easily within a conventional framework, such as

discussing a group project with a colleague. This electronic discussion, with other

students you know and trust, can be judged against meeting face-to-face in terms

of the cost of a time delay and the quality of discussion that is needed. Such

choices are relatively simple to evaluate. Some of the e-learning options that are

page 20 of
offered to students are far more difficult to evaluate. We should begin to question

how a student could assess the educational value of ‘participating in an online

discussion group’? With what should it be compared?

Entwistle provides a detailed analysis of some approaches to designing courses to

‘encourage student learning directed towards personal understanding’ (Entwistle

2000, p 1). Assuming that we are designing online courses intelligently to match

course design with the learning process, the evidence from the students in this

study is that what is offered through technology must be comparable in educational

value and effort if students are to be able to choose between a conventional mode

of study and an ‘e-learning’ model.

There are of course pragmatists for whom the value of participating in an online

discussion group is simply measured as ‘20% of the assessment for the module’.

We would wish that e-learning should be able to present a better case than that!

Addendum

As with action research, the overall intention of the research was to build a theory

that could be used to guide future developments within the University. The process

of introducing technical changes into Blackboard™ is inevitably slow, but other

simple changes to improve the effectiveness of the system are possible: increasing

consistency in the way information is presented, providing more logical integration

page 21 of
of the facilities that are available, and increasing its reliability. The intention is to

provide students with a reliable, personalized portal to the University that supports,

as a first stage, a coherent system for time and information management. From

there we can progress to the design of educational activities that can be

equivalently presented within a conventional and web-based environment where

the students are able to choose between modes of study and not between activities

that are not directly comparable.

The focus of the research has now shifted to follow what the students are telling us,

and considerably increased in scale. The phase of theoretical sampling now

continues across all the Faculties in the University and with access to cohorts of

more than 400 students. We hope that by following a strict model of theoretical

sampling we shall not need to work with them all!

References

Alsop, G. and Tompsett C. P. (2002). A Grounded Theory Investigation of

Students’ Perceptions of Learning Management Systems, The proceedings of the

9th Improving Student Learning symposium, 2001. Oxford Centre for Staff and

Learning Development, Oxford (in print)

Blackboard Inc. (2001) http://company.blackboard.com. dated 04-01-2001

page 22 of
Bliss, J. and Ogborn, J. (1977) Students’ reactions to undergraduate science,

Heinnemann, London.

Entwistle N. (2000) Constructive alignment to improve the quality of learning in

higher education. Paper presented at the Dutch Educational Research Conference,

University of Leiden, May 24, 2000

Glaser, B. G. (1978). Theoretical Sensitivity. Mill Valley, Ca., Sociology Press.

Glaser, B. G. and Strauss A. L. (1967). The discovery of grounded theory:

strategies for qualitative research. London, Weidenfeld and Nicholson.

Halliday, M. A. K. (1976) In System and Function in Language(ed., Kress, G. R.)

Oxford University Press, London, pp. 3-6.

Miles, M. B. and Huberman A. M. (1994). Qualitative data analysis: an expanded

source book. London, Sage.

Strauss, A. L. and Corbin J. (1990). Basics of qualitative research: grounded

theory procedures and techniques, Sage.

WebCT, Inc. (2002) http://www.webct.com/company

page 23 of