METHODOLOGICAL ISSUES IN NURSING RESEARCH

The importance of conducting and reporting pilot studies:

the example of the Scottish Births Survey
Edwin R. van Teijlingen MA MEd PhD
Senior Lecturer in Public Health, Department of Public Health and Dugald Baird Centre for Research on Women's Health,
Medical School, University of Aberdeen, Aberdeen, UK

Anne-Marie Rennie BSc MSc RGN RM

Research Midwife, Dugald Baird Centre for Research on Women's Health, Aberdeen Maternity Hospital, Aberdeen, UK

Vanora Hundley BN MSc RGN RM

Lecturer, Centre for Advanced Studies in Nursing, University of Aberdeen, Foresterhill Health Centre, Aberdeen, UK

and Wendy Graham DPhil

Professor/Director, Dugald Baird Centre for Research on Women's Health, University of Aberdeen, Aberdeen, UK

Submitted for publication 7 March 2000

Accepted for publication 19 December 2000

Correspondence:
Edwin van Teijlingen,
Department of Public Health
and Dugald Baird Centre for Research
on Women's Health,
Medical School,
University of Aberdeen,
Aberdeen AB25 2ZD,
UK.
E-mail: van.teijlingen@abdn.ac.uk

2001 Blackwell Science Ltd

V A N T E I J L I N G E N E .R ., R E N N I E A -M ., H U N D L E Y V . & G R A H A M , W .
(2001)

Journal of Advanced Nursing 34(3), 289295

The importance of conducting and reporting pilot studies: the example of the
Scottish Births Survey
Background. In many research papers, pilot studies are only reported as a means of
justifying the methods. This justication might refer to the overall research design,
or simply to the validity and reliability of the research tools. It is unusual for reports
of pilot studies to include practical problems faced by the researcher(s). Pilot studies
are relevant to best practice in research, but their potential for other researchers
appears to be ignored.
Objective. The primary aim of this study was to identify the most appropriate
method for conducting a national survey of maternity care.
Methods. Pilot studies were conducted in ve hospitals to establish the best of four
possible methods of approaching women, distributing questionnaires and encouraging the return of these questionnaires. Variations in the pilot studies included (a)
whether or not the questionnaires were anonymous, (b) the staff involved in
distributing the questionnaires and (c) whether questionnaires were distributed via
central or local processes. For this purpose, ve maternity hospitals of different sizes
in Scotland were included.
Results. Problems in contacting women as a result of changes in the Data Protection
Act (1998) required us to rely heavily on service providers. However, this resulted in
a number of difculties. These included poor distribution rates in areas where
distribution relied upon service providers, unauthorized changes to the study
protocol and limited or inaccurate information regarding the numbers of questionnaires distributed.
Conclusions. The pilot raised a number of fundamental issues related to the process
of conducting a large-scale survey, including the method of distributing the

289

E.R. van Teijlingen et al.

questionnaire, gaining access to patients, and reliance on `gatekeepers'. This paper

highlights the lessons learned as well as the balancing act of using research methods
in the most optimal way under the combined pressure of time, ethical considerations
and the inuences of stakeholders. Reporting the kinds of practical issues that occur
during pilot studies might help others avoid similar pitfalls and mistakes.
Keywords: pilot studies, methodology, midwifery research, satisfaction studies, UK
Data Protection Act (1998), maternity care

Introduction
This paper reports the lessons learnt from a number of small
pilot studies conducted in preparation for the Scottish Births
Survey, a national survey of maternity services in Scotland
(Hundley et al. 2000). This paper argues for more and better
reporting of the process and outcomes of pilot studies.
The Scottish Births Survey aimed to include all women who
had delivered a baby in Scotland during a given 10-day
period in the autumn of 1998. Considering the scale of the
sample, the geographical size of Scotland and the limited
resources, the obvious (that is realistically viable) research
method was either a postal questionnaire or a telephone
survey. The latter option was not considered viable as some
women did not have a telephone and these women were likely
to come from a particular socio-economic background
(Bowling 1997). Thus, using telephone interviews would
not give opinions from women across the whole socioeconomic spectrum. Furthermore, identifying women and
gaining access to their telephone numbers would have been a
problem. Hence the decision was made to conduct a postal
questionnaire survey.

Ethical issues
Deciding to conduct a postal survey immediately led to a
number of ethical considerations. Ethical approval was
sought from the Multi-Research Ethics Committee, which
considers applications for research conducted in more than
one region of Scotland. However, the committee felt this to
be unnecessary because it regarded the main study to be an
audit and not as research (Wilson et al. 1999). However,
classifying a study as audit does not `solve' ethical considerations such as: `How could we make sure that the survey was
not going to upset women who had had a delivery in the
designated period, but whose baby was seriously ill and/or
had subsequently died?' One could argue that such women
should be included in the survey to cover the range of
experiences of maternity care in the population. At the same
time it is clear that women with these experiences should be
290

treated with sensitivity. It was decided to leave the decision

whether or not to invite a woman to participate in the study
to the midwife who was caring for her in the postnatal
period.

Methods
A self-completion questionnaire was developed by the survey
team, drawing on the criteria agreed by the multidisciplinary
Standard Setting Group for Scotland, and using existing,
validated questionnaires (Mason 1989, Audit Commission
1996, Lamping & Rowe 1996, Hundley et al. 1997, Rennie
et al. 1998). The majority of the questions were closed,
although a section for open comments was included. The
questionnaire was pretested on 10 postnatal women to assess
its acceptability and reviewed by the research team and the
funders for validity. Minor amendments were made to
the text and the questionnaire was then piloted to assess
the acceptability of the revised questionnaire as well as the
feasibility of the proposed research process.
The pilot studies were restricted to births within a
limited period in July 1998 in order to replicate the full
survey as much as possible. The ve maternity units
involved in the pilot studies, and four different methods of
distributing and returning questionnaires are shown in
Table 1. The units were selected by size (i.e. number of
deliveries per year). As the main study covered all maternity units in Scotland, a survey process was needed which
worked in both small and large units. Hence, two large
hospitals were selected for the pilot studies, a medium-sized
one and two smaller ones.
In hospital `Large 1', the questionnaire was distributed
anonymously by midwives working in the community, thus,
making it impossible for the research team to send out
reminders to nonrespondents. The original plan for hospital
`Large 2' was to distribute the questionnaires with a study
number attached, which allowed reminders to be sent to
those women who did not return their questionnaires.
However, the local head of midwifery voiced concerns about
the ethics of this strategy, partly as a result of one previous

2001 Blackwell Science Ltd, Journal of Advanced Nursing, 34(3), 289295

Methodological issues in nursing research

Importance of conducting and reporting pilot studies

Table 1 Five pilot studies: each with a different research process

Type of unit

Identication? (Named/
numbered or anonymous)
Distribution of questionnaire
Method of reminder

Large 1

Large 2

Small 1

Small 2

Medium 1

Anonymous
CMW*
N/A

Named/numbered
CMW
Head of midwifery

Named/numbered
CMW
Research team

Named/numbered
CMW
Research team

Anonymous
Hospital and CMW (leaet)
Hospital records department

*CMW: Midwife working in the community.

local incident, where the parents of an ill baby had been sent
a questionnaire that was felt to be inappropriate, and partly
because of the changes in the United Kingdom Data Protection Act (1998). Consequently, reminders were sent out via
the head of midwifery in case there were any problems with
the new-born baby.
In units `Small 1' and `2' the midwifery managers did not
raise any objections to providing us with the names of women
to whom the questionnaire was administered. Consequently
numbered questionnaires were distributed by midwives in the
community, who also returned to us a list with the women's
names (after gaining consent) matched to the study number
on the questionnaire. This allowed us to send out reminders
directly to the women if required. Finally, in unit `Medium 1'
the plan was to distribute numbered questionnaires to the
women on discharge from hospital. At the same time, staff in
the hospital record department completed an additional form
for each woman containing the woman's name, address and a
unique study number. The names and numbers were held
locally to allow the hospital to send out of reminders. In
addition, the midwives in the hospital were asked to hand out
a leaet to the women to inform them about the survey.
All ve hospitals were asked to inform us about the
number of questionnaires handed out to women to allow us
to calculate the response rates (see Table 2).

Results
The ndings of the pilot studies related to both process and
outcome. As the outcomes were more straightforward, these
will be presented rst. Table2 shows the actual number of
births that took place during the period of the pilot studies.
The number of births in hospital `Large 2' (n 27) did not
differ greatly from the number in `Medium 1' (n 22) and
this may be because of seasonal uctuations in births. It
became clear during the pilot studies that midwives were not
informing the research team of all questionnaires handed out
to new mothers in their area. This was particularly obvious in
units `Large 1' and `Medium 1'. The proportion of women
that the midwives reported receiving the questionnaire varied
from 0% to 96%. Moreover, response rates, when calculated
on the basis of information provided by the midwives, were
often unrealistically high. In `Large 1' midwives reported that
they had handed out 12 questionnaires, representing only
24% of all births registered in the pilot period, yet 24
questionnaires were returned. Similarly, in `Medium 1',
where 22 births took place, no questionnaires were reported
by the midwives to have been handed out, however, 10 were
returned. In the two small hospitals, the number given
equalled those returned, although in `Small 1', the midwives
included only 20% of all registered births in the pilot.

Table 2 Overview of the results of the ve pilot studies

Type of unit
Large 1
Number of births
(in 10-day period in July 1998)
Number of questionnaires
distributed according to midwives
Response rate on basis of
information from distributors (%)
Number of responses (n)
Response rate on basis
of actual number of births (%)

Large 2

Small 1

Small 2

Medium 1

50

27

22

12

26

200
24

58
15

100
1

100
2

10

100 (2/2)

45 (10/22)

48 (24/50)

55 (15/27)

2001 Blackwell Science Ltd, Journal of Advanced Nursing, 34(3), 289295

20 (1/1)

291

E.R. van Teijlingen et al.

These discrepancies inuenced the response rate, which

was dependent on the denominator used. Most of the
response rates in Table 2 are erroneously high when calculated on the basis of the number of questionnaires handed out
by midwives. However, they were fairly low if the number of
ofcially registered births during the period was used as the
denominator.
The pilot studies indicated that the agreed process of
distributing the questionnaires was not always adhered to
(for example in Medium 1). Without consulting the research
team, the person responsible for distributing the questionnaires decided that it was better to distribute them through
the community midwives rather than through the hospital
records department. Furthermore, the poor communication
discussed above meant that questionnaires were returned,
which we did not know were administered in the rst place.
More importantly, we could not send reminders to nonresponding women, which could have helped to improve our
response rate.

Discussion
Pilot studies are a crucial step in the research process. The
results of the pilot studies made the research team reconsider
aspects of the main survey. We opted for a system whereby a
midwife in the community handed out a numbered questionnaire on the tenth postnatal day. After gaining consent, the
midwife also lled in a form with the woman's name and
address and the study number on the questionnaire. The
latter form was then forwarded to the research team.

Lessons learned
The politics of the research process is sometimes equally or
more important than other considerations in deciding which
research design to use. In our case involving midwives was
perceived to be benecial for two reasons: (1) increased
recruitment rates as a result of face-to-face recruitment and
(2) a greater sense of participation in and ownership of the
national audit amongst the midwifery profession. However,
using intermediaries to distribute the questionnaire added an
extra link in the chain of distribution and communication.
The inuence of a `gatekeeper' (Holloway 1997) probably
weakened the research process. The midwives could also have
had an effect on our ndings by inuencing the women's
responses either directly (consciously or unconsciously), or
indirectly with women giving responses that they thought
were more acceptable to the midwives (Riley 1977).
The fact that managers in some units chose to change the
method of distribution during our pilot, indicates the chal292

lenges of conducting research within a real life setting. It

highlights the importance of an awareness of current
concerns and how these may affect the proposed research.
These concerns were addressed in the main study by ensuring
that all managers were contacted personally and given the
opportunity to discuss their concerns and clarify the research
process.
The women in the study seemed to be able to answer most
of the questions without too much difculty. The experience
of conducting the pilot studies prepared the research team
and the funding agency for the possibility of a poor response
rate. It made us consider other ways to increase the response
rate, such as sending posters to maternity hospitals, and
alerted us to considering the representativeness of the main
survey, in particular the characteristics of nonresponders
(Barriball & While 1999). It also highlighted the need to
compare data with other studies conducted on this population, and to compare the overall number of births reported by
midwives with the number of births ofcially registered in the
time period. However, we feel a general note of caution is
needed here, namely that pilot studies frequently do not have
a reliable statistical foundation, because they are nearly
always based on small numbers. Thus, pilot study ndings
may offer some indication of the likely size of response rates
of the main survey, but they do not guarantee success.
The pilot studies also emphasized the need for a
programme of publicity to ensure that the midwives were
`on board'. Therefore, prior to the main survey, we sent out
posters announcing the survey to all maternity units for
display in midwives' coffee rooms. In addition, every midwife
involved in providing postnatal care at home was sent a letter
informing him or her of the survey. The Royal College of
Midwives Scottish Board also asked midwifery supervisors on
behalf of the research team to encourage midwives in their
area to participate and support the survey.

The (possible) impact of The Data Protection Act (1998)

The managers of the various maternity services had different
perceptions as to what the forthcoming changes in The Data
Protection Act (1998) allowed them to do regarding the
involvement of their clients in research. At the time of the
pilot and the full survey, the 1998 Data Protection Act had
not yet ofcially come into force. However, the Act had
begun to inuence people's assumptions regarding appropriate and legitimate ways of conducting research even before
it was implemented on 1 March 2000 (Bourne 2000).
However, the lack of formal guidelines at the time about
how to interpret the forthcoming Data Protection Act made
the perceptions and the subsequent actions of these managers

2001 Blackwell Science Ltd, Journal of Advanced Nursing, 34(3), 289295

Methodological issues in nursing research

even more important. This is a clear example of what is

referred to as `the denition of the situation', namely that `if
men (sic) dene situations as real, they are real in their
consequences' (Joseph 1990, p. 175). The Data Protection
Act (1998) for the rst time covered `personal data falling
within certain types of traditional ling system or within
certain types of record for example healthcare records'
(Bourne 2000, p. 27). The impact of the Data Protection Act
on UK health care and health service research is still not fully
known, but it has been argued that `research involving
processing sensitive personal data could be halted' (Goddard
2000, p. 7). The reason for this could be problems ranging
from identifying the sampling frame to recruitment.
Researchers are unlikely to gain access to sensitive data to
identify eligible study participants, and therefore they might
have to rely on service providers as gatekeepers for such
access. Service providers might also decide to exclude certain
`sensitive' groups, for example, patients with negative
outcomes, or ethnic minorities with language problems. In
addition, having to add these additional steps in the research
process might make a study simply too expensive.

The importance of pilot studies

Full reports of pilot studies are rare in the research literature
(Lindquist 1991, Muoio et al. 1995), with some notable
exceptions (Baker 1990, Clarke et al. 1998, Downs & Black
1998, Robinson et al. 1998). Pilot studies full a range of
important functions (Table 3) and can provide valuable
insights for other researchers, but they are likely to be
`underdiscussed, underused and underreported' (Prescott &
Soeken 1989, p. 60). When reported, they often only justify
the research methods and/or particular research tool used.
Too often research papers only refer to one element of the
pilot study, for example, to the `pretesting' or `pilot testing' of
a questionnaire (Green et al. 1988, De Vaus 1993, van der
Hulst 1999). Such papers often simply state: `the questionnaire was tested for validity and reliability.' Indeed Moser
and Kalton (1986, p. 49) suggested that testing the adequacy
of a questionnaire is `probably the most valuable function of
the pilot study'.
However, even when pilot studies are mentioned in more
detail in academic papers and reports, researchers regularly
comment that they `had learned from the pilot study' and
made the necessary changes, without offering the reader
details about what exactly was learned. Some of these
processes and outcomes from both successful and failed pilot
studies might be very useful to others embarking on projects
using similar methods and/or instruments. This is particularly
important because pilot studies can be `time-consuming,

Importance of conducting and reporting pilot studies

Table 3 Reasons for conducting pilot studies
Developing and testing adequacy of research instruments
Assessing the feasibility of a (full-scale) study/survey
Assessing people's willingness to participate/potential or
likely response rates
Designing a research protocol
Assessing whether the research protocol is realistic and workable
Assessing whether data collectors understand the protocol
Establishing whether sampling frame and technique are
adequate and effective
Identifying logistical problems which might occur using
proposed methods
Estimating variability in outcomes to help determining
sample size
Collecting preliminary data
Determining what resources (nance/staff) are needed
for main study
Assessing the proposed data analysis techniques to
uncover potential problems
Developing a research question and/or research plan
Assessing whether or not each data collector obtains
similarly valid information
Training researchers in as many elements of the research
process as possible
Educating students about research methods and the
research process
Convincing funding bodies that research team is competent
and knowledgeable
Convincing funding bodies that the main study is feasible
and worth funding
Convincing other stakeholders (service providers, ethics
committees, managers, politicians, etc.) that the
proposed main study is worth supporting
(Source: Moser & Kalton 1986; Prescott & Soeken 1989; Rosenberg
& Daly 1993; Crombie & Davies 1996; Courts 1997; Beursckens
et al. 1998; Drummond & Coyle 1998; Burns & Grove 1999; Baird
2000)

frustrating, and fraught with unanticipated problems, but it is

better to deal with them before investing a great deal of
time, money, and effort in the full study' (Mason & Zuercher
1995). It has also been argued that the current research
climate demands accountability from researchers, which
means that there is a need to ensure the best possible use of
research results (Crosswaite & Curtice 1994). We would like
to go one step further and argue that researchers have an
ethical obligation to make the best use of their research
experience by reporting issues arising from all parts of a
study, including the pilot phase.

Conclusion
Our message is that everybody needs pilot studies, not just
their own, but also those of other researchers in the eld.

2001 Blackwell Science Ltd, Journal of Advanced Nursing, 34(3), 289295

293

E.R. van Teijlingen et al.

Well-designed and well-conducted pilot studies can inform us

about the best research process and occasionally about likely
outcomes. Investigators should be encouraged to report more
ndings from pilot studies, and in particular to report in more
detail the actual improvements made to the study design and
the research process. The decisions about how, where, when
and why to conduct a study are not always made on research
or technical considerations alone. Other pressures, especially
political, can inuence decisions, which in the rst instance
seem to be `simply' research design questions. The recent
changes in the UK Data Protection Act (1998) will result in
problems for researchers in identifying and approaching
potential participants for national surveys if a restricted or
narrow interpretation prevails among managers.

Acknowledgements
These pilot studies would not have been possible without the
participation of the Scottish community midwives, heads of
midwifery and link supervisors. Thanks are also due to the
staff at the Dugald Baird Centre for Research on Women's
Health for their help in conducting the pilot studies. The
survey was commissioned and funded by the Scottish
Programme for Clinical Effectiveness in Reproductive
Health, Edinburgh. Finally, we like to thank the anonymous
reviewers for their positive and helpful comments on our rst
submission.

References
Audit Commission (1996) Maternity Care Survey. Audit Commission/Institute of Child Health, Bristol, UK.
Baird C. (2000) Taking the mystery out of research. The pilot study.
Orthopaedic Nursing 19, 4243.
Baker R. (1990) Development of a questionnaire to assess patients'
satisfaction with consultations in general practice. British Journal
of General Practice 40, 487490.
Barriball K.L. & While A.E. (1999) Non-response in survey research:
a methodological discussion and development of an explanatory
model. Journal of Advanced Nursing 30, 677686.
Beurskens A.J.H.M., De Vet H.C.W. & Kant I.J. (1998) Dwalingen
in de methodologie. VIII. Pilotonderzoeken: zin en onzin. Neder2 lands Tijdschrift Voor de Geneeskunde 142, 21422145.
Bourne I. (2000) The Data Protection Act 1998 selected highlights
of the new legislation. The British Journal of Healthcare Computing & Information Management 17, 2729.
Bowling A. (1997) Research Methods in Health: Investigating Health
& Health Services. Open University Press, Buckingham, UK, p. 232.
Burns N. & Grove S.K. (1999) Understanding Nursing Research, 2nd
edn. W.B. Saunders Company, Philadelphia, PA, p. 40.
Clarke R., Breeze E., Sherliker P., Shipley M., Youngman L., Fletcher
A., Fuhrer R., Leon D., Parish S., Collins R. & Marmot M. (1998)
Design, objectives, and lessons from a pilot 25 year follow-up

294

re-survey of survivors in the Whitehall study of London Civil

Servants. Journal of Epidemiology & Community Health 52, 364
369.
Courts N. (1997) Cabinet on research. From idea to research
projects: steps in designing a research project. Tar Heel Nurse 59,
2223.
Crombie I.K. & Davies H.T.O. (1996) Research in Health Care:
Design, Conduct and Interpretations of Health Services Research.
Wiley, London, p. 239.
Crosswaite C. & Curtice L. (1994) Disseminating research results-the
challenge of bridging the gap between health research and health
action. Health Promotion International 9, 289.
De Vaus D.A. (1993) Surveys in Social Research, 3rd edn. UCL Press,
London, p. 99.
Downs S.H. & Black N. (1998) The feasibility of creating a checklist
for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions.
Journal of Epidemiology & Community Health 52, 377384.
Drummond M. & Coyle D. (1998) The role of pilot studies in the
economic evaluation of health technologies. International Journal
of Technology Assessment in Health Care 14, 405408.
Goddard A. (2000) Preventive action-or legal conict. Times Higher
Education 418(Suppl. 1), 8.
Green J.M., Coupland V.A. & Kitzinger J. (1988) Great Expectations: A Prospective Study of Women's Expectations and Experiences of Childbirth, Vol. 1. Child Care & Development Group,
University of Cambridge, Cambridge, p. 2.6.
Holloway I. (1997) Basic Concepts for Qualitative Research. Blackwell Science, Oxford, p. 87.
van der Hulst L. (1999) Dutch midwives: relational care and birth
location. Health & Social Care in the Community 7, 242247.
Hundley V., Milne J., Glazener C. & Mollison J. (1997) Satisfaction
and the three C's continuity, choice and control women's views
from a randomised controlled trial of midwife-led care. British
Journal of Obstetrics & Gynaecology 104, 12731280.
Hundley V., Rennie A.M., Fitzmaurice A., Graham W., Teijlingen
van E. & Penney G. (2000) A national survey of women's views of
their maternity care in Scotland. Midwifery 16, 303313.
Joseph M. (1990) Sociology for Everyone, 2nd edn. Polity Press,
Cambridge.
Lamping D.L. & Rowe P. (1996) Users' Manual for Purchasers and
Providers: Survey of Women's Experience of Maternity Services.
(Short form) Health Services Research Unit, London School of
Hygiene and Tropical Medicine, London.
Lindquist R. (1991) Don't forget the pilot work! Heart Lung 20,
9192.
Mason D.J. & Zuercher S.L. (1995) Pilot studies in clinical nursing
3 research. Journal of the New York State Nursing Association 26, 11.
Mason V. (1989) Women's Experience of Maternity Care a Survey
Manual. Ofce of Population Censuses and Surveys, HMSO,
London.
Moser C.A. & Kalton G. (1986) Survey Methods in Social
4 Investigation, 2nd edn. Gower, London, pp. 4752 & 9394.
Muoio R., Wolcott L. & Seigel H. (1995) A win-win situation: The
pilot program. Journal of Continuing Education in Nursing 26,
230233.
Prescott P.A. & Soeken K.L. (1989) The potential uses of pilot work.
Nursing Research 38, 6062.

2001 Blackwell Science Ltd, Journal of Advanced Nursing, 34(3), 289295

Methodological issues in nursing research

Riley E.M.D. (1977) What do women want? The question of choice
in the conduct of labour. In Benets and Hazards of the New
Obstetrics (Chard T. & Richards M. eds), pp. 6271, Spastics
International Medical Publications, London.
Rennie A.M., Hundley V., Gurney E. & Graham W. (1998)
Women's priorities for care before and after delivery. British
Journal of Midwifery 6, 434438.
5 Robinson S.M., Mackenzie-Ross S., Campbell-Hewson G., Egleston
C.V. & Prevost A.T. (1998) Psychological effect of witnessed
resuscitation on bereaved relatives. Lancet 352, 614617.

Importance of conducting and reporting pilot studies

Rosenberg K.M. & Daly H.B. (1993) Foundations of Behavioural
Research: a Basic Question Approach. Harcourt Brace College
Publishers, Fort Worth/London, pp. 182184.
The Data Protection Act (1998) Data Protection Act 1998: Elizabeth
II. Chap. 29. The Data Protection Registrar, The Stationary Ofce
Ltd, London.
Wilson A., Grimshaw G., Baker R. & Thompson J. (1999) Differentiating between audit and research: postal survey of health
authorities' views. British Medical Journal 319, 1235.

2001 Blackwell Science Ltd, Journal of Advanced Nursing, 34(3), 289295

295