ORIGINAL REPORTS

Acquisition of Evidence-Based Surgery Skills

in Plastic Surgery Residency Training
Claire L. F. Temple, MD, MSc, FRCSC, and Douglas C. Ross, MD, FRCSC
Division of Plastic Surgery, University of Western Ontario, Schulich School of Medicine and Dentistry, London,
Ontario, Canada
INTRODUCTION: The teaching and learning of critical ap-

praisal skills and evidence-based practices by surgical residents

has been identified as an unmet need in many surgical training
programs.
METHODS: Monthly journal clubs over a calendar year were

the setting for a critical appraisal curriculum. Preassigned

homework assignments and carefully selected articles with specific methodologies were posted electronically and formed the
course material. Pretests and posttests on medical statistics
and methodology were administered. Presurveys and postsurveys on attitudes toward evidence-based surgery (EBS)
were administered.
RESULTS: Precourse surveys revealed a lack of confidence in
residents knowledge of epidemiology and biostatistics, with an
increase in confidence postcourse (2.6 vs 2.9; p 0.4). Precourse and postcourse, there was strong support for more critical appraisal training in residency (5.1 vs. 4.8; p 0.1) and an
agreement that understanding evidence-based practices is important for the clinical practice (4.6 vs. 4.6; p 0.4) as well as
the research endeavors of a plastic surgeon (5.4 vs. 5.5; p 0.8).
Pretest scores, when compared with PGY level, showed an
increase in knowledge with increasing PGY level (p 0.6).
Average pretest scores were 6.5 of a total of 15 points, or 43%.
Posttest scores were improved, at 7.8 of 15, or 52% (p 0.6).
Sixty-four percent of learners felt that journal club was a good
venue for teaching critical appraisal skills precurriculum. Fifty
percent of learners were still of that impression at course completion (p 0.3). The modest improvement in test scores indicates an impact on critical appraisal skills, but reliance on
journal clubs to teach these skills is insufficient.
CONCLUSIONS: Through monthly journal clubs and selfdirected assignments, critical appraisal skills were improved
across PGY levels in an academic surgical training program;
however, other settings and methods of teaching are required to

Correspondence: Inquiries to Claire Temple, MD, MSc, FRCSC, Division of Plastic Surgery, University of Western, Ontario, Schulich School of Medicine and Dentistry, 268
Grosvenor Street, London, Ontario, Canada; fax: (519) 646-6049; e-mail:
ctemple4@uwo.ca

augment a curriculum in evidence-based surgery. (J Surg 68:

167-171. 2011 Association of Program Directors in Surgery.
Published by Elsevier Inc. All rights reserved.)
KEY WORDS: evidence-based medicine, critical appraisal,

plastic surgery, Journal Club

COMPETENCIES: Patient Care, Medical Knowledge, Practice

Based Learning and Improvement

INTRODUCTION
Evidence-based surgery (EBS) involves integrating clinical expertise and the best available clinical evidence from systematic
research to make sound, informed surgical decisions.1 The Office of Evidence-Based Surgery, established within the American College of Surgeons, perceives that most surgical specialists base their practice on uncontrolled case series and
uncontested expert opinion.2
To alter this behavior, ideally students would be introduced
to evidence-based practices in medical school.3 Unfortunately,
even when exposed to critical appraisal principles, many medical school graduates feel that their clinical practice is not based
on the best available evidence.4
Thus, often it falls on the shoulders of the academic surgeon
to train residents in EBS. This is challenging as few surgeons
themselves are trained in EBS. From a systematic review of the
effectiveness of critical appraisal skills training for clinicians,
Taylor et al.5 concluded that there is a paucity of educators
within the field of evidence-based health.
A second challenge is to find the right venue for instruction
of these skills. Surgical residents are clinically busy, and academic teaching time is squeezed by a continuously increasing
volume of knowledge coupled with a mandated decrease in
work hours. Yet, critical appraisal cannot just be absorbed
these skills need to be taught explicitly and then reinforced
during the care of specific patients. The weekly formal lecture
series is one venue to integrate acquisition of these skills; however, dry lectures may not produce lasting retention. Self-directed learning alone may not be realistic for the busy surgical
resident. One-off half-day clinical courses in critical appraisal

Journal of Surgical Education 2011 Association of Program Directors in Surgery

Published by Elsevier Inc. All rights reserved.

1931-7204/$30.00
doi:10.1016/j.jsurg.2010.12.004

167

have not been particularly effective.6 Integrating EBS into journal clubs is an acceptable way to begin to integrate formal teaching of critical appraisal skills into residency training.7
As a first step to tackling this deficiency in our program, we
elected to introduce formal teaching into the University of
Western Ontario Plastic Surgery Residency Training Program
via a combination of journal club sessions and self-directed
learning assignments. Our hypothesis was that doing EBS in
an integrated fashion would allow it to be done efficiently and
would facilitate its routine use in daily practice.

METHODS
A formal program for teaching EBS was run over one academic
year. Materials were provided to the learners on a passwordprotected website, including links to the Users Guides to the
Surgical Literature series from the Canadian Journal of
Surgery.8-17 Three homework assignments were posted, with
topics covering randomized controlled trials (RCTs), metaanalyses, and diagnostic tests. Carefully selected journal articles
with methodologies matching the homework assignments were
delivered electronically.
Formal teaching was incorporated into monthly, 3-hour evening journal clubs. At the first journal club, residents were asked
to list any courses they had taken in epidemiology and biostatistics. A short questionnaire sampling attitudes toward EBS
was administered. A 6-point Likert scale was used, with responses ranging from strongly disagree to strongly agree.
Domains sampled included the student confidence in their EBS
skills, the need for better teaching of EBS in residency, the
usefulness of EBS skills in clinical and academic surgery, and
the preferred forum for acquiring these skills. Finally, residents
completed a pretest covering basic topics in EBS. The test was
proctored to promote independent test taking among the residents. No time limit was set on this examination.
At alternating journal clubs, a theme for the journal club (ie,
RCTs) was selected, and a carefully selected journal article illustrative of that topic was assigned. In addition, a corresponding homework assignment on that topic was completed by each
resident and brought to the journal club. The learners were
encouraged to access multiple resources for this assignment,
including textbooks, Internet resources, and the applicable users guide. An in-depth discussion on the methodology of the
article was undertaken, and the homework assignment was reviewed. Additional articles were also reviewed as per the consultant hosting the journal club that evening.
At the final journal club in the year-long program, a posttest,
identical to the pretest, was administered. Likewise, an identical
survey on attitudes toward EBS was given. Each pretest and
posttest was then anonymized for features, including name
(which was optional), PGY level, and whether the test was the
pretest or posttest. Examinations were then graded using a standardized answer key by a single surgeon (C.T.).
Precourse and postcourse attitudes and pretest and posttest
scores were compared with paired t-tests. Linear regression was
168

used to assess PGY level and test score results. A p-value of less
than 0.05 was considered significant. Analyses were completed
using InStat 3, version 3.0 b (In-STAT Inc, Scottsdale, Arizona) for Macintosh.

RESULTS
Ten plastic surgery residents in the University of Western Ontario plastic surgery training program participated. There were
two each of PGY 1, 2, and 3 residents, three PGY 4 residents,
and one PGY 5 resident. Six had had prior undergraduate or
medical school courses in epidemiology or biostatistics. Four
participants had had none.
Eight journal club sessions were held over a single academic
calendar year. Of the 10 residents, on average 8 were present at
each session. No resident consistently missed the sessions. Two
were invariably away either on vacation or were busy on call. On
average, 4 consultant surgeons attended each session.
Although no time limit was set for the examination, all learners had completed both the survey on attitudes and the test by
20 minutes. Of the 10 residents, 7 had complete results, including a pretest and posttest. Two had a pretest only, and 1 had a
posttest only.
Regarding precourse perceptions, most participants lacked
confidence in their knowledge of epidemiology and biostatistics, scoring 2.6 on a 6-point Likert scale with poor confidence a 1 and high confidence a 6. After the course, there was
a nonsignificant increase in confidence (2.6 vs 2.9; p 0.4).
Precourse, there was strong agreement that there should be
more critical appraisal training in residency, scoring 5.1 on a
6-point scale of agreement. After the course, there was a similar
sentiment (5.1 vs 4.8; p 0.1).
Precourse and postcourse, there was strong endorsement that
understanding critical appraisal is important for the clinical
practice of a plastic surgeon (4.6 vs. 4.6; p 0.4) and for a
plastic surgeon scientist (5.4 vs. 5.5; p 0.8).
Pretest scores, when regressed on PGY level, showed no association (p 0.6) in knowledge with increasing PGY level.
This would indicate that these skills are not being learned informally within the existing training program curriculum.
Average pretest scores were 6.5 of a total of 15 points, or
43%. Posttest scores were improved at 7.8 of 15, or 52%; however, this difference did not reach statistical significance (p
0.6). This might represent beta error, in that there may be an
insufficient number of subjects to reject the null hypothesis of
no difference in scores.
Qualitative impressions from informal feedback from the
year-long course included a widespread appreciation of the effort that went into the program. There was moderate resentment for the time required to complete the self-directed assignments. Although initially 64% of students felt that journal club
was a good venue for teaching critical appraisal skills, 50% were
still of that impression upon completion of the curriculum (p
0.3).

Journal of Surgical Education Volume 68/Number 3 May/June 2011

DISCUSSION
EBS is a critical part of daily clinical practice in surgery. The
surgeon must integrate the best-available literature with patient
preferences to develop a treatment plan.1 Inspiring learners to
see the practical importance of incorporating evidence-based
principles in patient management is, in and of itself, an important training goal.
The clinic and operating room settings are useful to inspire questions. Although self-directed learning with the appropriate tools and reading guides to assist in interpreting
various study designs could be used, specific small group
teaching in a clearly formatted and structured curriculum
dedicated to teaching quantitative methods has been shown
to be most effective.18 Short-burst educational interventions
where clinicians attend a single, half-day critical appraisal
course have not been an effective method of enhancing EBM
skills.5 Although traditional journal clubs have been criticized for focusing on critical appraisal skills while undervaluing other areas, such as biostatistics,19 Ebbert et al.7 concluded that journal clubs are a useful way to improve
understanding of epidemiology and biostatistics, reading
habits, and the use of medical literature in practice.
Our combination of integrating journal club with preparatory self-directed assignments was meant to optimize teaching
critical appraisal skills. Interestingly, however, only 50% of the
learners felt that journal club was the ideal venue for EBS learning at the completion of the curriculum. Although it has been
postulated that the teaching and learning of EBM is most effective when contextually based in clinical practice (vs didactic
lectures or as part of traditional journal clubs), the evidence for
this is lacking.20,21 Perhaps other suitable locations for acquiring these skills include the weekly, didactic plastic surgery
teaching sessions or the principles of surgery lecture series for
early year surgical residents.
There is a need for improvement in critical appraisal skills
amongst surgical residents and consultants. This gap in
knowledge was evident in the poor performance on the pretest in our study. Although the examination tested basic
concepts in medical statistics and research methodology, the
group scored just 43%. The pretest survey of attitudes
among the resident group demonstrated an appreciation of
the importance of more EBS training as well as an acknowledgment of their own deficit in EBS knowledge. In a survey
of senior otolaryngology residents, Amin et al.22 also found a
welcoming attitude to evidence-based medicine and an appreciation of the need to teach these skills formally. Thus,
surgical trainees understand the importance of EBS skills
and are willing to work to enhance those skills.
A reasonable question to ask is why, despite a year-long
horizontal program to enhance EBS knowledge, the posttest
scores only increased to 52% from 43%? Similar findings
were noted in a meta-analysis on effectiveness of instruction
in critical appraisal, in which residents showed a small
change in knowledge (mean gain: 1.3%; standard deviation

[SD]: 1.7%), even less than medical students (mean gain:

17.0%; SD: 4.0%).23 It is possible that substantial gains in
EBS knowledge might not be detected by our posttest examinations because of a lack of valid assessment of critical appraisal skills.24
Multiple obstacles are likely to inhibit the effective teaching of EBS. In a qualitative study using grounded theory,
Bhandari et al.25 identified multiple barriers that limited the
residents ability to apply EBS to their care of patients. Some
of these included lack of ready access to surgical EBS resources, lack of education in EBS, time constraints, lack of
priority, and fear of staff disapproval. Suggestions for improvement included hiring staff surgeons with EBS training,
offering coursework in EBS to surgical staff, improving interdepartmental communication, and providing greater flexibility for EBS training.
An intriguing study by Kitto et al.26 suggested that surgical cultures played a significant role in the learning of
EBS. Using qualitative methods, the authors found that surgeons who demonstrated both clinician and scientist roles
within their practices were more likely to be effective teachers of EBS within the traditional residency structure. Evidence to support an integrated teaching model of scientist
and clinician for EBS was reported by Haines and Nicholas,27 who described an intensive program in the principles
of EBS incorporated into a neurosurgical training program.
In place of the usual 2-hour biweekly professors rounds,
case-based sessions were led jointly by both a neurosurgeon
and an epidemiologist. Residents were assisted in developing
appropriate questions, searching the literature, critically analyzing the papers, and summarizing the results. Six topics
were analyzed in the first 2 years of the course, with results
ranging from finding insufficient evidence to formulate
treatment recommendations to identifying solid literature
that permitted the development of criteria for cervical spine
clearance that was incorporated into the institutions emergency department. The resources used were already in place
in the institution, and the addition of specific time and
epidemiologic expertise was all that was required to make
this program successful.
The 4 steps required to practice EBS include (1) formulate
a question based on a clinical situation, (2) conduct a focused literature search, (3) critically appraise the literature to
find the best evidence, and (4) integrate the information and
act in accordance with the best available evidence.28 We have
been focusing primarily on the (3), critical appraisal. Our
program did include framing a question into Pico format29
but did not incorporate formal instruction on literature
search. Rather, trainees focused on appraisal of an assigned
article. Via mandatory preassignments, the methodology of
certain study designs was learned. As we expand the program, we will add focus on literature search skills. Securing
the support of an expert medical librarian will be critical.

Journal of Surgical Education Volume 68/Number 3 May/June 2011

169

CONCLUSIONS
Through monthly journal clubs and self-directed assignments,
critical appraisal skills were improved across PGY levels in an
academic surgical training program. Knowledge deficiencies
were identified and were improved modestly by the curriculum.
This pilot study suggested other techniques and areas to expand
the curriculum, including incorporating some didactic lectures
in medical statistics as well as how to do a literature search. A
commitment to critical appraisal teaching is an ongoing component of teaching core competencies to residents. The ultimate goal is to make our residents not only good surgeons but
also good doctors, and this goal will be enhanced by an ability to
bring the most up to date and accurate information on how to
treat their patients.

ing population-based volume-outcome relationships in

surgery. Can J Surg. 2002;45:109-115.
12. Birch DW, Eady A, Robertson D, Pauw SD, Tandan V.

Users guides to the surgical literature: how to perform a

literature search. Can J Surg. 2003;43:136-141.
13. Bhandari M, Devereaux PJ, Montori V, Cina C, Tandon

V, Guyatt GH. Users guides to the surgical literature:

how to use a systematic literature review and meta-analysis. Can J Surg. 2004;46:60-67.
14. Thoma A, Farrokhyar F, Bhandari M, Tandan V. Users

guides to the surgical literature: how to assess a randomized

controlled trial in surgery. Can J Surg. 2004;47:200-208.
15. Birch DW, Goldsmith CH, Tandan V. Users guides to

the surgical literature: self-audit and practice appraisal for

surgeons. Can J Surg. 2005;48:57-62.

REFERENCES
1. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Rich-

16. Mastracci TM, Thoma A, Farrokhyar F, Tandan VR,

ardson WS. Evidence-based medicine: what it is and what

it isnt. BMJ. 1996;312:71-72.

Cina CS. Users guides to the surgical literature: how to

use a decision analysis. Can J Surg. 2007;50:403-409.

2. Toedtler LJ, Thompson LL, Rohatgi C. Training sur-

17. Thoma A, Carnacchi SD, Lovrics PJ, Goldsmith CH. Users

geons to do evidence-based surgery: a collaborative approach. J Am Coll Surg. 2004;199:293-299.

guides to the surgical literature: how to assess an article on healthrelated quality of life. Can J Surg. 2008;51:215-224.

3. Rubin P, Franchi-Christoper D. New edition of tomor-

18. Fingerhut A, Borie R, Chadii D. How to teach evidence-

rows doctors. Med Teach. 2002;24:368-369.

based surgery. World J Surg. 2005;29:592-595.

4. Doran T, Maudsley G, Zakhour H. Time to think? Ques-

tionnaire survey of pre-registration house officers experiences of critical appraisal in the Mersey Deanery. Med
Educ. 2007;41:487-494.
5. Taylor R, Reeves B, Ewings P, et al. A systematic review of

the effectiveness of critical appraisal skills training for clinicians. Med Educ. 2000;34:120-125.
6. Taylor RS, Reeves BC, Ewings PE, Taylor RJ. Critical

19. Hatala R, Keitz SA, Wilson MC, Guyatt G. Beyond journal

clubs. Moving toward an integrated evidence-based medicine curriculum. J Gen Intern Med. 2006;21:538-541.
20. Khan KS, Coomarasamy A. A hierarchy of effective teach-

ing and learning to acquire competence in evidencedbased medicine. BMC Med Educ. 2006;15:59.
21. Kulier R, Coppus SF, Zamora J, et al. The effectiveness of

appraisal skills training for health care professionals: a randomized controlled trial. BMC Med Educ. 2004;4:30.

a clinically integrated e-learning course in evidence-based

medicine: a cluster randomised controlled trial. BMC Med
Educ. 2009;12:21.

7. Ebbert JO, Montori VM, Schultz HJ. The journal club in

22. Amin M, Saunders JA, Fenton JE. Pilot study of the

poastgraduate medical education: as systematic review.

Med Teach. 2001;23:455-461.
8. Archibald S, Bhandair M, Thoma A. Users guides to the

surgical literature: how to use an article about a diagnostic

test. Can J Surg. 2001;44:17-23.
9. Thoma A, Sprague S, Tandan V. Users guides to the

surgical literature: how to use an article of economic analysis. Can J Surg. 2001;44:347-354.
10. Urschel JD, Goldsmith CH, Tandan VR, Miller JD. Users

guides to the surgical literature: how to use an article evaluating surgical interventions. Can J Surg. 2001;44:95-100.
11. Hong D, Tandan VR, Goldsmith CH, Smunovic M. Users

guides to the surgical literature: how to use an article report170

knowledge and attitude towards evidence based medicine

of otolaryngology higher surgical trainees. Clin Otolaryngol. 2007;32:133-135.
23. Norman GR, Shannon SI. Effectiveness of instruction in

critical appraisal (evidence-based medicine) skills: a critical appraisal. CMAJ. 1998;158:177-181.

24. MacRae HM, Regehr G, Brenneman F, McKenzie M,

McLeod RS. Assessment of critical appraisal skills. Am J

Surg. 2004;187:120-123.
25. Bhandari M, Montor V, Devereaux PJ, et al. Challenges

to the practice of evidence-based medicine during residents surgical training: a qualitative study using grounded
theory. Acad Med. 2003;78:1183-1190.

Journal of Surgical Education Volume 68/Number 3 May/June 2011

26. Kitto S, Petrovic A, Gruen RL, Smith JA. Evidence-based med-

28. Sackett DL, Straus SE, Richardson WS, et al. Evidence

icine training and implementation in surgery: the role of surgical

cultures. J Eval Clin Pract. 2010; e-pub ahead of print.

Based Medicine. How to Practice and Teach EBM. 2nd

edition. Edinburgh: Churchill Livingstone; 2000.

27. Haines SJ, Nicholas JS. Teaching evidence-based medi-

29. Richardson WS, Wilson MC, Nishikawa J, Hayward RS.

cine to surgical subspecialty residents. J Am Coll Surg.

2003;197:285-289.

The well-built clinical question: a key to evidence-based

decisions. ACP J Club. 1995;123:A12-A13.

Journal of Surgical Education Volume 68/Number 3 May/June 2011

171