Oncology: Prostate/Testis/Penis/Urethra

Clinical Practice Guidelines on Prostate Cancer:

A Critical Appraisal
Mohit Gupta, John McCauley, Amy Farkas, Ahmet Gudeloglu,
Molly M. Neuberger, Yen-Yi Ho, Lawrence Yeung, Johannes Vieweg* and
Philipp Dahm
From the Department of Urology, University of Florida (MG, JM, AF, AG, MMN, LY, JV) and Malcom Randall Veterans
Affairs Medical Center, Gainesville (PD), Florida, and Department of Urology (PD) and Division of Biostatistics, School of
Public Health (YYH), University of Minnesota and Urology Section, Minneapolis Veterans Affairs Health Care System
(MMN, PD), Minneapolis, Minnesota

Purpose: Clinical practice guidelines are increasingly being used by leading

Abbreviations
organizations to promote high quality evidence-based patient care. However, the
and Acronyms
methodological quality of clinical practice guidelines developed by different
organizations varies considerably. We assessed published clinical practice ABS American Brachytherapy
Society
guidelines on the treatment of localized prostate cancer to evaluate the rigor,
applicability and transparency of their recommendations. AGREE Appraisal of Guidelines
for Research and Evaluation
Materials and Methods: We searched for English based clinical practice guide-
lines on treatment of localized prostate cancer from leading organizations in the AUA American Urological
Association
15-year period from 1999 to 2014. Clinical practice guidelines limited to early
detection, screening, staging and/or diagnosis of prostate cancer were excluded CPG clinical practice guideline
from analysis. Four independent reviewers used the validated AGREE II IOM Institute of Medicine
instrument to assess the quality of clinical practice guidelines in 6 domains, NCCN National Comprehensive
including 1) scope and purpose, 2) stakeholder involvement, 3) rigor of devel- Cancer Network
opment, 4) clarity of presentation, 5) applicability and 6) editorial independence. NICE National Institute for
Results: A total of 13 clinical practice guidelines met inclusion criteria. Overall Health and Care Excellence
the highest median scores were in the AGREE II domains of clarity of presen-
tation, editorial independence, and scope and purpose. The lowest median score Accepted for publication October 29, 2014.
* Financial interest and/or other relationship
was for applicability (28.1%). Although the median score of editorial indepen- with American Urological Association.
dence was high (85.4%), variability was also substantial (IQR 12.5e100). NICE Correspondence: Department of Urology,
and AUA clinical practice guidelines consistently scored well in most domains. University of Minnesota, Minneapolis Veterans
Affairs Health Care System, Urology Section
Conclusions: Clinical practice guidelines from different organizations on 112D, One Veterans Dr., Minneapolis, Minnesota
treatment of localized prostate cancer are of variable quality and fall short of 55417 (telephone: 612-467-3532; FAX: 612-467-
2232; e-mail: pdahm@umn.edu).
current standards in certain areas, especially in applicability and stakeholder
involvement. Improvements in these key domains can enhance the impact and For another article on a related
implementation of clinical practice guidelines. topic see page 1382.

Key Words: prostatic neoplasms, practice guidelines as topic,

evidence-based medicine, government, Florida

CLINICAL practice guidelines are to promoting high quality, evidence-

important tools to help clinicians based and safe patient care. They
and patients reach evidence-based hold promise for improving the
decisions about health care. The quality, appropriateness and cost-
development of CPGs has been central effectiveness of medical therapies.

0022-5347/15/1934-1153/0 http://dx.doi.org/10.1016/j.juro.2014.10.105
THE JOURNAL OF UROLOGY
2015 by AMERICAN UROLOGICAL ASSOCIATION EDUCATION AND RESEARCH, INC.
Vol. 193, 1153-1158, April 2015
Printed in U.S.A.
www.jurology.com j 1153
1154 CLINICAL PRACTICE GUIDELINES ON PROSTATE CANCER
Thus, leading organizations in the field of urology
are increasingly recognizing the importance of CPGs
and dedicating considerable resources toward
developing and disseminating them. CPGs differ
from systematic reviews, cost analyses and decision
models by making explicit recommendations aimed
at directly influencing patient, clinician and policy
maker decision making. They are also becoming the
basis of quality of care measures that are likely to
affect urologist reimbursements with pay for per-
formance measures on the horizon.1,2
Ideally CPGs from different professional organi-
zations would use consistent, high quality method-
ology to reach similar clinical recommendations.
Unfortunately the methodological quality of CPGs
developed by different organizations varies consid-
erably. These differences reflect the specific mission,
size, financial resources, membership and target
audience of each organization. Therefore, before
specific recommendations from CPGs are imple-
mented into clinical practice their underlying
methodology and quality of evidence should be crit-
ically reviewed.
Accordingly we appraised published CPGs from
leading organizations on the treatment of prostate
cancer. Our immediate goal was to guide efforts
of the Florida Prostate Cancer Advisory
Council (http://prostatecanceradvisorycouncil.org)
to develop a state legislature commissioned system
of care for Florida. Using the AGREE II instrument
we assessed the methodological rigor and trans-
parency of those CPGs as well as the variability
among them.
MATERIALS AND METHODS
We searched for CPGs on the therapeutic management of
prostate cancer using 3 databases, including 1) the
National Guideline Clearinghouse (http://www.guideline.
gov), a public resource of AHRQ (Agency for Healthcare
Research and Quality), 2) the guideline database of G-I-N
(Guidelines International Network, http://www.g-i-n.net),
an international nonprofit organization devoted to
the development and dissemination of CPGs, and 3)
PubMed (http://www.ncbi.nlm.nih.gov/pubmed), which
searches the United States NLM (National Library
of Medicine).
For each of those databases we used broad, sensitive
search strategies to identify relevant CPGs from leading
organizations during the 15-year study period of 1999
through 2014. We included the most recent updates
of previously published guidelines. CPGs limited to early
detection, screening, staging and/or diagnosis were
excluded from analysis. We also excluded publications
(eg editorials and letters) that simply discussed
guidelines. We limited our study to CPGs published in
English.
To assess the quality of the CPGs in our study we
applied structured data abstraction. Four independent
reviewers with prior evidence-based medicine training
assessed methodological quality using the validated
AGREE II instrument.3,4 It includes 23 items that map to
6 domains, including 1) scope and purpose (3 items), 2)
stakeholder involvement (3 items), 3) rigor of develop-
ment (8 items), 4) clarity of presentation (3 items), 5)
applicability (4 items) and 6) editorial independence
(2 items) (supplementary table, http://jurology.com/). The
4 reviewers completed the user training recommended by
the AGREE II developers as well as 2 training rounds of
CPG assessment using bladder cancer guidelines. They
independently scored CPGs on a scale of 0 to 7 on each
item per AGREE II instrument recommendations, quan-
tifying the extent that criteria were met. Reviewer scores
were then expressed as standardized domain scores on a
percent scale of 0% to 100%. We calculated domain scores
by adding all scores of individual items in a domain and
scaling the total as a percent of the maximum possible
score for that domain.
All assessments were based on the published full text
versions of the CPGs and on any supporting documenta-
tion as referenced. Discrepancies were resolved by
consensus after discussion among reviewers. If several
versions of a CPG from an organization were available, we
formally reviewed only the most recently published
version. To test our hypothesis we performed descriptive
statistics and nonparametric tests using SPSS, version
21. We calculated the intraclass correlation and the
within question variation for each of the 23 AGREE II
questions as a measure of interrater reliability. Intraclass
correlation was considered poor, fair, good and excellent
for values in the range of less than 0.4, 0.40 to 0.59, 0.60 to
0.74 and 0.75 to 1.0, respectively.5
RESULTS
Ultimately 13 CPGs met our study inclusion criteria
(supplementary Appendix, http://jurology.com/). Six
CPGs were from organizations originating in the
United States and the other 7 were from interna-
tional government entities. Overall the highest
median scores were in 3 AGREE II domains,
including domain 4dclarity of presentation (87.5%),
domain 6deditorial independence (85.4%) and
domain 1dscope and purpose (84.7%) (see table and
figure). The lowest median score of 28.1% was for
applicability (domain 5). Although the median score
of editorial independence (domain 6) was high at
85.4%, variability was also substantial with an IQR
of 12.5% to 100%.
To better understand the observed AGREE II
scores, especially for domains with low scores, we
analyzed the responses that contributed to each
domain (supplementary table, http://jurology.com/).
Scores of applicability (domain 5) were low due to
the low median scores (less than 50%) on questions
on the presentation of monitoring and/or auditing
criteria (17.9%) and on the consideration of re-
source implication (42.9%). In regard to stakeholder
 CLINICAL PRACTICE GUIDELINES ON PROSTATE CANCER 1155

Mean AGREE II domain scores of 13 prostate cancer guidelines

% Domain (type)

1 (scope 2 (stakeholder 3 (development 4 (presentation 6 (editorial

Guideline purpose) involvement) rigor) clarity) 5 (applicability) independence)
ABS 81.94 56.94 54.17 84.72 13.54 33.33
Aragon Institute of Health Sciences 95.83 68.06 88.02 94.44 54.17 89.58
ASCO 93.06 38.89 63.54 81.94 31.25 93.75
AUA 100.00 81.94 82.81 93.06 11.46 89.58
AUA/American Society for Radiation Oncology 84.72 44.44 25.00 62.50 8.33 16.67
Cancer Council Australia 81.94 90.28 93.23 98.61 50.00 97.92
British Association of Urological Surgeons 58.33 25.00 18.23 73.61 21.88 12.50
European Association of Urology 68.06 63.89 66.15 98.61 64.58 100.00
NCCN 80.56 58.33 57.29 93.06 52.08 85.42
NICE 98.61 100.00 87.50 87.50 77.08 97.92
New Zealand Ministry of Health 73.61 59.72 22.40 81.94 28.13 22.92
Royal College of Surgeons in Ireland 90.74 53.70 50.00 94.44 4.17 33.33
Society of Urologic Oncology 91.67 52.78 50.52 62.50 26.04 83.33
Medians 84.72 58.33 57.29 87.50 28.13 85.42

involvement (domain 2) the views and preferences DISCUSSION

of the target population were rarely sought (45.7%).
To assess the quality of the CPGs we calculated
the domain scores of individual guidelines (see
table). Overall the United Kingdom NICE CPG
achieved the highest scores of greater than 80% for
all domains except applicability (domain 5), with a
score that was still high at 77.1%. The AUA CPG
also scored consistently above 80% but the score for
applicability (domain 5) was only 11.5%.
The average intraclass correlation of the 23
AGREE II questions was 0.54 (95% CI 0.29e0.79)
with a within question variation of 1.77. There was
no apparent association between the average
AGREE II score and the intraclass correlation.
AGREE II scores of 13 prostate cancer guidelines by domain
IOM defined CPGs as statements that include
recommendations intended to optimize patient care
that are informed by a systematic review of evidence
and an assessment of the benefits and harms of
alternative care options.6 There is increasing in-
terest in using CPGs as a tool to assimilate the best
available evidence for specific clinical questions to
guide evidence-based clinical practice and thereby
improve quality of care, decrease variation and
improve cost-effectiveness of health care delivery.
Consequently IOM identified 8 standards to help
develop rigorous, trustworthy CPGs. They include
transparency and disclosure of conflicts of interest,
identification of group/author composition and evi-
dence supporting recommendations. These criteria
closely mirror those of the AGREE II survey, which
may serve as an instrument for clinicians to identify
high quality CPGs.
We critically assessed the methodological quality
of 13 CPGs for prostate cancer. We had 2 key find-
ings. 1) We found a large degree of heterogeneity in
the methodological quality of CPGs developed by
different organizations, underscoring the need for
users to critically appraise such documents before
applying them to decisions about individual patient
care or health policy.1 2) We found major short-
comings in the domains of stakeholder involvement
and applicability that potentially undermine the
validity of CPGs, raising concerns about their
dissemination and impact.
Clinician use of CPGs represents a final trans-
lation hurdle of applying scientific research to pa-
tient care. The inconsistencies and lack of
methodological rigor identified in our study, which
are recognized as part of IOM standards, threaten
to undermine the common goal of advancing high
1156 CLINICAL PRACTICE GUIDELINES ON PROSTATE CANCER
quality patient care. For example, in the domain of
applicability (domain 5) our evaluation demon-
strated that most guidelines did not address the
barriers and facilitators to implementing recom-
mendations and did not provide ways to audit
effective implementation. As a result, clinicians
may have difficulty enacting the clinical recom-
mendations set forth in CPGs and organizations
that propose these guidelines may have limited
means to monitor how their recommendations are
put into practice.
As another example the IOM identified trans-
parency and disclosure as a key element in devel-
oping trustworthy CPGs. In domain 6 the guideline
scores were based largely on the degree to which the
developers provided explicit disclosures regarding
conflicts of interest (external funding, financial
conflicts among authors, etc). Guidelines that scored
high in this domain disclosed financial conflicts of
interest and maintained explicit statements to pro-
vide assurance that the views and interests of the
funding body behind the CPG did not unduly in-
fluence the final recommendations. Most guidelines
recognized the importance of these conflicts of in-
terest on editorial independence and clearly dis-
closed them. The mean score of this domain was
85%, although there was significant variability
among guidelines.
Few studies to date have specifically addressed
the methodological quality of CPGs on prostate
cancer. Prior studies from our group provided a
critical overview of such CPGs by major professional
organizations but we focused on the lack of meth-
odological rigor of development. We did not apply
the AGREE II instrument until the current study.7,8
An assessment similar to our current study was
performed by the CEP (Capacity Enhancement
Program) of the Cancer Guidelines Advisory Group
in Canada with the objective of assessing the quality
and comprehensiveness of prostate cancer CPGs.9
That study used the AGREE II instrument to sys-
tematically evaluate 46 English language CPGs
published from 2003 through 2008. In contrast to
our study, the Canadian study had a broader scope.
It included questions on prevention and promotion,
screening, diagnostic assessment and staging,
which are questions that we excluded. The Cana-
dian study also included several guideline docu-
ments per organization if more than one was
available. Its mean domain scores were 60% (range
14% to 94%) for scope and purpose, 40% (range 0%
to 83%) for stakeholder involvement, 42% (range 5%
to 98%) for rigor of development, 67% (range 27% to
97%) for clarity of presentation, 25% (range 4% to
73%) for applicability and 31% (range 0% to 94%) for
editorial independence. Briefly, scores in the Cana-
dian study reflected major variability between CPGs
and a particular need for improvement in the
domain of applicability, which had the lowest score.
As in our study, the Canadian study showed that
NICE CPGs consistently had the highest scores
across domains.
In a similar study Qaseem et al focused exclu-
sively on prostate cancer screening CPGs by 4 major
organizations, including ACS (American Cancer
Society), AUA, USPSTF (United States Preventive
Services Task Force) and ACPM (American College
of Preventive Medicine).10 They found the best
scores for scope and purpose (domain 1) and clarity
of presentation (domain 4), and the lowest score for
stakeholder involvement (domain 2).
Lastly our findings are consistent with those of a
systematic review of all studies that used the
AGREE instrument to assess CPG quality.11 That
systematic review showed that despite some
improvement with time quality had to further
improve. Enhanced international collaboration was
suggested as an approach to help increase the effi-
ciency of the CPG development process.
Given that CPGs are designed to guide clinician
behavior and provide explicit recommendations for
the treatment of typical index patients based on the
best available research evidence, our study high-
lights the importance of urologists carefully
reviewing the quality of CPGs before using them.1
As our study demonstrates, the methodological
quality developed by different organizations varies
considerably, impacting the confidence that we can
place in their recommendations. Therefore, urolo-
gists should be familiar with the defining charac-
teristics of clinical guidelines that deserve the
evidence-based label. Meanwhile guideline
developers in urology should strive to raise the bar
by adopting a transparent, methodologically
rigorous and ideally unified framework to rate the
quality of evidence and move from evidence to
recommendations.12
We recognize several limitations to our study.
1) We relied exclusively on published full text ver-
sions of the CPGs and on any supporting docu-
mentation as referenced. We did not reach out to
CPG authors or professional organizations to obtain
additional information. However, given that the
published CPG remains the only practical source
of information for most users, our findings are
relevant and important.
2) Our study was limited by the inherent
subjectivity of any assessment of methodological
quality. However, AGREE II was shown to be a
reliable assessment tool and our evaluation was
performed by 4 (rather than a minimum of 2)
independent reviewers.3,4 All 4 reviewers completed
online training modules specific to the AGREE II
instrument and successfully participated in the
 CLINICAL PRACTICE GUIDELINES ON PROSTATE CANCER 1157

pilot testing of our data abstraction form. Despite
the involvement of a larger number of independent
reviewers interrater reliability was acceptable.
Our study findings are important for organiza-
tions such as the Florida Prostate Cancer Advisory
Council and for payers such as BlueCross Blue-
Shield as they seek to develop measures of
accountability to assess and monitor quality of care,
and for guiding improvement initiatives.13,14 Qual-
ity of care measures should be based on strong
recommendations15 developed using rigorous
methodology, as reflected by high AGREE II scores,
ideally across all domains. However, the quality of
evidence supporting many CPGs has not been sys-
tematically investigated. For example, Poonacha
et al found that recommendation issues in guide-
lines by NCCN were largely developed based on
lower levels of evidence.16 Given the relatively low
scores achieved by the NCCN CPG in the domains of
stakeholder involvement, rigor of development and
applicability, its prominent role in advising drug
coverage decisions for the Centers for Medicare and
Medicaid Services appears questionable.17
In contrast, high standards in the form of high
AGREE II scores were met by the NICE CPG, which
provides additional cost-effectiveness analyses that
routinely help inform health policy decisions for the
United Kingdom NHS (National Health Service).
NICE has tremendous resources with an annual
budget equivalent to more than $70 million. How-
ever, its CPG is specific to the United Kingdom
health care setting and, thus, it may have limited
applicability to other countries.18
The AUA CPG comes close to meeting the
AGREE II standards except for the applicability
domain. Ongoing efforts to standardize reporting
and facilitate the integration of recommendations
into electronic medical records systems with tools
such as BRIDGE-Wiz are expected to increase the
applicability of CPGs, enhancing their impact and
promoting their implementation.19,20
Formal consideration of resource utilization may
be a more elusive goal. In a recent study Schwartz
and Pearson found that only about 50% of all major
professional organizations considered costs in their
CPGs and only about 25% used a formal grading
system.2 Barriers to increased consideration of costs
include patient concerns over bed side rationing.
The Choosing Wisely initiative of the ABIM
(American Board of Internal Medicine) Founda-
tion (http://www.choosingwisely.org/doctor-patient-
lists/) represents a step in the right direction since
it identifies medical tests and procedures that likely
provide little benefit, raise costs and in some cases
cause harm.21
CONCLUSIONS
Our study reflects growing awareness of the need
for improving CPG methodology and quality. In our
systematic appraisal we found that published CPGs
on the treatment of prostate cancer are of variable
methodological quality and frequently fell short of
current standards. We noted significant deficiencies
in the domains of applicability and stakeholder
involvement as well as a large degree of heteroge-
neity in the methodological quality of CPGs devel-
oped by different organizations. These shortcomings
limit the effectiveness of CPGs for use by policy
makers and health care providers.
ACKNOWLEDGMENT
Dr. Mary E. Knatterud, Department of Urology, Uni-
versity of Minnesota, provided editorial assistance.

REFERENCES
1. Dahm P, Yeung LL, Gallucci M et al: How to use a
clinical practice guideline. J Urol 2009; 181: 472.
2. Schwartz JA and Pearson SD: Cost consideration
in the clinical guidance documents of physician
specialty societies in the United States. JAMA
Intern Med 2013; 173: 1091.
3. Brouwers MC, Kho ME, Browman GP et al:
Development of the AGREE II, part 1:
performance, usefulness and areas for
improvement. CMAJ 2010; 182: 1045.
4. Brouwers MC, Kho ME, Browman GP et al:
Development of the AGREE II, part 2: assess-
ment of validity of items and tools to support
application. CMAJ 2010; 182: E472.
5. Hallgren KA: Computing inter-rater reliability for
observational data: an overview and tutorial.
Tutor Quant Methods Psychol 2012; 8: 23.
6. Institute of Medicine (U.S.). Committee on
Standards for Developing Trustworthy Clinical
Practice Guidelines: Clinical Practice Guidelines
We Can Trust. Edited by R Graham, M Mancher,
D Miller Wolman et al. Washington, D.C.:
National Academies Press 2011; pp 4 and 266.
7. Dahm P, Kunz R and Schunemann H:
Evidence-based clinical practice guidelines for
prostate cancer: the need for a unified approach.
Curr Opin Urol 2007; 17: 200.
8. Dahm P, Yeung LL, Chang SS et al: A critical
review of clinical practice guidelines for the
management of clinically localized prostate
cancer. J Urol 2008; 180: 451.
9. Cancer Practice Guidelines Status Report:
Prostate Cancer. Toronto: Capacity Enhancement
Program of the Cancer Guidelines Advisory
Group.
10. Qaseem A, Barry MJ, Denberg TD et al:
Screening for prostate cancer: a guidance
statement from the Clinical Guidelines
Committee of the American College of
Physicians. Ann Intern Med 2013; 158: 761.
11. Alonso-Coello P, Irfan A, Sola I et al: The
quality of clinical practice guidelines over the
last two decades: a systematic review of
guideline appraisal studies. Qual Saf Health Care
2010; 19: e58.
12. Dahm P, Chapple CR, Konety BR et al: The future
of clinical practice guidelines in urology. Eur Urol
2011; 60: 72.
1158 CLINICAL PRACTICE GUIDELINES ON PROSTATE CANCER
13. Miller DC, Murtagh DS, Suh RS et al:
Establishment of a urological surgery quality
collaborative. J Urol 2010; 184: 2485.
14. Miller DC and Saigal CS: Quality of care
indicators for prostate cancer: progress toward
consensus. Urol Oncol 2009; 27: 427.
15. Guyatt GH, Oxman AD, Vist GE et al: GRADE:
what is quality of evidence and why is it
important to clinicians? BMJ 2008; 336: 995.
16. Poonacha TK and Go RS: Level of scientific
evidence underlying recommendations arising
from the National Comprehensive Cancer
Network clinical practice guidelines. J Clin Oncol
2011; 29: 186.
17. Abernethy AP, Raman G, Balk EM et al:
Systematic review: reliability of compendia
methods for off-label oncology indications.
Ann Intern Med 2009; 150: 336.
18. Chalkidou K: Comparative effectiveness review
within the U.K.s National Institute for Health
and Clinical Excellence. Issue Brief (Commonw
Fund) 2009; 59: 1.
19. Shiffman RN, Michel G, Rosenfeld RM et al:
Building better guidelines with BRIDGE-Wiz:
development and evaluation of a software
assistant to promote clarity, transparency, and
implementability. J Am Med Inform Assoc 2012;
19: 94.
20. Shiffman RN, Shekelle P, Overhage JM et al:
Standardized reporting of clinical practice
guidelines: a proposal from the Conference on
Guideline Standardization. Ann Intern Med 2003;
139: 493.
21. Gliwa C and Pearson SD: Evidentiary rationales
for the Choosing Wisely Top 5 lists. JAMA 2014;
311: 1443.