Brief Communication
Reporting on Statistical Methods To Adjust for Confounding:
A Cross-Sectional Survey
Marcus Mullner, MD; Hugh Matthews, BSc, MBBS; and Douglas G. Altman, DSc
Background: The use of complex statistical models to adjust for
confounding is common in medical research.
Objective: To determine the frequency and adequacy of adjust-
ment for confounding in medical articles.
Design: Cross-sectional survey.
Setting: 34 scientific medical journals with a high impact factor.
Measurements: Frequency of reporting on methods used to
adjust for confounding in 537 original research articles published
in January 1998.
Results: Of the 537 articles, 169 specified that adjustment for
D iscovering the determinants of disease is often not
straightforward, particularly if the disease or the
risk factor is rare or not easily recognized (1). In case
control, cohort, and other nonrandomized studies, the
groups being compared are likely to vary with respect to
several demographic, clinical, and other characteristics.
In randomized trials, authors often try to demonstrate
that the observed treatment effect is not explained by
some difference in baseline characteristics (2). The def-
inition of confounding is that there are alternative ex-
planations for an observed association between a risk
factor and a health outcome. This may occur when one
or more of these demographic or clinical characteristics
are associated with one another and with the outcome of
interest (3). Social class, for example, is known to be
associated with cardiovascular risk factors, such as smok-
ing, serum cholesterol level, and leisure physical activity,
as well as with mortality. This difference in demo-
graphic and clinical characteristics may account for
about half of the excess coronary and all-cause mortality
in blue-collar workers compared with white-collar work-
ers (4).
If confounding cannot be avoided at the design
stage of a study, disentangling the web of causation is
often difficult, and more or less complex statistical
methods are needed. Readers of published scientific ar-
ticles need to know whether and how the authors ap-
propriately adjusted for confounding (5).
122 15 January 2002 Annals of Internal Medicine Volume 136 Number 2
confounding was used. In 1 paper in 10, it was unclear which
statistical method was used or for which variables adjustment was
made. In 45% of papers, it was not clear how multicategory or
continuous variables were treated in the analysis. Inadequate re-
porting was less frequent if an author was affiliated with a de-
partment of statistics, epidemiology, or public health and if arti-
cles were published in journals with a high impact factor.
Conclusions: Details of methods used to adjust for confounding
are frequently not reported in original research articles.
Ann Intern Med. 2002;136:122-126. www.annals.org
For author affiliations, current addresses, and contributions, see end of text.
In this cross-sectional study, we sought to determine
how frequently adjustment is reported in medical scien-
tific articles and whether reporting is sufficiently de-
tailed.
METHODS
Journal Selection
Scientific medical journals were included if they
were published in English, were available in the British
Medical Associations library, and had an impact factor
(6) that placed it in the highest 20% of journals within
its medical specialty. We excluded review journals and
journals specializing in statistics, epidemiology, and
public health.
Article Selection
Two of the authors independently assessed all Jan-
uary 1998 issues of the selected journals to identify full-
length original research articles. Short reports, scientific
letters, case reports, and review articles, and animal stud-
ies were excluded.
Data Collection
We assessed whether adjustment for baseline vari-
ables or confounding was performed and whether the
paper specified which method was used, the variables for
which adjustment was made, and how these variables
www.annals.org
 Reporting on Statistical Methods To Adjust for Confounding
Figure. Articles assessed for reporting of adjustment for confounding or baseline differences and whether adjustment
was reported in the methods or results section.
were handled in the analysis (that is, whether it was
specified how continuous and multicategory variables
were entered into the statistical model). Inappropriate
reporting was defined as not reporting or insufficiently
reporting on one or more of these points. For simplicity,
we assumed that all multiple regression analyses were
done to adjust for confounders, even though in some
cases they were done to identify prognostic variables.
Each paper was independently assessed by two of the
authors. Agreement between the two authors was good
to very good ( 0.61 to 0.96). Disagreement was
usually caused by oversight rather than differing opin-
ions. In case of disagreement or uncertainty, the third
author (a senior statistician) was consulted.
Statistical Analysis
The univariate association between inappropriate
reporting and several variablesfor example, impact
factor (quartiles); at least one of the authors being affil-
iated with department of statistics, epidemiology, or
public health; and number of authors (1 or 2, 3 to 5, 6
www.annals.org
Brief Communication
to 10, or 10)was assessed by using chi-square tests
or chi-square tests for trend. We used multiple logistic
regression to investigate simultaneously variables that
showed an association with inappropriate reporting.
Data were processed by using Excel 97 software (Mi-
crosoft Corp., Redmond, Washington) and Stata, re-
lease 6 (Stata Corp., College Station, Texas).
RESULTS
Thirty-four journals met our inclusion criteria (Ap-
pendix). The median impact factor of these journals in
1996 was 4.26 (interquartile range, 2.64 to 5.74). We
identified 537 articles that fulfilled our inclusion criteria
(Figure), of which 169 (32%) reported adjustment for
confounding or baseline differences.
Reporting of Methods
Of the 169 articles, 152 (90%) appropriately re-
ported methods to adjust for confounding (Figure), 7
reported no method, and 10 mentioned but did not
15 January 2002 Annals of Internal Medicine Volume 136 Number 2 123
Brief Communication Reporting on Statistical Methods To Adjust for Confounding

Table. Association between Inadequate Reporting of Adjustment for Confounders and Study Characteristics in 169
Original Research Articles

Characteristic Studies with Relative Risk (95% CI) P Value*

Inadequate Reporting,
n/n (%)
At least one author affiliated with department
of statistics, epidemiology, or public health
No 59/105 (56) 1.00 (referent)
Yes 17/64 (27) 0.47 (0.300.73) 0.001
Number of authors
12 4/8 (50) 1.00 (referent)
35 28/64 (44) 0.88 (0.421.85)
610 37/83 (45) 0.89 (0.431.86) 0.2
10 7/14 (50) 1.00 (0.422.38)
Impact factor (quartiles)
3.34 25/45 (56) 1.00 (referent)
3.354.67 22/47 (47) 0.84 (0.561.26)
4.689.28 21/40 (53) 0.95 (0.641.40) 0.008
9.28 8/37 (22) 0.39 (0.200.76)

* Chi-square test for comparing two proportions or test for trend.

adequately specify a method. In these 10 articles, the
authors used the phrases multiple regression or mul-
tivariable analysis.
Of articles that specified the method, multiple logis-
tic regression analysis was the most frequently used (n
76), followed by multiple Cox proportional hazards
models (n 43); multiple linear regression (n 33);
and other methods (n 31), including stratified analy-
sis, partial correlation, direct and indirect standardiza-
tion, mixed-effect modeling, and age-adjusted z-score.
Some papers included more than one method of ad-
justed analysis, but each paper was counted only once in
the following analyses.
Reporting of Variables
Of the 169 articles that reported adjustment for
confounding, 154 (91%) clearly specified variables for
which adjustment was made, and 93 (55%) clearly
stated how variables were handled (Figure). Only 93
articles (55%) met criteria for appropriate reporting: 51
articles had one inadequacy, 17 articles had two inade-
quacies, and 8 articles had three inadequacies.
Examples of Inappropriate Reporting
Example 1
A randomized, controlled trial investigated whether
surfactant administered to preterm infants reduces the
incidence of severe complications. The authors provided
great detail to demonstrate that treatment and control
124 15 January 2002 Annals of Internal Medicine Volume 136 Number 2
groups were comparable at baseline. In a table, the au-
thors report baseline adjusted incidence of pulmonary
complications. However, the reader is not told why
adjustment was needed to present a main end point,
how this adjustment was performed, or which of the 14
or more baseline variables were included (7).
Example 2
A randomized, controlled trial compared two anti-
biotics for the treatment of gonorrhea. The authors
stated that After correction for baseline abnormalities
there was no significant difference in laboratory abnor-
malities. However, they did not indicate which baseline
abnormalities were meant and how this correction was
performed (8).
Example 3
A randomized, controlled trial compared balsalazide
with mesalamine in patients with acute ulcerative colitis.
The authors stated that Logistic regression techniques
were used to identify prognostic factors significantly as-
sociated with remission. The reader is told which vari-
ables were finally significantly associated with the out-
come but not which variables were entered into the
model (9).
Example 4
Breathing patterns and respiratory muscle perfor-
mance measures during weaning were examined in a
www.annals.org
 Reporting on Statistical Methods To Adjust for Confounding
case series of 17 patients receiving prolonged mechanical
ventilation. In the methods section, the authors state
that adjusted means were calculated for the variables
presenting a group effect, but the reader is not told
how or where this calculation was performed (10).
Determinants of Inappropriate Reporting
In 64 of the 169 articles (38%), at least one author
was a methodologist (that is, he or she was affiliated
with a department of statistics, epidemiology, or public
health). Among papers with a methodologist-author, the
rate of any inappropriate reporting was about half
that of papers without a methodologist-author (Table).
The rate of inappropriate reporting tended to decrease
as the journal impact factor increased, but this effect was
largely due to a lower rate of inappropriate reporting in
the journals in the highest quartile of impact factor. The
number of authors was not associated with inadequate
reporting.
A multiple logistic regression model with any inap-
propriate reporting as the dependent variable and meth-
odologist-author and impact factor (quartiles) as predic-
tor variables showed that these two effects were largely
independent (results not shown). Among the journals in
the lower three quartiles of impact factor, 12 of 42
(29%) articles with a methodologist-author and 56 of
90 (62%) without a methodologist-author had inappro-
priate reporting. In contrast, among the journals in the
top quartile of impact factor, the rate of inappropriate
reporting was similar among articles with and without a
methodologist-author (3 of 15 [20%] articles vs. 5 of 22
[23%] articles, respectively). However, because the
number of papers was small and this split by impact
factor was not prespecified, P values are not presented.
DISCUSSION
Statistical methods are often misused, and poorly
presenting them leaves the reader unable to critically
interpret the findings of an original research study (11,
12). Some studies use a selection procedure to reduce
the number of variables for which adjustment is needed
to those that are statistically significant. In such cases,
authors should report all variables considered in addi-
tion to those for which adjustment was actually made.
Not reporting whether variables are treated as con-
tinuous or as categorical data may make a difference in
www.annals.org
Brief Communication
the interpretation of the results. Transformation to ful-
fill the assumption of a normal distribution may or may
not have been performed (13). Likewise, for categorical
variables, the definition and number of categories are
needed. Assuming a linear association when it is nonlin-
ear may mask an association (14), having too few and
broad categories may lead to considerable residual con-
founding, and having too many categories may reduce
the power of a model (15, 16).
The reasons for these shortcomings may be mani-
fold. It is possible that all the necessary information was
present before peer review and was omitted during the
publication process. However, errors and omissions are
more likely the consequence of a system failure at many
levels (17), including that of the authors, reviewers, stat-
isticians, and editors (18).
Although data analysis may be correct despite inap-
propriate reporting, such reporting leaves readers unable
to assess whether the data were processed appropriately.
Having a methodologist as an author seems to have a
protective effect, which is in accordance with the find-
ings of an earlier study (15). Why articles published in
journals with a very high impact factor have a lower rate
of inappropriate reporting remains a matter of specula-
tion. It may relate to the fact that these journals more
frequently use statistical reviewers than do lower-rank-
ing journals (19).
We suggest that readers, authors, referees, and edi-
tors try to assess whether original articles state which
statistical method was used to adjust for confounders,
for which variables adjustment was performed, and the
way in which the variables were handled in the analysis.
APPENDIX
The following journals were included in the study: American
Journal of Cardiology, American Journal of Medicine, American
Journal of Obstetrics and Gynecology, Anaesthesia, Anesthesiology,
Annals of Internal Medicine, Archives of Dermatology, Archives of
Internal Medicine, BMJ, Blood, Brain, British Journal of Anaesthe-
sia, British Journal of Cancer, British Journal of Dermatology, Brit-
ish Journal of Obstetrics and Gynaecology, British Journal of Surgery,
Circulation, Critical Care Medicine, Gastroenterology, Gut, JAMA,
Journal of the American College of Cardiology, Journal of Gerontol-
ogy, Journal of the National Cancer Institute, Journal of Pediatrics,
Journal of Rheumatology, Kidney International, The Lancet, New
England Journal of Medicine, Neurology, Pediatrics, Thorax,
Thrombosis and Haemostasis, and Transplantation.
15 January 2002 Annals of Internal Medicine Volume 136 Number 2 125
Brief Communication Reporting on Statistical Methods To Adjust for Confounding
From BMJ, London, and ICRF Medical Statistics Group, Oxford,
United Kingdom.
Acknowledgments: The authors thank the BMJ staff, particularly Rich-
ard Smith, for providing the environment that enabled this research
project.
Requests for Single Reprints: Marcus Mullner, MD, Universitatsklinik
fur Notfallmedizin, Allgemeines Krankenhaus Wien, Wahringer Gurtel
18-20/6D, A-1090 Vienna, Austria; e-mail, marcus.muellner@univie.ac.at.
Current Author Addresses: Dr. Mullner: Universitatsklinik fur Notfall-
medizin, Allgemeines Krankenhaus Wien, Wahringer Gurtel 18-20/6D,
A-1090 Vienna, Austria.
Mr. Matthews: Sandbanks, Graveney, Faversham, Kent ME13 9DJ,
United Kingdom.
Dr. Altman: ICRF Medical Statistics Group, Centre for Statistics in
Medicine, Institute of Health Sciences, Old Road, Headington, Oxford
OX3 7LF, United Kingdom.
Author Contributions: Conception and design: M. Mullner, H. Mat-
thews, D.G. Altman.
Analysis and interpretation of the data: M. Mullner, D.G. Altman.
Drafting of the article: M. Mullner, H. Matthews, D.G. Altman.
Critical revision of the article for important intellectual content: M.
Mullner, H. Matthews, D.G. Altman.
Final approval of the article: M. Mullner, H. Matthews, D.G. Altman.
Statistical expertise: M. Mullner, D.G. Altman.
Administrative, technical, or logistic support: M. Mullner.
Collection and assembly of data: M. Mullner, H. Matthews.
References
1. Hill AB. The environment and disease: association or causation? Journal of the
Royal Society of Medicine. 1965;58:295-300.
2. Altman DG. Adjustment for covariate imbalance. In: Armitage P, Colton T,
eds. Encyclopaedia of Biostatistics. New York: Wiley; 1998:1000-5.
3. Hennekens CH, Buring JE, Mayrent SH. Epidemiology in Medicine. Boston:
Little, Brown; 1987.
4. Pekkanen J, Tuomilehto J, Uutela A, Vartiainen E, Nissinen A. Social class,
health behaviour, and mortality among men and women in eastern Finland.
BMJ. 1995;311:589-93. [PMID: 7663252]
126 15 January 2002 Annals of Internal Medicine Volume 136 Number 2
5. Uniform requirements for manuscripts submitted to biomedical journals.
International Committee of Medical Journal Editors. Ann Intern Med 1997;
126:34-47. [PMID: 8992922]
6. Garfield E. How can impact factors be improved? BMJ. 1996;313:411-3.
[PMID: 8761234]
7. Lotze A, Mitchell BR, Bulas DI, Zola EM, Shalwitz RA, Gunkel JH. Mul-
ticenter study of surfactant (beractant) use in the treatment of term infants with
severe respiratory failure. Survanta in Term Infants Study Group. J Pediatr. 1998;
132:40-7. [PMID: 9469998]
8. Jones RB, Schwebke J, Thorpe EM Jr, Dalu ZA, Leone P, Johnson RB.
Randomized trial of trovafloxacin and ofloxacin for single-dose therapy of gon-
orrhea. Trovafloxacin Gonorrhea Study Group. Am J Med. 1998;104:28-32.
[PMID: 9528716]
9. Green JR, Lobo AJ, Holdsworth CD, Leicester RJ, Gibson JA, Kerr GD, et
al. Balsalazide is more effective and better tolerated than mesalamine in the treat-
ment of acute ulcerative colitis. The Abacus Investigator Group. Gastroenterol-
ogy. 1998;114:15-22. [PMID: 9428213]
10. Capdevila X, Perrigault PF, Ramonatxo M, Roustan JP, Peray P, dAthis F,
et al. Changes in breathing pattern and respiratory muscle performance parame-
ters during difficult weaning. Crit Care Med. 1998;26:79-87. [PMID: 9428547]
11. Bender R, Grouven U. Logistic regression models used in medical research
are poorly presented [Letter]. BMJ. 1996;313:628. [PMID: 8806274]
12. Khan KS, Chien PF, Dwarakanath LS. Logistic regression models in obstet-
rics and gynecology literature. Obstet Gynecol. 1999;93:1014-20. [PMID:
10362173]
13. Bland JM, Altman DG. Transforming data. BMJ. 1996;312:770. [PMID:
8605469]
14. Katz MH. Multivariable Analysis: A Practical Guide for Clinicians. New
York: Cambridge Univ Pr; 1999.
15. Altman DG, De Stavola BL, Love SB, Stepniewska KA. Review of survival
analyses published in cancer journals. Br J Cancer. 1995;72:511-8. [PMID:
7640241]
16. Brenner H. A potential pitfall in control of covariates in epidemiologic stud-
ies. Epidemiology. 1998;9:68-71. [PMID: 9430271]
17. Reason J. Human error: models and management. BMJ. 2000;320:768-70.
[PMID: 10720363]
18. Godlee F, Gale CR, Martyn CN. Effect on the quality of peer review of
blinding reviewers and asking them to sign their reports: a randomized controlled
trial. JAMA. 1998;280:237-40. [PMID: 9676667]
19. Goodman SN, Altman DG, George SL. Statistical reviewing policies of
medical journals: caveat lector? J Gen Intern Med. 1998;13:753-6. [PMID:
9824521]
2002 American College of PhysiciansAmerican Society of Internal
Medicine
www.annals.org