Iraqi Death Estimates Celled Too High; Methods Faulted 2 The recently published study from a Johns Hopkins

team of epidemiologists regarding mortality in Iraq 1 met with several widely publicised responses, one of which was this: George W. Bush immediately dismissed the study, characterizing its methodology as pretty well discredited. p1 li 10 2 I think it is important to dwell for somewhat longer than Bohannon had space to on the implications of Mr Bushs statement. The Centres for Diseases Control, The World Health Organisation (WHO), the Training in Epidemiology and Public Health Network, and others, recently convened to examine training of public health professionals in the context of Violence and Health. A review of the role of epidemiologists was the result 3. Of interest, after an admittedly poorly specified literature review, the authors noted that rapid survey methods are most often used to describe the current needs of a population in conflict, and furthermore The most common approaches for selecting participants in rapid population surveys during conflict are simple random sampling and cluster sampling. Cluster sampling for rapid surveys represents a trade off between precision and the cost of data acquisition. It has a long pedigree in the non-conflict field, and is perhaps most widely known for its use within the WHOs Expanded Program of Immunization (EPI) 4,5. For those so interested, I provide a few references, by no means an exhaustive catalogue, of its use in the immunisation setting 6,7. A glance at the titles will reveal that this method has been of importance in some of the major successes of modern medicine: the eradication of smallpox, and near eradication of poliomyelitis. The method has since been extended to fields such as assessment of bed net distribution for malaria prevention 8, use of drugs to prevent malaria during pregnancy 9 and is providing more accurate insights into the dynamics of the current Sub-Saharan HIV epidemic 10. It is important to realise that the classic WHO 30 cluster, 7 household survey is not much good for rare events 4, at least if the confidence interval is to be narrow. Mortality, thankfully, is a comparatively rare event. The method however is standard and its weaknesses and modifications well discussed. This has not provided a barrier to epidemiologists using it to study mortality related to conflict, often with a modification of the cluster or household numbers, for example in the Democratic Republic of Congo 11, Mozambique 12, Kosovo 13 and Sudan 14. Once again I do not present an exhaustive literature search. Recently there has been a comparison of systematic and cluster methods for assessing mortality in the same population, near contemporaneously 15. Although there were differences in some indices, none were statistically significant, and distributions were well preserved. Not surprisingly the confidence intervals for cluster sampling were wider and some estimates were not possible. As stated above, this represents the

compromise made for expensive data. In the case of Iraq, where those thought to be associated with the invading forces have been systematically threatened, targeted, and sometimes killed 16, the compromise would seem justifiable. I hope this goes some way towards explaining just how the methodology chosen by Burnham et al has been pretty well discredited. The Method May Be Sound, But Several Critics Question The Way It Was Carried Out In This Study 2 Amusingly, the usual charge of political motivation is rolled out. As is almost universal this charge is made by politicians, and so I think we can allow it to detain us no further. Actually, I think we should dally just long enough to consider scientific bias. It is of course well recognized, and can permeate to the highest levels. Rather famously we have the chicanery of the tobacco industry, with falsified results, scientists secretly in their pay, expert witness scientists who spend more time in the dock than the lab, and so on 17, 18. Perhaps more informative is the bias introduced by pharmaceutical companies. A recent study found significant biases favouring positive outcomes for drug company products, despite robust study methodology: the mechanism being to choose carefully weighted questions and publication bias 19. The charge of such abuse of the scientific method is serious, and as such should be backed up with evidence as in the two case cited above. I am not yet aware of any such evidence pertaining to Burnham et al 1. I would place Dr Hicks concern regarding sampling times in this category. Thankfully the rest of the article moves to areas less distasteful. The sampling technique has been criticised by a Johnson et al 20, as expressed in Bohannon 2 and in various internet releases. The proposition is that the technique used by Burnham et al 1 leads to a higher probability of occupants of houses near main streets being interviewed. The assumption is that these occupants have a higher mortality rate, and hence the Burnham numbers are an overestimate. The rather complex random sampling technique is somewhat paraphrased by Bohannon as the paper indicates that the survey team avoided small back alleys for safety reasons 2, but I think we should ignore the paradox that the team surveyed more dangerous areas by avoiding the most dangerous ones and excuse it as journalistic exuberance. With regard to Johnson et al 20, the interested reader will peruse their derived ratio of sampled mortality probability versus overall mortality probability. Admittedly applicable, it would be nice to see how this fits with map based probabilities using Burnhams stated method of household location. Even then the model relies on certain assumptions regarding population mobility and relative risk of mortality, neither of which we have much idea about for Iraq. As it is we are just invited to look at google maps and make up our own minds. Actually the lack of knowledge of main street bias coefficients could be calculated. Serendipitously there is a previous study, Roberts et al 21, which asked the same question in the same area, but was able to use a random GPS based starting point

rather than a main street identification. As the data for the two studies largely agree it would seem that the bias is negligible. Perhaps the more mathematically able reader could use the paired data to estimate a value for Johnsons bias coefficient in Iraq. I think it prudent to discount the claim that it is a crime 2 to forego challenging the lack of detail about the neighbourhoods surveyed. Firstly, the authors may very well have good reason for not releasing primary data, not least as the survey guaranteed anonymity. In any case rather than claiming it is a crime it may be wiser to adopt standard scientific practice of drawing conclusions from available data, and specifying what the limits of these conclusions are without more data. In summary I think that the way the methods have been faulted, at least according to Bohannon is as follows: The President of the USA, whatever you may say about him, is not an epidemiologist, and he has provided conclusive proof of this fact with his first statement. Lets leave him out of this. Either Burnham et al have faulty ethics and are pressing a point rather than doing science. This can be difficult to show, especially with partial data, but it is a pretty serious accusation to be carried in the pages of a journal such as Science without corroboration. Or they applied a standard technique erroneously. The major problem with the paper is suggested to be the sampling frame. No others are mentioned here. The quantification of this error as provided by the critics is at best partial, and largely theoretical. I have provided a suggested route for estimating it, but in any case a nonsusceptible survey matches this one. In General When Roberts et al 21 was published, there was initial furore, denial, anger, and probably quite a lot of glee from certain leftist groups. The quality of criticism in the media was on the whole devoid of epidemiological basis, and even the UK government had to dismiss it because it was based on an extrapolation, and because it did not chime with other, less systematic methods. Personally I hold epidemiology to be among the saving graces of humanity. I seriously think that it gives us more insight into our human world than any other branch of science. The problem is that it often demonstrates things that are otherwise hidden from view, and as it is a human science, often they have direct implications about human behaviour. Doll after all began by assuming that tarmac caused lung cancer, and came under severe pressure to play down the smoking thing 22. There has been political conflict since the very beginning of the public health movement, which is based upon sound epidemiology 23, but eventually policy changes have been made for the public good, and often at considerable private expense (consider the case of the London water companies).

Although I fully accept that the cluster sample study of Burnham most recently published is not without its weaknesses, I am dismayed at the attention it has received. Firstly we are quite happy to rely on similar methods to ensure our global health interventions work. We are usually quite happy to base our self-congratulation in this regard firmly on their shoulders when one considers smallpox, polio and measles for example. We have even gone so far as to base our calls for armed conflict on this method. Furthermore, it is disturbing to find criticism I would excuse even for a broadsheet newspaper to spread to the pages of science journalism. The scrutiny applied to this paper raises the image of a group of vultures picking at bones. I know certainly that nothing I could write would withstand such an onslaught intact. And what is more neither can any of the papers I cite here, not least those of Johnson et al. It is the foundation of science to be ethically committed to the dissemination of the truth. We have developed a method to seek out truth and test it. It is deeply disingenuous to selectively malign pointers towards this truth because we might not like their flavour, especially so when we are lucky enough to know the correct mechanisms to test assertions we find challenging. Elaborate mud-slinging is not among these. On a final note, why on earth are we surprised? The conflict papers cited here are fairly uniform in their conclusion that conflicts kill people, and predominantly they kill civilians. For further evidence I suggest a perusal of the history of conflict epidemiology 24. Estimated civilian mortalities are as follows: Vietnam 2x106, Korea 3x106, WWII 33x106, Spanish Civil 1x106, and WWI 10x106. War injuries form the eight, sixth and twelfth most common cause of death for the age groups 5-14, 15-29, 30-44 respectively, with the actual numbers being 45x103, 95x103 and 75x103 annually, based on techniques far less robust than those of Burnham and Roberts. Warfare, via public health and infrastructure disruption has at times resulted in more deaths than through direct trauma. 1 2 3 4 Burnham et al, Mortality after the 2003 Invasion of Iraq: a Cross Sectional Cluster Sample Survey, Lancet 2006, 368, 9545, 1421 Bohannon J, Iraqi Death Estimates Called Too High; Methods Faulted, Science 2006, 314, 496 McDonnell S et al, The role of the Applied Epidemiologist in Armed Conflict, Emerging Themes in Epidemiology 2004, 1,4 Bennett S et al, A Simplified General Method for Cluster Sample Surveys of Health in Developing Countries, World Health Statistics Quarterly, 1991, 44, 98

Hoshaw-Woodard S et al, Description and Comparison of the Methods of Cluster Sampling and Lot Quality Assurance Sampling to Assess Immunization Coverage WHO/V&B/01.26, 2001 Henderson RH, Assessment of Vaccination Coverage, Vaccination Scar Rates and Smallpox Scarring in Five Areas of West Africa, Bulletin of the World Health Organisation, 1973, 48, 173 Balraj V & John TJ, Evaluation of a Poliomyelitis Immunisation Campaign in Madras City, Bulletin of the World Health Organisation 1986, 64, 6, 861 Grabowsky M et al, Distributing Insecticide Treated Bed Nets During Measles Vaccination: A Low Cost Means of Achieving High and Equitable Coverage, Bulletin of the World Health Organisation, 2005, 83, 195 Fylkesnes K et al, Studying Dynamics of the HIV epidemic: Population Based Data Compared With Sentinel Surveillance in Zambia, AIDS, 1998, 12, 10, 1227

7 8

10 Holtz TH et al, Use of Antenatal Care Services and Intermittent Preventive Treatment for Malaria Among Pregnant Women in Blantyre District, Malawi, Tropical Medicine and International Health, 2004, 9, 1, 77 11 Coghlan B, Mortality in the Democratic Republic of Congo: a Nationwide Survey Lancet 2006, 367, 9504, 44 12 Cutts FT et al, Child and Maternal Mortality During a Period of Conflict in Beria City, Mozambique, International Journal of Epidemiology, 1996, 25, 2, 349 13 Spiegel P & Salama P, War and Mortality in Kosovo, 1998-1999: an epidemiological testimony, Lancet 2000, 355, 9222, 2204 14 Get the Sudan Reference 15 Rose AMC et al, A Comparison of Cluster and Systematic Sampling Methods For Measuring Crude Mortality, Bulletin of the World Health Organisation, 2006, 84, 4, 290 16 Sands P, Interpreters Used By British Army Hunted Down By Iraqi Death Squads, Independent, World News, 2006, Nov 17th 17 KcKee, Smoke and Mirrors, European Journal of Public Health, 2000, 10, 161 18 Friedman LC et al, Learning from the Tobacco Industry about Science and Regulation, American Journal of Public Health, 2005, Sup 1, 95, 516

19 Lexchin J et al, Pharmaceutical Industry Sponsorship and Research Outcome and Quality: a Systematic Review, British Medical Journal, 2003, 326, 1167 20 Johnson et al, Bias in Epidemiological Studies of Conflict Mortality, http://www.rhul.ac.uk/economics/Research/conflict-analysis/iraqmortality/index.html accessed 24/11/06 21 Roberts et al, Mortality Before and After the 2003 Invasion of Iraq: a Cluster Sample Survey, Lancet, 2004, 364, 9448 22 Richmond C, Sir Richard Doll, British Medical Journal, 2005, 331, 295 23 Sram I & Ashton J, Millenium Report to Sir Edwin Chadwick, British Medical Journal, 1998, 317, 592 24 Garfield RM & Neugut MI, Epidemiologic Analysis of Warfare, a Historical Review, Journal of the American Medical Association, 1991, 266, 5, 688