Deaths in the Iraq War

Unfinished Business

An Arena of Controversy

An Arena of Controversy

We will Sort out the Truth Today

Really - I’m not joking

The truth becomes abundantly clear once you organize all the data sources into a single graph and investigate these sources.

I’ll present two graphs, actually, but they are fairly small variations on each other.

Violent Deaths in the Iraq War at a Glance

The Big Picture

We can see two separate universes in the picture

1. In Universe 1 the number of violent deaths seems to grow exponentially, reaching 1,000,000 already in 2007 and after that - who knows.
2. In Universe 2 the number of violent deaths grows steadily to the point where now it probably exceeds 300,000.

No Good News

Both of these universes are disturbing.

But they cannot both be true.

Those who value the truth are confronted with an important question: which universe should we choose?

Background

All the sources except for Iraq Body Count are surveys

All the numbers in the picture are for violent deaths (not excess deaths) and are cumulative totals from the beginning of the war (March 2003) up through that point.

There is a dot in the graph when a survey provides just a single estimate and full curves when there are monthly estimates (or tallies in the case of Iraq Body Count).

Background - The Surveys

All but one of the surveys are household surveys which means that respondents are asked about deaths and living people within their households, i.e., people living under the same roof.

The remaining survey asks people about the fates (living or dead) of their siblings and is based on the same sample as one of the household surveys.

Estimation methods are different for these two types of surveys but the underlying idea is the same - to extrapolate from in-sample deaths to deaths for the whole population of Iraq.

Iraq Body Count

The methodology of Iraq Body Count (IBC) is qualitatively different from the methodologies of the surveys - in brief:

“Iraq Body Count restricts its published database to documented (not inferred, extrapolated or otherwise estimated) deaths of civilians from post-invasion violence in Iraq, established to the standard of evidence specified below.”

Iraq Body Count

“Deaths in the database are derived from a comprehensive survey of commercial media and NGO-based reports, along with official records that have been released into the public sphere. Reports range from specific, incident based accounts to figures from hospitals, morgues, and other documentary data-gathering agencies.”

IBC Augmented

The main IBC methodology, as sketched on the last slide, guides the documentation of civilian deaths but IBC also has compiled information on combatants killed.

I integrate these combatant figures into the pictures in this presentation - doing so is required for making head-to-head comparison with the surveys because these surveys mix combatants with civilians.

IBC Augmented

I have also augmented the IBC numbers with estimates of the number of civilians deaths in the US official database, Sigacts which was brought into the public domain by Wikileaks, that will eventually be absorbed into the IBC database.

This estimate is based on drawing a random sample from the Sigacts data and matching the events in the sample against deaths in the IBC database and then extrapolating the matching outcome (and non-matching) outcome from the sample to the full population of Sigacts events.

Universe 1

Universe 1, consisting of the ORB survey, the Burnham et al. (2006) survey and the Roberts et al. (2004) is a universe of alternative facts that we should discard.

I explain this judgement in the next few slides.

The ORB Survey

ORB actually conducted a series of surveys - all but one of these surveys asked people about deaths in their families.

One ORB survey (purportedly) asked people about deaths in their households, rather than their families, and this household survey is the source of their estimate of 1 million violent deaths.

The ORB Survey - Families versus Households

Respondents might report deaths of uncles, aunts, cousins and even close family friends when asked about their families.

So we expect much larger numbers of reported deaths for a family question than we do for a household question.

And this is what ORB reported …..except in 4 governorates.

The ORB Survey - Families versus Households

In these 4 governorates, that account for 80% of the violent deaths in the survey, the percentages or respondents reporting deaths of household members were substantially higher than the percentages reporting family members killed!

These results are, simply, not credible - see this paper for details.

Burnham et al. (2006)

The principle author of this paper was censured by the American Association for Public Opinion Research for refusing to disclose fundamental features of his survey such as how he drew his sample.

Gilbert Burnham was also suspended by Johns Hopkins University for telling the university’s ethics committee that he would not collect unique identifiers of respondents, thus receiving a light-touch ethical review, but his team went ahead and collected unique identifiers anyway.

Burnham et al. (2006)

Worse, there is is evidence of fabricated data in the survey.

To take just one example, there is a cluster in Baghdad that recorded 24 violent deaths in early July 2006 in car bombings, spread across 18 households, .

A cluster consist of 40 contiguous households.

Burnham et al. (2006)

There was, indeed, a big bombing in Baghdad during that time period but it was a bombing of a market.

Did 24 neighbors go to the market together one day all holding hands and wind up at the wrong place at the wrong time?

I don’t think so.

Burnham et al. (2006)

That was just one particularly egregious piece of evidence of fabrication example taken from a mountain of such evidence.

You need to read the paper linked to above to appreciate the full scope of this evidence.

In short, though, Burnham et al. 2004 is also not credible.

Roberts et al. (2004)

This is the third pillar of what I’ve called the alternative facts universe and I think it would be OK to pass over it on the grounds that all potentially corroborating evidence has already collapsed.

This observation is all the more true given that Roberts et al. (2004) is the most extreme of the three surveys in the alternative facts universe, reaching above 230,000 deaths already in September of 2004.

Still, it is worth noting that fully 52 of the 71 violent deaths in the sample come from a single cluster in Fallujah.

Where Things Stand

The alternative facts universe has collapsed and we are left with the 5 sources of Universe 2, Iraq Body Count and the 4 surveys.

The figure shown many slides ago show them to be remarkably consistent with each other.

Yet some of these sources do have some issues that we should discuss.

Iraq Family Health Survey (IFHS)

Iraq Family Health Survey

The “arbitrary fudge” upwards, referred to in an article in Science quoted in the previous slide, was really 50%, not 35% as it says above.

It is the difference between 151,000 and roughly 100,000, which is 50% if you take 100,000, rather than 151,000, as the base for calculation.

Iraq Family Health Survey

The IFHS made a further, less arbitrary, upward adjustment from roughly 80,000 to 100,000 to account for the fact that excessive violence had prevented them from doing interviews in some of the clusters that had been selected in Baghdad.

However, the way that the IFHS did its missing cluster adjustment is questionable.

Iraq Family Health Survey

They pencilled in a quantity of violent deaths for the missed clusters exactly sufficient to bring the ratio of violence in Baghdad to violence in the rest of Iraq up to parity with this same ratio for IBC.

However, it is plausible that IBC is better able to detect violence in Baghdad than it is at detecting violence in the rest of Iraq - so the missing cluster adjustment, which has a huge effect on the final IFHS estimate, was excessive.

University Collaborative Iraq Mortality Survey (UCIMS)

When respondents reported deaths to the UCIMS there were follow up questions asking these respondents to produce death certificates to confirm their reported deaths.

Many of these respondents replied either that they did not have death certificates or that they did have them but, nevertheless, did not show them to the interviewers.

University Collaborative Iraq Mortality Survey

There can, of course, be real violent deaths that were reported correctly but that are not backed by death certificates.

Still, there is a cloud over each of these reports.

Introducing the Final Picture of this Talk

The following picture has exactly three changes compared to the first picture from many slide ago.

1. I have eliminated the “arbitrary fudge” and the missing cluster adjustment for the IFHS.
2. I have eliminated deaths not backed by death certificates in the UCIMS
3. I eliminated the 52 deaths in the Fallujah cluster of Roberts et al. (2004)

A New Picture Using Lower Estimates

Extraordinary Consistency

The sources in Universe 2 were already quite consistent with each other in the first picture but not the consistency is extraordinary.

We have a pretty clear picture of the death toll in the Iraq War - more than 300,000 people killed.

A Bloodbath

The sources in the alternative facts universe exaggerate deaths by as much as an order of magnitude.

But as we absorb this sad fact into our thinking we shouldn’t lapse into minimizing the true human cost of the Iraq War - it has been massive.