Wed 23 Mar 2005
Andy S, last seen criticizing the Lancet study without reading it, has now read it. Sort of. He writes:
The Kurdish and Shiite provinces were not undersampled. The cluster for Dehuk was moved into adjacent Ninawa, so Dehuk was undersampled and Ninawa oversampled, but overall, the Kurdish provinces got the correct number of samples. Similarly, the southern provinces were not undersampled. I really don’t know why people keep making reckless and false claims about the Lancet study.Of Iraq’s 18 provinces only 12 were actually visited. … Now clusters assigned to the unsurveyed provinces were replaced in the sample by selecting clusters in adjacent provinces as proxies. The net effect of this is that of the five provinces in northern Iraq only Ninawa and Sulaymaniya were surveyed. …In a similar manner Iraqâ��s three southernmost provinces were left unsurveyed.
Somehow or other the Northern Kurdish population and the Southern Shiite population were undersampled whilst the Sunni provinces were completely covered!
I wonder how that happened?
Also, Shannon Love has had another go at the study in this post. He does have one reasonable point—that the summaries are unclear on which results depend on the inclusion of Falluja and which do not. In this passage,
Making conservative assumptions, we think that about 100000 excess deaths, or more have happened since the 2003 invasion of Iraq. Violence accounted for most of the excess deaths and air strikes from coalition forces accounted for most violent deaths.while the statement “Violence accounted for most of the excess deaths” is robust, that is, true whether or not Falluja is included, the statement about air strikes is only true if Falluja is included. This means we should have less confidence in this statement and the study does not make this clear. This is, however, a pretty minor flaw.
Unfortunately, Love goes on to destroy his credibility with this claim:
When you realize that without the Falluja data the study tells a very different story than the one widely reported and that the Falluja data could only have been collected with active collusion of the Baathist and the Jihadist who ruled Falluja at the time, the publication of this study assumes a very sinister cast. Either through intention or willful disregard, the researchers and publisher acted as a propaganda tool for the Fascist elements in Iraq. Given the degree to which they carefully spun their results, I conclude the effect was intended.Yes, he really did accuse the researchers and the Lancet of conducting a deliberate fraud on behalf of Islamic terrorists. This would be defamatory if it weren’t so completely silly.
And there is some robust discussion in the comments to Love’s post.
Update: Sigh. Brendan Nyhan cites Kaplan’s badly flawed critique.
March 23rd, 2005 at 12:32 pm
From the law end, I find the argument misses one large point. Let’s say the Lancet boys are in the dock giving evidence about what they saw in a brawl. Tough cross-examination, points made and lost.
Ok, but then what happens is their counsel stands up and says “I call the other witness.” The US Army climbs into the dock. Lawyer says “OK, that’s what they say happened. What do you say happened?”
Silence
“I’m sorry, I can’t hear you.”
“We don’t do body counts.”
“So you were there, you had a full opportunity to produce your own estimate, you had every motive to do so, I put it to this court that it’s virtually impossible to believe that you didn’t do so, and you won’t share your estimate with this court. Aren’t we entitled to believe that you feel it would not be advantageous to you? That, indeed, it would be less favourable to you than the Lancet estimate?”
If I was in the jury I know it’d sway me.
March 23rd, 2005 at 5:28 pm
Great point Chris. The fact that the Bush administration and its military are so intent upon avoiding the issue speaks volumes!
March 23rd, 2005 at 5:35 pm
Not only Bush wants to avoid the issue. In Britain there has been more of a debate over the casualty figures with several MP:s demanding that a better study be funded, but Blair refuses.
March 23rd, 2005 at 5:48 pm
Count the Casualties campaign: link
March 23rd, 2005 at 6:46 pm
I don’t agree that the summary is unclear. The alternative would be to ignore the Fallujah datapoint entirely, which since it is likely to be broadly representative of Najaf, Ramallah and Sammara, none of which were sampled, would be very wrong. The cum- and ex- Fallujah results are clearly marked as such; I really don’t think anyone’s been actually confused by this.
March 23rd, 2005 at 6:50 pm
“I really don’t think anyone’s been actually confused by this.“
Unless they really, really wanted to be!
March 23rd, 2005 at 8:19 pm
Good God that Shannon Love is a piece of work. His latest piece “Fisking Fallujah” basically abandons all his previous criticisms and more or less agrees that all evidence points to an ex-Fallujah 100K excess deaths, but then just starts picking nits on the summary and how it doesn’t bend over backwards to minimise the deaths and ignore the evidence.
I’m afraid I was rather rude to him in his comments section.
March 23rd, 2005 at 9:10 pm
Chris, spot on! Fact is, the US and UK governments don’t want to count the number of Iraqi’s killed in the occupation, because they are probably aware that the number is very high and this would reflect badly on the mendacious propoganda campaign they have waged for the past three years leading up to and including the invasion. I a sure if you check other cases of western aggression - Viet Nam, various countries in Latin America etc., you’ll also see that no official civilian body counts were made by ‘our side’. So the Iraq case is in keeping with recent history.
March 24th, 2005 at 2:51 am
This is, however, a pretty minor flaw.
It is obviously not a minor flaw. The study is being used to claim that 100,000 civilians were killed by US forces, neither of which claim it supports.
The “produce your own body count” argument is simply a variation on ad ignorantium.
March 24th, 2005 at 3:58 am
The Lancet study does NOT claim that 100,000 civilians were killed. The Lancet study does NOT claim that US forces killed all of them. Let us at least get that straight.
March 24th, 2005 at 4:08 am
claiming 100k “civilian casualties” = “picking nits”
implying the US was directly responsible for most or all deaths = “picking nits”
indiscriminately commingling excluded and included data = “picking nits”
Right. And you guys are scientists? Egads.
March 24th, 2005 at 4:57 am
claiming 100k “civilian casualties” = “picking nits”
The study does not do this.
“implying the US was directly responsible for most or all deaths = “picking nits”
The study does not do this
“indiscriminately commingling excluded and included data = “picking nits”
The study does not do this, even though Shannon Love thinks it does.
Right. And you guys are scientists?
Yep, and I’m guessing you’re not.
Egads
Jings crivens.
March 24th, 2005 at 5:24 am
“Yep, and I’m guessing you’re not.”
By way of pompous appeals to authority, this one’s a beaut. I personally doubt whether statisticians and/or epidemiologists have the necessary credentials to weigh in on civilian deaths in war. To my knowledge murder rates are not estimated via cluster sampling. How many other wartime surveys of this type can you reference, or is this in fact the first of its kind?
The misrepresentation of these results makes for a perfect case in abuse of statistics by non-statisticians on all sides of the ideological spectrum. You only seem to care when such abuse is directed at the survey itself. Whether or not the Lancet study makes these highly misleading claims, the editorial accompanying the study in the Lancet DOES.
Talk about picking nits!
March 24th, 2005 at 6:37 am
telluride, if your big beef is that the results of the study have been misstated in the media, then perhaps you should direct you complaints at the media for doing their usual shoddy job of reporting the news.
March 24th, 2005 at 8:45 am
Since it was in direct and pointed response to your “Right. And you guys are scientists?”, this one has to be called a fair hit by dsquared.
In fact, epidemiologists are exactly the discipline best qualified to conduct a study of this kind. Although (as the authors note) this was an extreme case, the same methods are commonly brought to bear in troubled or war-torn regions where mortality and morbidity figures are desperately needed. See, for example, references 18, 19, 21, 22 and 23 in the Lancet study itself.
March 24th, 2005 at 9:32 am
this one has to be called a fair hit by dsquared.
…fair enough. But re: verecundiam fallacies you might agree that non-statisticians are arguing for the study (and parroting its misinterpretation) as forcefully as its critics. If the study is meant to be useful as an instrument of policy, it needs to be comprehensible to laymen, and it obviously isn’t.
the same methods are commonly brought to bear in troubled or war-torn regions
I don’t believe the citations in the Lancet make for very good precedents. In all instances but one (the Kosovo study) the primary cause of death was actually disease, not violent death of a political character (this applies to the Congo study as well.) Targeted political violence is qualitatively different from malaria.
Also in the Kosovo study, 67 deaths represented a total death toll of 12,000 (1/179). The Lancet study is an order of magnitude less precise.
March 24th, 2005 at 10:05 am
“If the study is meant to be useful as an instrument of policy, it needs to be comprehensible to laymen, and it obviously isn’t.”
Certainly there is a problem with the way the study has been publicised. Most people who have heard of it have a faulty idea of what it shows. Either they think at least 100,000 civilians were killed, or they think it could just as easily have been 8,000 and that includes insurgents - or some such rubbish.
But the Economist newspaper published a very fair and readable summary. There is not much excuse for journalists and bloggers who failed to do that much. The paper is readily comprehensible to anyone who has done a good basic stats course.
March 24th, 2005 at 11:56 am
They do, in fact, appear to be undersampling Kurdish areas. Suleimania is the only Kurdish majority governorate that is sampled. The pairing leaves Dehok, Arbil and Tameen without a single cluster, and gives Suleimania only 3. Nineweh and Salahedin (both with large Sunni populations) get a total of 7.
In the one Kurdish governorate that they do sample (and which only represents an eleventh of the survey) they have a large drop in death rates. I wouldn’t make too much of that, though, because the numbers involved are too low. I’d guess, deaths before might have been 5 and after 2, adjust for the time span difference and we’ve got an apparent 2/3 drop.
If you look at their clustering, knowing the ethnic make-up of the provinces, it seems reasonably likely to me that Sunnis could be 45% of the sample, Shiites 55% and Kurds 10%.
If that is so, it’s poor, because Sunni areas are likely to have experienced the greatest increase in death rates, or respectively the smallest decline.
Turnout in Kurdish provinces was so high in January’s elections that I’ve come to doubt the census figures, which compounds the problem, because nearly a fifth of Iraq’s population seems to have been sampled as if it was less than a tenth (I had some interesting discussions over at Iraqelect.com just after the elections, trying to divine likely voter numbers).
March 24th, 2005 at 6:39 pm
it seems reasonably likely to me that Sunnis could be 45% of the sample, Shiites 55% and Kurds 10%.
Since those numbers add up to 110%, it doesn’t seem so likely to me.
March 24th, 2005 at 8:51 pm
dsquared, I think most Kurds are Sunnis so Heiko’s arithmetic may be ok. Of course that dilutes the force of her point that Sunni areas are likely to have experienced the greatest increase in death rates; that holds best for areas dominated by Sunni Arabs.
However the claim to “knowing the ethnic mix of Iraqi provinces” would invite scepticism even if it came from an Iraqi demographer, which it probably wouldn’t.
March 24th, 2005 at 11:37 pm
45 and 55 respectively were ex Kurds, sorry for not making that clear.
My spreadsheet numbers:
Shiite 3 Baghdad
Sunni 8 Anbar, Nineweh, Salahedin
Kurdish 3 Suleimania
Sunni 4 Baghdad
Sunni 2 Babil, Diala
Shiite 13 everything else
That’s 47% Sunni, and 53% Shiite among Arabs, and 9% Kurds out of the total (which generously round to 45, 55 and 10).
If you look at Sunnis and Shiites, and Kurds, as a fraction of all clusters, it’s 42, 48 and 9.
However, a number of governorates are mixed, namely, Baghdad, Salahedin, Nineweh, Babil and Diala, so Kurds (who are virtually all Sunni) and Shiite Arabs could represent more clusters. In fact, some clusters themselves could contain a mix of the three groups.
The ethnic mix of Iraqi provinces is fairly well known, at least as well as the raw population numbers, and certainly well enough for the above back of the envelope calculations.
March 25th, 2005 at 1:24 am
Heiko,
Contrast your view with that of Juan Cole (the context of his remarks is the idea of partitioning Iraq, which he opposes):
“Iraq is not divided neatly into three ethnic enclaves. It is all mixed up. There are a million Kurds in Baghdad, a million Sunnis in the Shiite deep south, and lots of mixed provinces (Ta’mim, Ninevah, Diyalah, Babil, Baghdad, etc.). There is a lot of intermarriage among various Iraqi groups.”
I know Cole is not your favourite commentator, but you will find similar comments on the blogs of Salam Pax and Zeyad, two Iraqis who are, broadly speaking, sympathetic to the whole regime-change thing. I am pretty sure that books on Iraq such as Dilip Hiro’s take a similar line, although it’s a while since I read them.
March 25th, 2005 at 1:51 am
Kevin,
on this Juan Cole and myself are in perfect agreement. Iraq is not neatly divided intro three ethnic enclaves. Said fact makes partition such an unlikely proposition (and from what I’ve read about his position on that matter it’s quite realistic and comparable to my own).
But there are provinces with very large majorities of the respective groups, namely Anbar, most of the South and the tree Kurdish governorates. Tameen has a very high Kurdish population, Salahedin and Nineweh many Sunnis.
I’ve taken account of this, and it doesn’t affect the central point. Anbar, Salahedin and Nineweh together appear to be oversampled and the 4 Kurdish dominated provinces in the North undersampled, and while we are only talking about the assignment of 2 clusters, this decision, particularly taking the measured death rates at face value, makes an enormous difference to the excess death estimate.
March 25th, 2005 at 1:54 am
But the ethnic thing is a red herring. “Kurds” in this context means “people living in the part of Iraq covered by the no-fly zone and/or the ‘buffer zone’, which would have seen a very different outcome from the rest of Iraq because there was nto so much fighting or insurgent activity there”. As far as I can see from this map, that region covers parts of Ninawa (I am surprised how close Mosul is to the zone) as well as Sulaymaniya, Arbil and Dehuk. It is pretty difficult to tell, but it seems to me that you would expect that a random GPS selection would have put all 3 of Sulaymaniya’s clusters into the PBZ, plus an expected 1 of Ninawa’s. Tameem has a lot of Kurds in it, but was not in the semi-autonomous region. I don’t see much evidence of a devastating critique here. It certainly does not “make an enormous difference” to the excess death estimate; there could not be a negative outlier as large as Fallujah was a positive outlier, because you can’t have a negative number of deaths.
March 25th, 2005 at 2:33 am
“knowing the ethnic make-up of the provinces, it seems reasonably likely to me that Sunnis could be 45% of the sample, Shiites 55% and Kurds 10%.“
This is certainly one of your better-thought-out arguments, though exactly how one would pull that off, I am not sure, unless someone discovered a new rule for percentages while I wasn’t looking. 45% Sunnis + 55% Shi`as + 10% Kurds = 110%.
March 25th, 2005 at 2:53 am
You can have a negative number of excess deaths, in fact, based on the measured death rates in Sulaimania, that’s exactly what happened there.
The map you use depicts the no-fly zones, not areas within the Kurdish autonomous region: link
Mosul has Kurdish dominated suburbs. One of the Nineweh clusters could be Kurdish dominated.
Ethnicity does factor into the equation not just because of violence, but also because Sunni areas got favoured treatment, like 24 h electricity, when some Shiite areas had to make do with 2 (while now often the inverse is true, as the insurgency is strongest in Sunni areas).
The pairing is unfortunate, because Nineweh clearly saw more coalition activitiy than Dehok, and Salahedin, which incidentally contains Tikrit and Samarra, saw more than Tameen.
The pairing does make an “enormous difference”. Ex Fallujah, all the excess death would disappear, if the weighting of Salahedin and Nineweh was reduced from 7 to 3 clusters, and the weighting of Arbil, Dehok, Sulaimania and Tameen was increased to 7, and all the raw death rates were the same.
This reweighting would be achieved, if say Dehok had been sampled and Tameen’s one cluster been given to Sulaimania instead of Salahedin. These two minor decisions, sample Dehok rather than Nineweh, and sample Sulaimania rather than Salahedin, would single-handedly destroy the 100,000 excess death figure, and instead, again assuming the measured death rates at face value, give lives saved.
March 25th, 2005 at 2:57 am
Barely and back of the envelope at any rate.
March 25th, 2005 at 3:02 am
Heiko, I don’t find your claim that moving just four clusters would remove the 100,000 excess death figure to be even slightly plausible. Please present your calculations or retract your claim.
March 25th, 2005 at 3:07 am
The back of the envelope depends on how one reads the graph. Is Nineweh up from 3 to 4, from 2.5 to 4, from 3 to 3.5?
Is Sulaimania down from 14 to 4, or from 14.5 to 3? Etc.
Depending on how one reads that off, sampling Dehok and Tameen, and assuming the same death rate reduction as in Sulaimania, one can get anything between a few lives saved and maybe 25,000 excess deaths.
March 25th, 2005 at 3:35 am
It wouldn’t be moving 4 clusters, but just 2 (ie Dehok and Tameen would be sampled, rather than reassigned to Nineweh and Salahedin).
Back of the envelope is as follows:
Deaths before were 46. An eleventh of that is just over 4. But Sulaimania has a death rate about triple the average pre-war estimate, so we’ve got to triple that to about 12, ie 11 or 12 or so of the 46 pre-war deaths were in Sulaimania.
If we go from 3 to 7 clusters with the same death rate, we add another 10*1.33 (10 being the difference between the average for 3 clusters in Salahedin and Nineweh pre-war and the 12 actual deaths in Sulaimania, note that Salahedin seems to be at 1 pre-war, which is really, really low, I assume the same for Anbar, but you can’t really read that off the graph easily).
We therefore go to 59 pre-war deaths. Now we multiply that by 1.4 and divide by 1.1 to adjust for person months, and get 75, compared to 78 after (ex Fallujah, and subtracting a few deaths in Nineweh and Salahedin, as they appear to have higher after war rates than Sulaimania, 7 instead of 4 is my rough guess looking at the graph). That would leave about 10,000 excess deaths.
This whole exercise is very sensitive to reading off the governorate death rates accurately, and is only illustrative, as I don’t think that the large death rate reduction in Sulaimania is a real phenomenon.
The pairing could also have assigned the Anbar cluster to Baghdad or Salahedin. It just goes to show how large the effect of getting the sampling wrong can be.
March 25th, 2005 at 4:07 am
And Tim clearly moving a single cluster can have an enormous impact, just consider Anbar.
I will concede though that upon checking my back-of-the envelope calculation again against the death rates indicated in the paper, it seems to me that I read them rather generously, and that a reduction to 25,000 excess deaths looks more correct.
I will also concede that I’ve given the calculation of this figure, as it is particularly hypothetical, not as much thought as I could have.
March 25th, 2005 at 5:17 am
Sorry, I was getting tired, you are, of course, correct that it’s moving 4 rather than just 2 clusters. Not only would Dehuk and Tameen have to be sampled, but they’d have to be reassigned one each from Nineweh and Salahedin to get from 3 to 7.
I’ll call it a day for today, as my concentration is clearly starting to go down seriously.
March 26th, 2005 at 9:15 am
Picking up after some rest.
The pre-war death rate in Sulaimania is so large (at nearly 15) and the rate in Salahedin (apparently around 1) so low, that shifting the weighting of these governorates around would have an enormous impact on the pre-war death rate.
So far so obvious, but this may be just due to chance, in which case the reassignment would have a much lower effect.
But, it may also be “real” in the sense that Kurds may have made up pre-war deaths, and Sunni Arabs not talked about them (due to their political stance, a handful of people lying for political reasons would do the trick).
And, more importantly, the authors should simply not have done the pairing in this fashion because they must have known that death rate movements in Kurdish areas might conceivably differ considerably from the two Sunni dominated provinces (be it because of violence or the favours Saddam was showering on Sunni Arabs, while he was in power), and that a good argument could be made, as to which provinces were more likely to experience a decline.
This is one of the cases, where there is at least the appearance of possible author bias impacting the results.
March 26th, 2005 at 9:41 am
Umm, Sulaymaniya was oversampled, so you seem to be suggesting that the this biased the results down.
March 26th, 2005 at 10:26 am
All right, well yes, if Sulaimania had a much higher death rate decline than Arbil ….
Anything a little more substantial? How do you defend the pairing of Tameen with Salahedin, and of Dehok with Nineweh?
And do you accept that given the death rates as measured (which may be as they are for a number of reasons, and some or all of it may be due to factors other than chance), applying them to 7 Kurdish clusters, and only 3 in Nineweh and Salahedin would have a rather material effect on the measured excess death?