Deltoid

0. Introduction

Volume 87:4 of the Journal of Criminal Law and Criminology contains three articles on the issue of the frequency of defensive gun use. The first presents David Hemenway’s critique of Gary Kleck’s 2.5 million estimate, the second is Kleck and Gertz’s reply and finally Tom Smith of the National Opinion Research Center comments on both papers.

I’ll try to summarize the arguments and comment where I think they are wrong.

1. Hemenway’s critique

1a. False Positives

Hemenway’s critique has two main arguments. The first is the problem of false positives. Let me define some terms first: FPR is the the false positive rate (the percentage of people without a DGU who claim to have had one). FNR is the false negative rate (the percentage of people with a DGU who deny having had one). P is the number of true positives (people with a DGU, whether or not they admitted it). N is the number of true negatives (people without a DGU, whether or not they claimed to have had one).
FP=FPRxN is the number of false positives.
FN=FNRxP is the number of false negatives.

Even if FNR is 38 times greater than FPR, Kleck’s estimate is too high by a factor of 17. This is because what determines whether an estimate is too high or too low is the relative sizes of FP and FN, not the rates. Even if Kleck is right and DGU is common, the great majority of people have not had a DGU in the past year. N is at least 75 times P and hence FP is larger than FN.

To put it another way, even if FPR is quite small (1.3%), Kleck’s would be an extreme overestimate. Hemenway then gives several examples of surveys (for example, on personal contact with space aliens) with FPRs of this magnitude or higher.

1b External validity

The false positive argument does not prove that Kleck’s number is an extreme overestimate. It merely demonstrates that it is probable. Hence we come to Hemenway’s second argument: All checks for external validity show that the estimate is highly exaggerated.

The checks that the Kleck estimate fails are with: NCVS estimates of burglaries,
Atlanta study on home invasion crimes,
NCVS estimates of sexual assaults,
NEISS estimates of gunshot wounds, and
UCR counts of homicides.

These arguments allow Hemenway to conclude that Kleck’s estimate is not reasonable.

2. Kleck and Gertz’s reply.

They open with an ad hominem argument: Don’t trust Hemenway - he’s associated with HCI. I hope I don’t need to point out what is wrong with this argument. Kleck and Gertz are rather heavy with the rhetoric throughout their reply. For example, they complain about the “illegitimacy” of Hemenway’s “idle speculation”. They assert that Hemenway’s critique is neither honest nor scientifically-based. They claim that the NCVS based DGU estimate is “dead” and Hemenway is the only one left who believes it isn’t.

2b False positives

Anyway, let’s consider their response to Hemenway’s “false positive” argument. Kleck and Gertz concede that there may be some false positives but argue that there are even more false negatives.

Hemenway finds a FPR of 1.3% and a FNR of 50% plausible. Kleck and Gertz counter by stating that a FPR of 0.2% and a FNR of 50% is more realistic. It seems that they only disagree about the FPR. Kleck and Gertz argue that:

1. Most DGUs involve illegal behaviour on the part of the defender

2. Therefore the FPR and the FNR will be similar to that for other illegal behaviour such as illegal drug use.

3. Studies of illegal drug use show that FN is much greater than FP.

There are some problems here:

1. Kleck and Gertz offer no evidence for their claim that gun ownership is usually illegal and hence most DGUs involve some illegality.

2. Even if DGUs are usually illegal it does not follow that the FPR will be similar to that for behaviours that are always illegal. If you are inventing a DGU it as possible to invent one that involves no illegal behaviour on your part. This is not possible for illegal drug use.

3. Kleck and Gertz present two studies that measured the FPR for illegal drug use. Pooling them, I find that the FPR was 1 in 70 or 1.3%. That is exactly the rate suggested by Hemenway and much greater than the rate that Kleck and Gertz claimed the studies supported. The reason why FN was greater than FP in these studies was that over half the population studied (convicts) were users of illegal drugs. Not even Kleck believes that over half the population of the US has a DGU each year.

To summarize: even the data presented by Kleck and Gertz on the “false positive” question supports Hemenway’s position.

2b External validity

Kleck and Gertz crank up the rhetoric even further here. They assert that checks for external validity “have repeatedly confirmed our estimates”. They claim that Hemenway’s “fallacious argument” is based on what is merely a “misperception” of inconsistency.

Anyway, Kleck and Gertz’s resolution of the inconsistency between their survey and the NCVS counts of crimes such as burglary is quite simple — the NCVS is wrong and their survey is right. They argue that DGUs typically involve unlawful gun possession and hence that the crimes defended against will not be reported to the NCVS at all. There are some problems with their argument:

1. Kleck and Gertz offer no evidence for their claim that gun ownership is usually illegal and hence most DGUs involve some illegality.

2. 64.2% of their DG users said that the police were informed of the incident. If they are going to tell the police (who might arrest them for illegal behaviour) about it, why wouldn’t they tell the NCVS (who can’t arrest them) about it?

3. Kleck and Gertz included a question about burglary in their survey. The estimate of the frequency of burglary that you get from this agrees with the NCVS count. Apparently we are supposed to believe that because respondents don’t want their illegal DGU to be found out by Kleck and Gertz’s interviewers they refuse to mention the burglary during which it occurred. But that they will tell the interviewers about this illegal DGU when asked if they’ve had a DGU.

4. Even if we accept the dubious premise that none of Kleck and Gertz’s DGUs against burglary are counted in NCVS burglaries, Kleck-Gertz’s result is still inconsistent with the NCVS. Instead of an impossible 80% of “at-home” burglaries involving a DGU, we get an impossible 40%.

Kleck and Gertz also claim that Hemenway’s logic is fallacious in drawing implications about the validity of the survey from estimates (such as that for the number of DGUs against burglary) made from smaller subsets of the data. They claim that the estimate of the total number of DGUs is based on a very large sample (n=4977) and hence is reliable, while the estimate for DGUs against burglaries is based on a far smaller subsample (194 DGUs, 40 against burglars) and hence is less reliable. Kleck and Gertz forget to mention that their 2.5 million total DGU estimate is based on the 67 out of the 194 DGUs that were personal and occurred within the past year. Since 40 is less than 67, the estimate for DGUs against burglary is indeed less reliable, but not by much.

In any case, whether an estimate is more or less reliable than another estimate is irrelevant. What is important is whether or not the estimate, after allowing for its unreliability, is inconsistent with the external check. A 95% confidence interval for the estimate of the number of criminals shot (the least reliable of the estimates) is 100,000-300,000. The low end of that estimate is still wildly inconsistent with NEISS counts of gunshot wounds.

Kleck-Gertz’s final counter to the external validity argument is to claim that there is no inconsistency with the NEISS count of gunshot wounds treated in hospitals because criminals would avoid treatment for minor DGU related wounds. However, the NCVS indicates that 90% of assault gunshot wounds are treated in a hospital, and the NEISS shows that about half of gunshot wounds are serious enough to require hospitalization. Consequently there should have been about 90,000 DGU related woundings serious enough to require hospitalization. Needless to say, this is still inconsistent with the NEISS.

3. Smith’s comments

Smith’s article (entitled “A call for a truce in the DGU war”) attempts to reconcile the Kleck-Gertz and NCVS estimates for DGUs.

3a NCVS estimate of DGUs

Smith believes that the NCVS undercounts DGUs for two reasons. The first reason is that DGUs against activities like trespassing are not counted. The second is that it does not ask a direct question about gun use. Smith dismisses Kleck and Gertz’s speculation that people would be more likely to tell their interviewers about sensitive incidents than Census Bureau employees, noting that the survey literature does not support their speculation.

3b Kleck-Gertz estimate of DGUs

Smith believes that Kleck-Gertz over-counts DGUs for several reasons. The first reason is telescoping. While this would be partly balanced by forgetting, Smith cites studies that indicate telescoping is greater than forgetting. The second reason is that the other studies that roughly agree with Kleck-Gertz tend to give lower estimates. The third reason is false positives, though Smith notes that neither Kleck and Gertz nor Hemenway have provided conclusive evidence on what the false positive rate would be, so it is hard to quantify this effect. The fourth reason is all the internal and external inconsistencies of the Kleck-Gertz survey (too many DGUs by respondent as opposed to other household members, too many DGUs within one year as opposed to within five years, too many woundings, too high a proportion of DGUs by women, 20% of DGUs occurring in gun-free homes and so on.)

3c Resolving things.

Smith states that more research is needed to resolve the issue and suggest that adding a few questions to the NCVS would be the best way to do this.

Dr. Paul H. Blackman writes:

Just a very few comments: Folks can report an incident to the police without reporting gun use. I once fetched a gun to encourage some burglars to leave, and reported the burglary but not the gun use, since the gun possession was unlawful.

The Kleck estimate is not just inconsistent with police records of gun use against crime, but with police records of crime. There were 300,000 DGUs amongst the robberies known to the police? Are we to suppose that criminals seek out armed victims.

Kleck/Gertz have a nice response to the ludicrous Hemenway reliance on Kellermann’s home invasion data as part of Hemenway’s argument that Kleck/Gertz’s data are implausible.

Really? It would seem that they felt that this nice response was less compelling than the ad hominem argument, because it got left out of the published paper.

Kleck’s rejection of the NEISS data on the number of gunshot wounds treated in emergency rooms is consistent with many previous estimates on the number of gunshot wounds annually. Kleck’s view that criminals wounded will not seek treatment unless the injury is life threatening is not undermined by the NEISS data that many emergency room treatments lead to hospitalization; persons can be hospitalized for injuries which are not life-threatening.

Untreated serious gun-shot wounds are likely to become life-threatening. Consider the mortality rate for gun-shot wounds in the days before antibiotics. In any case, even if we accept Kleck’s dubious estimate for the percentage of gunshot wounds that are life-threatening (this is a lower percentage than most previous estimates for the FATALITY rate), and even if we accept his dubious claim that NONE of the other wounded criminals would get treatment, it still implies the absurd proposition that most of the hospitalizations for assaultive wounds were actually life-threatening DGU-related wounds.

While Tom Smith does, indeed, say more research is needed, his estimate of protective gun uses is closer to the low end of the Kleck range than to the Hemenway estimate.

You are mistaken. He produces two estimates - one, based on Kleck’s estimate, is closer to Kleck’s estimate, while the other, based on the NCVS, is closer to the NCVS.

bob (really Edgar Suter?) writes:

From an early DRAFT of Gary Kleck’s TARGETING GUNS : FIREARMS AND THEIR CONTROL scheduled to be published this month:

The Medical/Public Health Literature on Guns and Violence

False Citation of Prior Research

One final problem in the medical/public health literature on guns-violence links is so widespread, serious, and misleading that it deserves extended attention: the false citation of previous research as supporting anti-gun/pro-control conclusions, buttressing the author’s current findings, when the studies actually did no such thing. There is probably no way to persuade readers that large numbers of supposedly competent and honest scholars would do this, other than by listing a large number of specific instances.

Consider, for example, a widely cited article by Arthur Kellermann and his colleagues[Kellermann AL, Rivara FP, Rushforth NB et al. “Gun ownership as a risk factor for homicide in the home.” N Engl J Med. 1993; 329(15): 1084-91. st p. 1090], wherein they claim that “cohort and interrupted time-series studies have demonstrated a strong link between the availability of gun and community rates of homicide,” citing four previous studies in support (their cites 2 and 15-17). Naive readers might assume that Kellermann et al. were citing four “cohort and interrupted time-series studies” that had empirically documented a guns- homicide association.

Well, those naive readers who just read the sentence Kleck quoted, but not this one which Kleck forgot to tell us about:

“Previous studies of risk factors for homicide have employed correlational analysis[15] or retrospective cohort[16] or time-series[17] designs to link rates of homicide to specific risk factors.”

In fact, the first of the cited publications was not an empirical study at all, but rather a review of the literature (their cite 2) which did not support this assertion. The part of this report cited by the Kellermann et al. (pp. 42-97) did not even address the issue in question,

Because it was cited in support of the claim that “Homicide rates declined in the United States during the early 1980s but rebounded thereafter.[2]”

Since Reiss and Roth was also cited in support of Kellermanns claims about a guns-homicide link Kellermann should not have made the cite so specific to that section. This is a pretty minor quibble by Kleck.

while the part that did address it did not conclude that there was a strong guns-homicide link. Instead, the review authors cited an earlier review as having indicated that studies “qenerally find that greater gun availability is associated with … somewhat greater rates of felony murder, but do not account for a large fraction of the variation,” and drew an unmistakably “no decision” conclusion, noting problems of causal order that Kellermann and his medical colleagues have consistently ignored (Reiss and Roth 1993, p. 268, emphasis added).

It would seem that Kleck did not read as far as the end of page 268, where Reiss and Roth write “The strongest empirical evidence on how variations in firearms availability affect levels of violent crime and felony homicides can be obtained from carefully controlled evaluations of changes in laws and especially in enforcment efforts, that reduce firearms availability.” They then go on to summarize these evaluations in table 6-1. This table shows 5 evaluations of which 2 were found to be fully effective, 2 partially effective, and 1 not effective. The 2 partially effective interventions were found to have reduced gun homicides and had no consistent effect on robberies, assaults and non-gun homicides.

It is incorrect for Kleck to characterize this as “no decision”.

Among the remaining three supposedly supportive studies, one did not even measure “the availability of guns,” nor did its authors claim to have done so, and thus it could not possibly have established any guns-homicide association, never mind a strong one (their cite 17, to Loftin et al. 1991). This was an interrupted time-series study, but neither it nor any such study has ever measured any association between gun availability and homicide rates.

There would seem to be a major difference of opinion between Kleck and Reiss-Roth. From the quote above you will note that Reiss-Roth consider these sorts of studies the strongest kind of study on the association between guns and homicide. Kleck, on the other hand, reckons that these studies tell us nothing at all about the link. Whatever the truth on this matter is (and I believe that R and R are a lot closer to it than K), it is the height of arrogance for Kleck to argue that anyone who disagrees with him is dishonest or incompetent.

The third cited study only weakly supported the authors’ claim (cite 15, Cook 1979). This study addressed only robbery homicides, which accounted for only 10% of U.S. homicides the year Kellermann et al. wrote (U.S. Federal Bureau of Investigation 1994, p. 21), and apparently confused an effect of robbery murders on rates of gun ownership with an effect of gun levels on robbery murder rates.

While Kleck is entitled to interpret the results of Cook’s study in a different way from Cook, he is not entitled to impugn the honesty of those who decide that Cook’s interpretation of Cook’s study is more plausible than Kleck’s.

Finally, the authors also cited a crude two-city study of their own (Sloan, Kellermann, Reay, et al. 1988) that simply ignored the causal order issue, used indirect measures of gun ownership that turned out to be inaccurate,

I must have missed it when Kleck demonstrated that gun ownership in Seattle was not higher than gun ownership in Vancouver.

and drew conclusions that ignored pronounced differences between the cities that were responsible for some or all of the observed differences in homicide.

So let me get this straight. If study Y clearly comes to conclusion X, and researcher Z says “there is evidence for X” and cites study Y, but Kleck feels that there are problems with study Y, then Z is guilty of falsely citing Y?? What incredible arrogance by Kleck!

Arthur Kellermann and Donald Reay [Kellermann AL. and Reay DT. “Protection or Peril? An Analysis of Firearms-Related Deaths in the Home.” N Engl J. Med 1986. 314: 1557-60.] cited similarly nonexistent published statistics to support their assertion that “less than 2 percent of homicides nationally are considered legally justifiable” (p. 1559, citing sources 11 and 13). Neither FBI publication offered any statistics on this matter, never mind a figure “less than 2 percent.”

Nonexistent statistics? Anyone can download a copy of the 1995 UCR and find these “nonexistent published statistics” on page 31. And yes, they do comprise less than 2% of homicides nationally. Now, Kellermann’s cite was for the 1983 UCR, so I suppose it is possible that the FBI has only recently started publishing the figures for justifiable homicides in the UCR. Even if this is the case, Kellermann’s error is minor – he should have cited the FBI’s SHR instead of their UCR. Big deal.

Further, although the FBI had unpublished data pertaining to some subtypes of “legally justifiable” homicides, these covered only a small subset of lawful defensive homicides (PB:112).

Which is exactly what Kellermann said in the very next sentence. It is disingenous for Kleck to take a quotation of Kellerman’s out of context to make it appear that Kellermann was asserting that only 2% of of homicides were lawful defensive homicides.