Deltoid

[On Sep 14 2002 I posted this to firearmsregprof. I also emailed it to John Lott. ]

Way back in 1993 in talk.politics.guns, C. D. Tavares wrote:

The answer is that the gun never needs to be fired in 98% of the instances of a successful self-defense with a gun. The criminals just leave abruptly, instead.”

When I queried him about this, he quickly corrected his error:

Kleck says in the magazine “Social Problems” (2/88):

“there were about 8,700-16,600 non-fatal, legally permissible woundings of criminals by gun armed civilians” annually, and “the rest of the one million estimated defensive gun uses, over 98% involved neither killings nor woundings but rather warning shots fired or guns pointed or referred to.”

So I certainly screwed up by not including the warning shots fired, plus the times the victim simply missed. From Kleck’s latest survey as reported here, the warning shots and misses are very significant (~14%).

Now, Lott has repeatedly made the same claim: “98% of the time when people use a gun defensively, merely brandishing the weapon is sufficient to stop an attack.”

A Google search for “98% lott brandishing gun” finds about 120 different sites where his claim is repeated (and only one of those is critical). Lott himself has made the claim on at least 25 different occasions. Otis Dudley Duncan has compiled a list of these that I extracted the examples below from.

Lott’s claims about the source of this statistic have changed over time:

“Polls of American citizens undertaken by organizations like the Los Angeles Times and Gallup showing that Americans defend themselves with guns between 764,000 and 3.6 million times each year, with the vast majority of cases simply involving people brandishing a gun to prevent attack.[1]”
John Lott “Does Allowing Law-Abiding Citizens to Carry Concealed Handguns Save Lives?” Valparaiso University Law Review, 31(2): 355-63, Spring, 1997.
[1]. Gary Kleck & Marc Gertz, Armed Resistance to Crime: The Prevalence and Nature of Self-Defense with a Gun, 86 J. CRIM. L. & CRIMINOLOGY 150, 153, 180, 180-82 (1995).

“Polls by the Los Angeles Times, Gallup and Peter Hart Associates show that there are at least 760,000, and possibly as many as 3.6 million, defensive uses of guns per year. In 98 percent of the cases, such polls show, people simply brandish the weapon to stop an attack.”
John Lott “Gun-Lock Proposal Bound to Misfire” Chicago Tribune, August 6, 1998.

“If national surveys are correct, 98 percent of the time that people use guns defensively, they merely have to brandish a weapon to break off an attack.”
John Lott “More Guns, Less Crime” (University of Chicago Press, 1998), p. 3. [Date of publication, May, 1998.]

“People used guns defensively to stop violent crimes over 2 million times in 1997. 98 percent of the time, when people use guns defensively, simply brandishing a gun is sufficient to cause a criminal to break off an attack. In less than 2 percent of the time is the gun fired. About three-quarters of those are warning shots.”
Oral statement, paraphrased, TV show, Hardball, CNBC, August 18, 1999

“Guns clearly deter criminals, with Americans using guns defensively over 2 million times each year — five times more frequently than the 430,000 times guns were used to commit crimes in 1997, according to research by Florida State University criminologist Gary Kleck. Kleck’s study of defensive gun uses found that ninety-eight percent of the time simply brandishing the weapon is sufficient to stop an attack.”
John Lott “Gun Locks: Bound to Misfire” online publication of the Independence Institute, Feb. 9, 2000.

“If a national survey that I conducted is correct, 98 percent of the time that people use guns defensively, they merely have to brandish a weapon to break off an attack.”
More Guns, Less Crime, second edition (University of Chicago Press, 2000), p. 3. [Publication date, May, 2000]

“As I told Duncan last year in a telephone conversation, I had no idea why the estimated 2.5 million defensive gun uses was attributed to me. The 2.5 million estimate obviously comes from Kleck. The about 2 million reference is the average of the 15 national surveys and is very similar to my own estimate of a little over 2 million defensive uses. The survey that I oversaw interviewed 2,424 people from across the United States. It was done in large part to see for myself whether the estimates put together by other researchers (such as Gary Kleck) were accurate. The estimates that I obtained implied about 2.1 million defensive gun uses, a number somewhat lower than Kleck’s. However I also found a significantly higher percentage of them (98 percent) involved simply brandishing a gun. My survey was conducted over 3 months during 1997. I had planned on including a discussion of it in my book, but did not do so because an unfortunate computer crash lost my hard disk right before the final draft of the book had to be turned in.”
John R. Lott, Jr.’s Reply to Otis Duncan’s recent article in The Criminologist, The Criminologist, vol. 25, no. 5, September/October 2000, page 6. http://www.cse.unsw.edu.au/~lambert/guns/lottduncan.html

“In fact, in 98% of the cases, simply brandishing a gun is sufficient to stop a crime. Research at Florida State University and at the University of Chicago indicates that only one out of 1,000 defensive gun uses results in the attacker’s death.”
LA Times Friday, March 30, 2001 Others Fear Being Placed at the Mercy of Criminals by John Lott Jr.

The reason why Lott’s story has changed is that neither the Kleck survey nor any of the polls that he cited found that only 2% of the time the weapon was fired. Kleck says, quite clearly, “24% of the incidents involved the defender firing their gun”. The other polls Lott mentioned give even higher numbers.

Lott is well aware of this, since he changed the wording in the second edition of “More Guns, Less Crime”. However, instead of changing it so that his readers were informed of what national surveys actually found, he claimed that it came from his own survey.

Further, a 2.1 million estimate implies that about 1% of his respondents reported a personal DGU in the past year. With a sample size of 2,424, that’s about 25 DGUs. 2% of 25 is 0.5. Clearly it is not possible for half a person to report firing their weapon. OK, so maybe Lott used a design similar to Kleck’s where the question asked about household use over a five year span. If he got a similar number of positive responses to Kleck, then he would have got 100 DGU reports and is is conceivable that only two reported shooting. However, he also claims that 3/4 were warning shots—3/4 of 2 is 1.5. Again, it is not possible for half a person to report a non-warning shot.

Otis Dudley Duncan has tried to find out more details about this survey. I quote from an email from him:

I queried him [Lott] by mail and e mail (I sent the mail certified but he claims not to have received the e mail, which was identical) and got nothing more from him besides what is in the passage in his “Reply” except for the detail that the interviewing was done by students using their own computers. They fed their results into his computer, and it was this one that experienced the crash. He assures me that others lost data in that crash too. Strangely enough, in the phone conversation of May 1999, presumably long after the crash, he said nothing about it. And he has left the timing rather ambiguous. If he did a survey in early 1997 but too late to mention in his Valparaiso Law Review article, he would still have got the 98% figure by the time his book went to press. Indeed, he says that he had planned to include a discussion of the survey in the book, but the crash occurred just before his deadline for turning in the book. (This could refer to the 2000 edition, but that would raise other problems.) So, it is not at all clear why he couldn’t have said the same thing on p. 3 in 1998 as he did in 2000. The timing strongly suggests that the figure of 98% came first, the story about the survey and its loss in the computer crash came afterward. A second letter to Lott this summer went unanswered. In it I had asked for specific evidence such as a copy of his questionnaire, a copy of the computer printout from which he derived his 98% etc., names of some of the students, etc. He has not answered that letter. I too would like to have an explanation from Lott. I considered the possibility that he indeed thought of doing some kind of survey, and maybe went so far as to have some students do trial interviews, but gave it up for whatever reason. And possibly something of that material went into the computer that subsequently crashed.

No one knows but Lott.

A plausible explanation of what has happened is that Lott made a similar error to CDT, but rather than correct it like CDT did, Lott chose to make up a story about a survey.

I appeal to the readers of this list—if you can think of a better explanation or if you have some knowledge of this mysterious Lott survey, please let me know.

“The reluctance of gun-control advocates to release their data is quite widespread. In May 1997 I tried to obtain data from the Police Foundation about a study that they had recently released by Philip Cook and Jens Ludwig, but after many telephone calls I was told by Earl Hamilton on May 27. “Well, lots of other researchers like Arthur Kellermann do not release their data.” I responded by saying that was true, but it was not something that other researchers approved of, nor did it give people much confidence in his results” John Lott, page 291 of “More Guns, Less Crime”

[On Sep 15 2002 I posted this to firearmsregprof.]

In response to my appeal for any information about this survey Lott claims to have carried out in 1997 I received this email from Geoffrey Huck. Unfortunately, as you can see, he was not able to provide any support for the existence of the survey, so I repeat my appeal: it seems increasingly likely that the survey is fictional—I seek evidence to the contrary.

From: Geoffrey Huck

I was John Lott’s editor at the University of Chicago Press for his book, More Guns, Less Crime, published originally in 1998 and then in a second edition in 2000. John has asked me to confirm for the record an incident that occurred in the summer of 1997, just as he was preparing the final version of the manuscript for submission to the Press. At that time, John reported to me that a bookshelf had fallen on his computer, seriously damaging his hard disk containing not only all his files and data for More Guns, but also work on some other projects as well. I recall that much of what was on the disk was lost and could not be recovered. We did have hard copy of most (but I think not all) of the book manuscript, however, and were able to proceed with that.

[On Sep 17 2002 I posted this to firearmsregprof.]

I’ve had replies to my queries about Lott’s survey from David Mustard and Gary Kleck.

Mustard said that he was not involved with the survey. All he was able to say was what he had been told by Lott: that Lott had conducted the survey in 1997 and lost the results in a computer crash.

Kleck felt that Lott’s survey should be ignored unless and until Lott publishes the results and methodology. This would be reasonable except for the fact of the 25+ occasions that Lott has made the “98% merely brandishing” claim.

Nothing from Lott.

[On Sep 18 2002 I posted this to firearmsregprof and emailed it to Lott.]

Some more information about DGU surveying from a Kleck email:

“We got 4,977 completions over 3 months, using an average of about 10 callers per day (and I believe all of them were FSU students)”

Also, look at the acknowledgments of Kleck and Gertz’s paper:

“The authors wish to thank David Bordua, Gary Mauser, Seymour Sudman, and James Wright for their help in designing the survey instrument. The authors also wish to thank the highly skilled staff responsible for the interviewing: Michael Trapp (Supervisor), David Antonacci, James Belcher, Robert Bunting, Melissa Cross, Sandy Hawker, Dana R. Jones, Harvey Langford, Jr., Susannah R. Maher, Nia Mastin-Walker, Brian Murray, Miranda Ross, Dale Sellers, Esty Zervigon, and for sampling work, Sandy Grguric.”

If there had been a huge fire or something and all of the computer and paper records of Kleck’s survey had been destroyed, we would still have twenty witnesses who could confirm that the survey had been conducted.

Lott says his survey interviewed 2,424 people, over three months, so he would have needed 5 callers operating a day. That would have required a pool of at least eight interviewers, plus a supervisor. I looked through Lott’s vita and I couldn’t see any paper where it looked like he had conducted a phone survey, so he would have had to consult with an expert on the design of phone surveys. And….

Put yourself in Lott’s shoes. It’s the beginning of 1997. You’re busy writing your book and talking to reporters and whatnot about your 1996 paper. You decide to do a survey in order “to see for myself whether the estimates put together by other researchers (such as Gary Kleck) were accurate”. Who are you going to call to discuss the survey design?

Yeah, that’s right. Gary Kleck.

But Lott didn’t. We know this because both Duncan and I have asked Kleck about it.

[I have sent a copy of this message to Lott, just as I have the others on this topic.]

Still nothing from Lott.

[On Sep 20 2002 I posted this to firearmsregprof and emailed it to Lott.]

James Lindgren writes:

After my post to this list saying that "a big national study doesn’t just disappear without a trace" because a computer crashes, John Lott called me and told me a long story about how the study was done (which I don’t choose to share just yet, if ever; Lott can speak for himself on his methods, if he wishes).  He didn’t ask me to do anything about it, and I wasn’t planning on posting anytime soon, but given the recent posts I thought I would.

Whether the study should be given credence is a different question from whether it was ever done.    The latter question is the important one for me. 

While it will probably be possible to get a lot of circumstantial evidence about Lott’s data losses on various projects in 1997 (I got an email from David Mustard to this effect and will check with Bill Landes myself), the crucial question on which direct evidence is needed is whether study was done.  Lott reports that he used UC undergrad volunteers to do the calling, using a sample drawn from a CD ROM with phone numbers.

Volunteers? Based on Kleck’s surveys you would need 300 person days of calling. I can imagine a student doing a day or three as a volunteer, but more than that? How could you get enough volunteers?

When his computer crashed, he decided not to publish the study and says that he made only passing reference to it in his book.

In the 1st edition of MGLC Lott writes: “If national surveys are correct, 98 percent of the time that people use guns defensively, they merely have to brandish a weapon to break off an attack.” He doesn’t mention his survey anywhere in the 1st edition. In the 2nd edition of MGLC (published in 2000) Lott changed it to: “If a national survey that I conducted is correct, 98 percent of the time that people use guns defensively, they merely have to brandish a weapon to break off an attack.” As far as I know, this is his first published mention of the survey.

This would not be a big deal, except that Lott has made his 98% claim on literally dozens of occasions. His claim promotes the “magic talisman” theory of gun self-defence — suggesting that all you need to do is wave a gun around a little. This theory is dangerous — it could potentionally lead to people being injured or killed.

Lott does not remember the names of the undergrads.  I offered to contact the UC alumni office to see if I could email the undergrads who were juniors and seniors in the 1996-97 academic year to locate the callers.  I intend to do just that in the next 3-4 weeks, if UC will go along.

Thank you so much for your assistance.

However, I find it disturbing that after discussing with you the means by which he could demonstrate that this survey was actually conducted, the best Lott could come up with was to attempt to contact students whose names he cannot remember. As has been discussed here before, there should be plenty more evidence — phone bills, survey questions, people he discussed the survey with whose names he recalls and so on.

[On Sep 25 2002 I posted this to firearmsregprof. I also emailed it to John Lott. ]

I have replies to queries about Lott’s survey from Tom Smith and Al Alschuler. Neither had heard anything about it. Other than Kleck, Tom Smith would be the logical person to discuss the survey design with—Lott acknowledges his assistance on other survey-related questions in MGLC. He’d also be a good source to obtain a CD of phone numbers from, but apparently Lott obtained this elsewhere.

Alschuler is in the U of Chicago Law School and wrote a reply to the paper where Lott first made the brandishing claim. He told me that the paper was due shortly after the conference it was presented to. That conference was held in mid November 1996. Consequently, this:

“Polls of American citizens undertaken by organizations like the Los Angeles Times and Gallup showing that Americans defend themselves with guns between 764,000 and 3.6 million times each year, with the vast majority of cases simply involving people brandishing a gun to prevent attack.[1]”
John Lott “Does Allowing Law-Abiding Citizens to Carry Concealed Handguns Save Lives?” Valparaiso University Law Review, 31(2): 355-63, Spring, 1997.

was written before Lott even started doing his survey. So how did he know that the “vast majority of cases” involved brandishing before he did this survey of his?

Now, in the quote above, he doesn’t give the exact number, whereas in later version like this one:

“98% of the time when people use guns defensively simply brandishing a gun [is] sufficient to cause a criminal break off an attack. In less than 2% of the time is the gun fired. Most of those are warning shots. Less then one half of one percent of the time is the gun fired at the attacker. And only one out of every thousand times that people use guns defensively does it result in the death of the attacker. In the vast majority of these times simply brandishing a gun is sufficient to make a criminal break off an attack and run away.”
John Lott on NPR’s “Justice Talking” June 28, 1999 Audio available from http://www.justicetalking.org/getshow.asp?showid=97 Quote is at time 55:25 in the audio

he does, so it is possible that he only knew the number approximately in 1996 and his survey gave him a more precise estimate, but the question remains: “Where did the 1996 “vast majority” estimate come from?”

I would be most interested in any suggestions that readers have.

[On Sep 27 2002 I posted this to firearmsregprof and emailed it to Lott.]

Peter Boucher, replying to this post, writes:

I don’t have a copy of Point Blank handy, but I seem to recall the 98% figure either explicitly in the text of that book, or directly derivable from the figures in the book.

Yes, as I noted at the beginning of the discussion, the 98% figure could have come from a misreading of “Point Blank”. However, the problem with this theory is that Lott has specifically denied it. He says that the figure comes from his survey. It also seems far too much of a coincidence for him to believe (mistakenly) in 1996 that the figure was 98% and then conduct a survey in 1997 that comes up with exactly the same number.

I found a few example of others who had reached the same conclusion by doing a google:

http://www.gunowners.org/fs9504.htm “Of the 2.4 million times citizens use their guns to defend themselves every year, 92% merely brandish their gun or fire a warning shot to scare off their attackers. Less than 8% of the time, a citizen will kill or wound his/her attacker. (5) … (5) Kleck, interview with Schulman, op. cit.”

Arrgh! another misreading of Kleck. The 92% includes shots fired at the attacker. Also this assumes that defensive gun use is 100% effective — just because the defender brandishes a gun, it doesn’t mean that the attacker automatically flees.

Anyway, the “vast majority” claim seems to be supported by Kleck-Gertz from 1995.

OK, maybe Lott felt that the 76% of DG users in Kleck-Gertz that said they didn’t fire their weapon was big enough that he considered it a “vast majority”. Now here’s the problem:

In 1996, based on Kleck’s survey, Lott believes the figure is 76%. In 1997 he conducts his own, smaller, survey which comes up with 98%. Quite a remarkable difference. Later in 1997 he summarizes the situation in MGLC. I can think of several ways you might describe this: “Surveys give widely varying estimates for the percentage of defenders that fire their weapons, ranging from 2% to 24%.” Or, maybe if there is some reason, so far not revealed by Lott, that he thinks his survey supercedes Kleck’s he could could write this: “A survey conducted by Gary Kleck found that 24% of defenders shoot, but I survey I conducted shows that the figure is 2% and I believe that figure is more accurate because <insert super-secret reason>”.

What Lott in fact wrote was “If national surveys are correct, 98 percent of the time that people use guns defensively, they merely have to brandish a weapon to break off an attack.” (MGLC p3) and “Kleck’s study of defensive gun uses found that ninety-eight percent of the time simply brandishing the weapon is sufficient to stop an attack.” (Gun Locks: Bound to Misfire). And on at least three dozen other occasions he presented the 98% figure without ever mentioning that there might be any doubt over its accuracy.

Nowhere did he even mention the existence of this survey of his until 2000 in the 2nd edition of MGLC. And even then he wrote “If a national survey that I conducted is correct, 98 percent of the time that people use guns defensively, they merely have to brandish a weapon to break off an attack.” Still no mention of the wildly divergent Kleck estimate.

[On Sep 27 2002 I posted this to firearmsregprof.]

Norman Heath writes:

Just a suggestion, but perhaps Lott simply made a rough comparison between the number of claimed DGUs and the total number of shootings. I.e. if total shootings is (making this up) 120,000 and we subtract 35,000 suicides, 5000 police shootings, 200 hunting accidents, 15,000 gun murders, then even without accounting for non-fatal criminal shootings the highest possible number of live-fire DGUs would be about 64,800 (again, using made-up numbers). If at least 764,00 people claimed DGUs, then the number of those who actually shot somebody in self defense would necessarily be far under 10% of total claimed DGUs.

This is basically how Kleck came by his original estimate that 98% of DGU did not involve anyone getting shot. To get the numbers he would need for this calculation he would have to go to Kleck’s work and would have come up with the same 98% figure for DGUs where no-one was wounded. However, it just seems too much of a coincidence that his own survey would just happen to produce exactly the same percentage for a different number (percentage who don’t fire).

It is not my mission in life to stick up for John Lott, about whom I know next to nothing. But this hopeful little fishing expedition trying to catch J.L. in a lie has a couple of problems, not the least of which is that the inquisitors are trying to prove a negative, the non-existence of a survey. This is very unlike checking an author’s claim against a cited document which does exist and can be examined. Even if no phone bills from the survey can be found, no undergrad students recall participating, etc., the case against the existence of a ‘98 survey can probably never be considered proven. The inquiry has little prospect of ever being conclusive, thus it appears to be more an exercise in insinuation than anything else.

On the other hand, given the amount of evidence that a national survey of the scale that Lott claims he conducted would leave, it should be trivially easy for him to prove that the survey was conducted. But he hasn’t done so.

Of course, anybody is free to try to dig up some dirt against an academic who represents the “other side.” But a little discretion is appropriate, in order that the inquiry itself does not become campaign of slander. It is one thing to use a listserve to ask casually where one might obtain Prof. X’s dissertation. It is another matter entirely to suggest on a listserve that Prof. X committed plagiarism in his dissertation and then ask where one might obtain a copy of the dissertation in order to find the plagiarism.

Unfortunately that approach does not work for the questions that we would like answered. For example: “What, specifically, were the questions in Lott’s survey?” If I asked the list that question, I think the reasonable answer would be that I should ask Lott. Duncan has asked Lott that question and Lott has not answered it. I could not see any way of making the enquiries that I felt were needed without there being some sort of implication that something untoward was going on, so I felt that I should explain my suspicions to the list.

And, raising the matter in a somewhat public forum has prompted Lott to respond to some extent, though I don’t understand why he would not tell the list the story of his survey.