Deltoid

In his interview in the Illinois Leader Lott says:

“If you look at the national news reports for ABC, NBC and CBS during 2001, they had about 190,000 words of contemporaneous gun crimes stories on their television morning and evening news reports. The average story is only 250 words or so. During that whole year there was not one single mention of people using their guns defensively, to either protect themselves or someone else.”

An alert reader points out that Lott seems to have missed this story:

Take, for example, the question of how often guns are used for self-defense.

Gun control advocates say firearms are used 108,000 times a year for self-defense.

Gun control opponents say the figure is as high as 2.5 million times a year.

and this:

“By far the safest course of action is to have a gun,” argues John Lott, author of More Guns, Less Crime.

He says residents of high-crime areas should have guns as a deterrent, and that those guns should not be locked away but readily available for use as a defense.

And this:

John R. Lott Jr.
Bad things can happen with guns, but guns can also make it easier for people to defend themselves and prevent bad things from happening. Potential victims use guns more than 2 million times a year to stop violent crimes. Crimes are stopped by defensive gun uses about five times as frequently as crimes are committed with guns.

And this:

Both troopers, who were suspended without pay, pleaded innocent, saying the shooting was self-defense. Hogan and Kenna said they fired at the van thinking the driver was trying to run them over.

Special Prosecutor James J. Gerrow admitted in court last week part of the state?s theory was wrong and then asked Smithson to dismiss the attempted murder charge against Hogan.

In chapter 3 of More Guns, Less Crime Lott presents an analysis based on two exit polls of gun ownership (conducted in 1988 and 1996) that purports to show that a 1% increase in a state’s gun ownership causes a 4.1% decrease in the violent crime rate and a 3.2% decrease in auto theft.

Lott’s two polls indicate that gun ownership increased by a remarkable 50% in just eight years, from 26% to 39%. However, this is contradicted by all other surveys on gun ownership. The best of these are the GSS surveys which actually show a modest decline over that period.

Even Lott found a 50% increase so unlikely that in this Usenet exchange he called it a “strawman”:

Doug Weil:

Two surveys ask questions in very different ways. The organization that collected the data says — the surveys are not comparable. You compare the data anyway, and produce a result so far afield from anything any regularly administered survey has produced (specifically a 50% increase in gun ownership in the general population over an 8 year period all other surveys show no increase in gun ownership) and claim that, well — you must be right because you controlled for differences in the survey questions.

John Lott:

The survey implies at 35% change in gun ownership. Why do I have a feeling that you are trying to exaggerate the poll data so that you can can set up a strawman?

In chapter 9 of the 2nd edition of More Guns, Less Crime Lott defends his use of these exit polls.

Now, in his analysis of safe storage laws in chapter 7 of The Bias Against Guns Lott does not use these exit polls. Instead, despite his suspicion that the GSS surveys are cooked, he uses the GSS surveys to measure how gun ownership changes as a result of a state passing a safe storage law. Using these polls he finds that gun ownership declined by one percentage point per year in the states with the laws and argue that the laws caused increases in crime rates.

Lott does not explain why, after stoutly defending his use of the exit polls to measure changes in gun ownership at the state level he abandoned them for his later paper. One possible explanation is that the exit polls say the opposite thing to the GSS surveys. The exit polls show substantial increases in gun ownership in the states that passed safe storage laws. I computed a regression relating the change in gun ownership as measured by the exit polls to the number of years that a safe storage law had been in place and found that the laws were associated with a 0.06 percentage point per year increase in gun ownership rates. This increase is not statistically significant, but it is the opposite sign to Lott’s result using the GSS surveys.

Since the exit polls show increases in gun ownership while the GSS surveys show decreases, it is plausible that if Lott had conducted his analysis in chapter 3 of More Guns, Less Crime using the GSS surveys he would have found that more guns were associated with more crime.

Lott has posted a transcript of the AEI event to publicize The Bias Against Guns. I’ll try to correct some of the false statements in the transcript:

In 2001, according to government survey evidence, there were about 450,000 crimes that were committed with guns. Of those, there were about 8,000 gun murders. Yet our best estimates indicate that last year Americans also used guns defensively, a little bit over 2 million times a year. Ninety-five percent or so of the time, simply brandishing a gun was sufficient to stop an attack.

Surveys that ask about both offensive and defensive uses consistently find more offensive uses than defensive uses. Lott’s comparison uses numbers from different surveys that cannot possibly both be true. Lott’s brandishing number involves ignoring significantly lower numbers from several larger professionally conducted surveys in favour of an estimate from just seven defensive gun users in his own amateurish survey. And he didn’t even calculate the number correctly.

Basically you have about 190,000 words during 2001 on gun crime stories, and zero words being spent on any of those news broadcasts about people using guns to stop a crime or protect themselves or someone else.

Not true, as pointed out earlier. The coverage certainly is lop-sided but is is silly to infer a bias against guns from it, any more than you could infer a bias against aeroplanes because plane crashes get extensive coverage while all the occasions when lives have been saved by air rescues get much less coverage.

at the Appalachian Law School in Virginia, in which several people were killed. There were two students at the school who had law enforcement backgrounds, that when the attack started, they ran to their cars, got their pistols, came back, pointed their guns at the attacker, ordered him to drop his gun; when he did so, they tackled him and held him until police arrived.

Now, if you go and do a Nexis search, which is a computerized news search of news stories around the country, you find in the one week after the attack well over 200 separate stories about the incident. However, only four mention the students having a gun in any way, and only two of those four mention the students actually used their gun to stop the attack. The typical coverage was in the Washington Post, which said, “Students pounced on the gunman and held him until help arrived”; and New York Newsday, that said, “The attacker was restrained by the students.” There are others which say, “Students tackled the man while he was still armed.”

…

Tracy Bridges, one of these students, told me that he had talked to reporters at over 50 different news organizations

Lott is well aware, since he was involved in an extensive discussion on this matter, that the students didn’t just have a law enforcement background but were current police officers. That he is counting the same wire service story printed in many papers over and over again. That stories printed on the first day of coverage didn’t mention the gun because the first witnesses that were interviewed didn’t even know about it. That most stories printed on the next day were covering the aftermath of the shooting rather than the shooting itself. That Tracy Bridges did not talk to reporters at over 50 different news organizations.

Furthermore, several of the stories made it quite clear that the shooter was out of ammunition, but Lott has not mentioned this in any of his op-eds, talks or his book.

And given the recent troubles with the New York Times, one of the stories that I go through in the book involves a major series of over 20,000 words that the New York Times had on so-called rampage killings. These were killings in a public place, where two or more people were killed. And the Times had claimed that over the last 50 years there were a hundred of these cases; 51 of them had occurred within just the five years from 1995 to 1999. And the New York Times had said we’re having this massive explosion of these attacks and that it’s imperative that we go and adopt new and stricter gun control laws in order to try to deal with it, even though many of these cases didn’t involve guns being used in the attack.

You know, there’s a sidebar where the Times briefly mentions that the series “does not include every attack.” But the omissions are so extremely skewed here that they produce a ninefold increase in attacks between the 1949 and 1994 periods versus the ‘95 to ‘99 periods. And I don’t know anybody who has looked at this data who would claim that there was this huge increase that seemed to occur right in 1995.

As I detailed earlier, and as is obvious if you read the New York Times article, the Times did not say that there was a “huge increase” or a “massive explosion” in 1995. That’s Lott’s straw man.

When these [safe storage] laws get passed, you’ll see about a 5 to 6 percentage point drop in gun ownership that occurs right when they get passed. And over time, you’ll see a huge increase in the rate at which people store their guns locked and unloaded. It goes from about in the low 30 percent range to almost 70 percent within five years after these states pass these laws.

David Hemenway’s review points out that Lott’s claims here are based on a misuse of GSS data, which is not designed for state level analysis as Lott uses it here. He is also cherry picking which gun ownership survey to use.

I just–I mean, I–since you deal with studying the media, I could actually get some advice from you on that type of thing, because–you know, there’s a study that’s come out in the Stanford Law Review, that Carl was referring to, that they’d given to the L.A. Times kind of as an exclusive to write up something on. And I remember talking to the reporter, and she said, well, you know, you say this and they say this, how am I supposed to evaluate what’s right? And my response to some extent is, look, there may be some things that are more difficult for you to evaluate, you know, like what’s the right statistic to use here–your statistical test. But there are lots of things that should be very easy for you to evaluate.

So for example, this paper was generally criticizing the work that I’d done in the past, and the first criticism that they brought up is that Lott never mentions the cost of guns. You know, he only mentions the benefits. And then they go through and they say if Lott were, you know, a reasonable researcher he would mention a cost–for example, they have a 1992 case in Louisiana, where a Japanese exchange student was shot when he walked into the wrong back door of a house. And my point–and I went through some other ones, I maybe went through, like, four or five of them. And I said, you know, those are pretty easy things to check. You know, is it true that Lott’s work never mentions the cost of guns? And that’s simply not true. I mean, from the very first sentence in the books–I would go and point to them–that the first sentence would say a gun can prevent things from happening, it can also make it easier for bad things to happen. Or that, you know, this Japanese exchange student thing was on page 2 of “More Guns, Less Crime”–that very case that they were saying if Lott was reasonable, he would mention something like that.

The only reason why Ayres and Donohue’s paper contains that error is that Lott insisted on its restoration to the paper. In an email the editor wrote:

(2) p119 Hattori story —> Your insistence seems really pointless, because making them restore their original phrase serves no purpose other than giving you the chance to say that they’re wrong. I will ask Donohue to restore this one, but please take an objective look at your paper, and I think you will realize that none of your points are affected or diminished by this change.

Lott does not mention that Ayres and Donohue acknowledge the error and correct it in their reply. Nor does he mention that this is the only error that he was able to find in a 120 page paper. Nor does he mention that what they were trying to point out was that Lott and Mustard had erroneously claimed that the Hattori shooting was not “unlawful”. Nor has Lott acknowledged or corrected this error, even though they had pointed it out to him before the Lott and Mustard paper was published.

If reporters are looking for some non-technical thing they can check they could compare Lott’s claims with what the New York Times article actually said.

Also in the transcript we have Carl Moody saying:

The second cut is, as you say, is the data available to other researchers [inaudible], and the answer is no for Kellermann, so I think he’s lying. He’s refused repeated requests for his data. So, no, I don’t trust him and I don’t think anybody here in this room should trust him, or anywhere else, for that matter.

Kellermann’s data has been available from the ICPSR (study 6898) for six years now. You would have hoped that Moody would have taken at least a tiny bit of care with his facts before accusing Kellermann of lying. (And why hasn’t Moody accused Lott of lying about the 1997 survey—Lott has not released the data from that survey.) Oh, and Lott at least, is well aware that Kellermann has released his data but he did not correct Moody’s false statement.

Lott has responded to parts of my post yesterday.

1) “Why do you use the government’s survey estimate for the number of crimes committed with guns but use other surveys in your two books for estimates on the number of defensive gun uses?”

The problem with the survey from the Bureau of Justice Statistics is that “virtually none of the victims who use guns defensively tell interviewers about it in the [National Crime Victimization Survey]” (Kleck, Targeting Guns, p. 2250. People aren’t allowed to say whether they have used a gun defensively unless they indicate that they have had a crime committed against them. As many people who used a gun defensively may have prevented a crime, and thus have avoided an incident serious enough to be included in the set of events asked by the interviewer, many successful defensive gun uses would never be recorded.

The NCVS asks questions like “Did anyone TRY to rob you?” Certainly it would not count a case where the defender used a gun to scare off a suspicious looking character before he made a move, but that isn’t a defensive use, it’s assault with a deadly weapon.

2) “Is it accurate to describe the students who stopped the attack as having law enforcement backgrounds?”

The students were attending law school in Virginia and were on leave from their deputy sheriff jobs in another state, North Carolina.

As Lott is well aware from the discussion he was involved in, Bridges and Gross were police officers. Lott’s description of them as merely having a law enforcement background implies that they were former police officers.

3) Does the New York Times actually claim that the “rampage” killings were increasing during the 1990s?

The first article (April 9, 2000) in the series contains a massive table that shows very clearly that most of the attacks over the last 50 years supposedly occurred within the last five years.

“The series of articles published in The Times this week based on that research offered several new insights. Although such killings account for only one-tenth of 1 percent of all homicides, the series confirmed the public perception that they appear to be increasing.” Editorial, “A Closer Look at Rampage Killings,” New York Times, April 13, 2000, p. A30.

In fact, the Times says

the database does not include every attack of this type over the last 50 years.

The reference from the editorial is to this passage, about a different dataset:

Yet there is a strong impression that they have become more common. In an effort to confirm the trend, The Times analyzed F.B.I. reports of all homicides since 1976. Each year there were 15,000 to 22,000 homicides, but very few involved three or more victims.

That universe shrank even more, to just a few dozen, when The Times weeded out those involving robbery or gang violence, and those in which the primary victim was a family member.

What is left is the closest thing there is to a census of rampage killings — about one-tenth of one percent of all killings.

And it shows that in the 1990’s, they increased.

Their number remained fairly consistent from 1976 to 1989, averaging about 23 a year, only once going above 30. But between 1990 and 1997, the last year for which data was available, the number averaged over 34, dipping below 30 only once, in 1994.

So, yes, they do say that such killings increased, from 23 a year to 34 a year. They don’t say that they increased nine-fold as Lott claimed. And they base their claim on FBI data of all such killings, not on a database of just 100 cases.

4) “Why did you use the CBS and Voter News Service surveys for gun ownership in your book More Guns, Less Crime but use the General Social Survey in your book The Bias Against Guns?”

The CBS and Voter News Service surveys have the advantage in that they are very large surveys (e.g., the VNS survey interviewed over 30,000 people). That makes it possible to get a fairly accurate measure of gun ownership rates in individual states. The problem with these surveys is that they cover only two years, 1988 and 1996. By contrast, the GSS has a very small sample in any given year, but the survey covers most states every other year. Which survey you use depends upon the questions that you want to ask. In the current book, The Bias Against Guns, states have adopted safe storage gun laws over many different years from 1989 to 1998 during the period that I studied. The question that I wanted to apply the survey data to was how gun ownership rates changed in the different states that adopted these laws in different years (see pp. 177 to 179 in the book). For the general question addressed in More Guns, Less Crime on whether the places with the biggest relative increases in gun ownership had the biggest relative drops in violent crime (pp. 113-14), I wanted to use the largest surveys available to get as accurate a measure as possible of the differences in gun ownership rates across states. However, in the chapter on gun storage laws in The Bias Against Guns, it was desirable to see how gun ownership rates changed immediately before and after the adoption of the law and the only way that I could do that was with the GSS data.

The GSS sample size is much too small to give a meaningful estimate of gun ownership rates in a given state. About the only way it could be used for this would be to pool many years of data, but then you get no indication of “how gun ownership rates changed immediately before and after the adoption of the law”.

Lott’s discussion of the CBS and VNS surveys is also misleading. While the CBS survey had a very large sample, the VNS survey, with a sample of 3,818 isn’t much larger than the GSS, and when you are considering the accuracy of differences between the two surveys it is the sample size of the smaller one that matters most.

Neither of Lott’s reasons seems like a good one, and he he still hasn’t addressed the fact that the surveys give opposite results—the GSS shows gun ownership declining, while the CBS+VNS polls show gun ownership increasing.

The Minneapolis Star Tribune has a story about David Gross, who, after all this time, is the only witness to Lott’s 1997 survey who has ever been found:

A major player and legal consultant on Minnesota’s new gun-permit law is a former board member of the National Rifle Association who was fired from the Minneapolis city attorney’s office for opposing gun buy-back programs and carrying a gun to work.

He also acknowledges shooting a deer in his back yard in St. Louis Park with a .357-caliber Magnum handgun for eating his raspberries, pointing a rifle at a neighbor many years ago who he claimed was harassing his wife, and attending his synagogue armed with a handgun in case of trouble.

David Gross, the self-described “right-hand man” to Hamline law Prof. Joe Olson, worked with Sen. Pat Pariseau, R-Farmington, and Rep. Lynda Boudreau, R-Faribault, to create the bill, which makes it easier for Minnesotans to obtain a permit to carry a gun in public places.

Gross said he put in “countless” hours over the past four years advising on the bill, including writing the section that says Minnesota “recognizes and declares that the second amendment of the U.S. Constitution guarantees the fundamental, individual right to bear and keep arms.”

and

For Gross, now in private law practice in a St. Louis Park, passage of the law has been somewhat of a crusade. He insists it has little to do with guns and everything to do with the Constitution. “Does my right to defend myself end at my front door? I don’t think so.”

The fight has cost Gross financially and professionally. He estimates that his battle has cost him $1 million in lost salary and benefits. He also lost the stature he had in the city attorney’s office.

All this gave Gross a very powerful motive for bailing Lott out. Back in January, with his efforts to get a carry law in Minnesota close to fruition, it would have been been disastrous for Gross if Lott, whose work and testimony had been greatly helpful to the pro-carry-law lobby, were to have been disgraced over the survey.

It seems way too much of a coincidence, not just that someone with this much of a motive should happen to have been surveyed, but that the only person to have come forward should be such a person.

And, as if all this wasn’t enough, way back in January I wrote:

Julian Sanchez posts some comments from someone [David Gross] who believes he was surveyed by Lott. Lott is in error when he states that there were no other gun use surveys at that time, but once these have been eliminated, we can regard it as established that Lott conducted a survey in 1997.

Well, those other surveys haven’t all been eliminated. It remains possible that Gross was surveyed in Hemenway’s 1996 defensive gun use survey. Update: Hemenway informs me that

He doesn’t seem to have been. No one from his state, of his gender and general age responded Yes to the self-defense gun use question, and no one in the whole survey told a story highly similar to his story…

I think that by now we can be more than 98% certain that Lott never conducted a survey in 1997.

In The Latest Misfires in Support of the More Guns, Less Crime Hypothesis Ayres and Donohue write:

In the wake of some of the criticisms that we have leveled against the Lott and Mustard thesis, John Lott appeared before a National Academy of Sciences panel examining the plausibility of the more guns, less crime thesis and presented them with a series of figures showing year-by-year estimates that appeared to show sharp and immediate declines in crime with adoption of concealed-carry laws. David Mustard even included these graphs in his initial comment on the Donohue paper in the Brookings book that PW refer to repeatedly in their current response. But Donohue privately showed Mustard as well as the Brookings editors that the graphs were the product of coding errors in creating the year-by-year dummies, and in the end Mustard conceded and withdrew them from his comment on Donohue.

Here is the graph and associated text that Mustard withdrew from his section in Evaluating Gun Policy: (more…)

Lott has a new entry on his blog. First, he approvingly links to an NRO opinion piece by John Derbyshire, who writes about the case of Tony Martin, who was convicted of murdering a 16-year old burglar. Derbyshire feels that Martin’s imprisonment is “preposterous”. Glenn Reynolds, in a rather overwrought column goes further, declaring Martin to be a “political prisoner” and wants Amnesty International to weigh in. Unfortunately, Reynolds and Derbyshire have uncritically accepted the Martin’s defense lawyers version of the events, apparently without checking to see if it was accurate. If you look at an account that presents the prosecution’s side of the story as well, you will find that Martin was accused of lying in wait for the burglars and shooting them without warning. The jury, who presumably were more familiar with the facts in the case than Derbyshire decided that the prosecution had proved their case.

Lott goes on to claim that “the British 1997 Act … literally made it a crime to use a gun defensively.” He offers no support for this remarkable claim and the fact that Martin was not charged for a crime of using a gun defensively rather undercuts Lott’s claim.

Next Lott reprints a passage from The Bias Against Guns where he asserts that there were dramatic crime increases in Britain and Australia following tighter gun laws. Mary Rosh made similar claims about Britain, which I corrected here. And Ken Parish already demolished Lott’s claims about crime in Australia. Lott also falsely claims that the new laws in Australia made “it a crime to use a gun defensively.”

Lott’s comments about Australia that I discussed yesterday follow a similar pattern to those of many American pro-gunners. First, they greatly exaggerate the restrictions introduced in 1996, claiming that Australia “banned guns” or, in Lott’s case claiming that Australia banned “most guns and [made] it a crime to use a gun defensively.” In fact, semi-automatic long guns were banned and there was no change in the law on self-defence. Next, the pro-gunners will assert or imply that Australians were made defenceless. In fact, the new laws made very little difference there. Finally, they present some cherry-picked crime statistics in an attempt to show that crime went up because of the new laws. The trouble here is that there are lots of different categories and crime goes up and down, so you can usually find one category where crime has increased.

One example of this sort of thing was debunked on the Urban Legends Reference Pages. Another example was an NRA video that was criticized by many people.

Lott’s most dramatic statistic is:

In Sydney, handgun crime rose by an incredible 440 percent from 1995 to 2001.

The source Lott gives for his claim is this article in Sydney’s Sunday Telegraph. They seem to have originally got it from this report., which found that the number of handgun shootings in NSW increased from 9 in 1995 to 42 in 2000. The increase seems much more dramatic when expressed as a percentage, and such small numbers tend to fluctuate greatly. Naturally Lott does not report the dramatic decline that followed in 2002.

Lott has a new posting where he responds to a letter from John Donohue to the Columbus Dispatch replying to a Lott op-ed. I earlier posted a link to the op-ed and a letter from Michael Maltz replying to it.

I’ll post more on Lott’s comments later, but for now I want to point to the most important thing in his posting:

see also the data and updated results available at www.johnlott.org.

“Updated results?” If you go to his site you will find corrected versions of the graphs and tables that Ayres and Donohue said were incorrect because of coding errors. The most interesting of these corrected tables is Table 3a (registration required), because Ayres and Donohue also give a corrected version. After ducking the question of coding errors for weeks and weeks it would seem that by issuing corrected graphs and figures Lott is at last sort of conceding that he made the errors. Of course, he also needs to correct the text that refers to the incorrect tables. For example, commenting on Table 3a:

Table 3a provides the exact results and significance levels behind these specifications, and reports the robust standard errors that adjust for clustering at the state level. The spline and the hybrid models indicate positive, but statistically insignificant, trends in violent crime rates prior to the right-to-carry law. After the law has been passed, however, violent crime rates decline. The change in trends is statistically significant at least at the ten-percent level for all individual violent crime categories for the spline estimates, implying that murder, rape, and robbery fall by over two percent per year during each additional year that right-to-carry laws are in effect.

When will Lott produce a corrected version?

Tom Spencer thinks that the latest information on David Gross might be the straw that breaks the camel’s back.

This is one of the graphs that Lott presented to the National Academy of Sciences Panel in 2002. David Mustard’s originally included it in his contribution to Evaluating Gun Policy, but it was removed after Donohue showed him that it was the product of coding errors made by Lott. Later graphs produced by Lott look quite different—as we saw yesterday, this seems to be all the acknowledgment you get from Lott when he makes an error.

Notice how the graph shows crime rates falling sharply and immediately after carry laws were adopted. These results were much more dramatic than any that Lott has produced before. Even if the coding errors were accidental, Lott should never have presented this graph to the panel. It should have been obvious to him that the results were too good to be true, but because they seemed to support his thesis, he accepted them. Even if not dishonest, this is inexcusable negligence and casts doubts over all of his other research results.

Lott has a new posting where he has some more about the important matter of the coding errors in his data. Sandwiched between some more complaints about unfair the Stanford Law Review has been and some imaginary errors in Ayres and Donohue, we have:

Of course, this is nothing new with their misleading attacks on David Mustard, where minor coding errors did not change what he had written. (Instead of letting David correct a small mistake which did not fundamentally change the results, David was forced to cut out what would have been a damaging evidence against Donohue. If correcting these minor points had changed the results in a way favorable to Donohue, why wouldn’t they let David publish the figure that he wanted to publish?)

I presented the figure that was cut out yesterday. Here is the relevant part of the associated text that was cut, along with figure 1 (which wasn’t cut). (The complete passage is here.)

Furthermore, extending the data through 1998 produces results consistent with the basic Lott-Mustard conclusions. Figure 2 depicts the coefficient estimates from state-level regressions of the years before and after the Right-to-Carry laws go into effect for the four violent crimes. … The passage of the law is associated with sharp decreases in murder, rape and robbery. Murder and robbery rates are higher in Shall-Issue states prior to the passage of the law and fall immediately after the law goes into effect. As in Figure 1, these drops are much larger than would be warranted by a reversion to the mean explanation because the post-law rates are substantially lower than the pre-law rates. Rape shows a slightly different pattern. Right-to-Carry states have similar rape rates as other states in the years prior to the law. However, these states experience sharp drops in rape rates after the law is implemented. Aggravated assault rates in counties that pass Shall-Issue laws are higher both before and after the law goes into effect, with little change before and after the law.

Now, Figure 1 uses one less year of data and groups years into pairs, but basically it should look similar to Figure 2. However, because of Lott’s coding errors it looks dramatically different. How on earth can Lott call this “a small mistake which did not fundamentally change the results”.

Lott refers to “minor coding errors did not change what he had written”, but Mustard wrote

The passage of the law is associated with sharp decreases in murder, rape and robbery.

That describes figure 2 (the one affected by the coding errors) but not figure 1, which doesn’t have any big decreases till eight years after the law (and even that decrease was because the set of states with eight years of post-law data was smaller than the set of states with six years of such data).

And can anyone discern what, in that deleted text, was the “damaging evidence against Donohue” that Lott claimed was cut. Anyone?

In summary, Lott’s claim that his coding errors in his NAS panel analysis were “minor” or “small” and do not “fundamentally change the results” is completely false.

I asked Ben Horwich, the president of volume 55 of the Stanford Law Review to comment on Lott’s latest complaints. He writes:

• I did not categorically promise Lott that we’d run a verbatim statement by Plassmann and Whitley. I did express my interest in working with them to clear up the confusion. I think that’s the crux of the misunderstanding.
• The statement that did run was prepared in consultation with Plassmann and Whitley; indeed, they provided the original draft. Of course, it’s modified a great deal from that version, but I wanted to print something that I stood behind 100%.
• Since the whole purpose of the statement was to clear up confusion and vindicate Plassmann and Whitley’s motives in the whole thing, it didn’t even make much sense to have it come from them alone. So whatever we ran, for it to be effective, it had to come from the Editors; and if it was to come from the Editors, then it had to be something we could sign on to. Hence it’s not the exact piece they’d have run if they could, but it seems like small change.

If you compare the Plassmann and Whitley statement with the one from the Stanford Law Review, there do not seem to be any really important changes—about the only thing is the Stanford Law Review editors do not accept Plassmann and Whitley’s charge the editors “violated an agreement”. I think the editors were quite right to reject the charge. The minor editorial correction that they made is hardly a substantive change.

It is ironic that Lott piously complains that

I don’t think that it looks very good for two senior academics to lash out at young people like this, especially when the attacks on them are unjustified.

while he repeatedly accuses the law students who edit the Stanford Law Review of breaking promises and lying (when they deny they were pressured by Donohue).

He also writes:

Florenz and Whitley are being painted as being so desperate for a publication that they would put their names on a flawed paper.

However, Plassmann and Whitley have put their names on a flawed paper. As Ayres and Donohue showed, the paper contains table after table and graph after graph based on miscoded data. And, instead of correcting the problem, or even admitting that it exists, they have issued a statement about the relatively trivial matter of Lott’s dispute with the Stanford Law Review,

In his 6/9/03 posting, Lott claims that Donohue has made a “large number of easily identifiable mistakes”. Even if true, such mistakes pale into insignificance compared with the coding errors that Lott made but will not admit to, but let’s examine Lott’s claims and see how many mistakes he has successfully identified:

he implies that David Olson’s paper was so flawed that Olson and Maltz had to withdraw the paper.

Lott has correctly identified a slip up by Donohue, since the paper has not been withdrawn. I checked with Donohue and he informs me that what he should have said was

one author of the other paper (as Lott well knows) has repudiated the paper after concluding that the data on which it (and Lott’s work) was based is unreliable.

Maltz’s own letter on the subject is here.

Lott continues:

claiming that David Mustard and I “never acknowledge” the costs of guns or the possible bad effects of concealed handgun laws,

Actually, Ayres and Donohue wrote:

they never acknowledge cases on the other side of the ledger where the presence of guns almost certainly led to killings.[12]

This is a reference to the Hattori case, which is a mistake that they acknowledged and corrected in their reply. This mistake is only in their paper because Lott insisted that they not be allowed to correct it. Lott has tried to score points off this mistake three times now—in the reply to their paper, in his AEI talk and on his blog.

Lott continues:

juxtaposing quotes to make it possibly appear that I was arguing about law-abiding citizens carrying guns on planes when the op-ed was about pilots carrying guns (p. 1199)

Actually, the op-ed was also about having off-duty police on planes carry guns and responsible citizens off planes carry guns. I did not get the sense from their quotes that Lott was arguing for law-abiding citizens to carry guns on planes.

that previous work did not deal with the possible impact that cocaine could explain the changes in crime rates attributed to concealed handgun laws,

In fact, they argue that Lott does not adequately control for the impact of crack cocaine, not that he does not try to control for it.

the measurement error problems in county level data

Ayres and Donohue deal with this charge quite thoroughly in their reply:

48. Following Michael Maltz, we have been concerned about relying on any analysis that uses county crime data, particularly if the data extends across the period before and after 1993 (when the reporting agency substantially changed its data collection method). See id. at 1260. But PW claim that we have misread Maltz, and that Maltz did not assert that reliance on the county dataset was unwise. Both claims are false. Indeed, we showed our statement and the PW response to Michael Maltz, and he rejected the PW allegation. Maltz said that, if anything, our paper actually understated the Maltz and Targonski criticism of the county-level data: They view the county data to be severely flawed overall, “especially if” (not “only if”) one extends the data across the break in the series that occurred in 1994. Email from Michael Maltz to John Donohue, supra note 13. In response to PW’s claim that the measurement problems are no worse in the county data than in the state data, Maltz replies: “[B]oth state- and county-level data are affected, but state-level data are affected much less profoundly.” Id.

PW also contend that Maltz and Targonski have no discussion of a post-1992 break in the quality of data. Plassmann & Whitley, supra note 13, at 1363. Maltz again disagrees:

We noted:

The 1994 NACJD codebook (ICPSR dataset 6669) explicitly notes this in a major heading, “Break in Series,” and describes the new imputation procedure it began using in 1994. It goes on to state,

These changes will result in a break in series from previous UCR county-level files. Consequently data from earlier year files should not be compared to data from 1994 and subsequent years because changes in procedures used to adjust for incomplete reporting at the ORI or jurisdiction level may be expected to have an impact on aggregates for counties in which some ORIs have not reported for all 12 months.

Email from Michael Maltz to John Donohue, supra note 13. “In other words,” Maltz continues, “Lott refuses to acknowledge that his entire county-level analysis in the second edition of his book is faulty.” Id. In response to the PW statement—”[n]or do Maltz and Targonski provide any evidence that state-level data are more dependable than county-level data,” Plassmann & Whitley, supra note 13, at 1363&mdash’Maltz replies: “We do so in our response to his response to our paper.” Email from Michael Maltz to John Donohue, supra note 13. At the least it must be conceded that there is no truth to the PW claim that we misinterpreted Maltz’s views.

Lott continues:

even Philadelphia’s concealed handgun laws are incorrectly described

In fact, in Confirming “More Guns, Less Crime”, even Lott doesn’t dispute that Philadelphia changed to “shall-issue” some years after the rest of the state. He just argues that while it was still “may-issue”, they became more liberal about granting permits.

So, to summarize: Lott has only succesfully identified two mistakes, one by Donohue in a letter where he wrote “withdrawn” instead of “repudiated”, and one in Ayres and Donohue’s paper that was only present because Lott insisted on its inclusion. Both of these mistakes have been acknowledged and corrected.

Lott’s track record on mistakes is not so good. I could go through a very long list, but I want to focus on the most important ones ones—the coding errors in the data for Confirming “More Guns, Less Crime”. Will Lott acknowledge and correct these errors?

Lott has an interview on strike-the-root.com. He repeats some of the false claims discussed here earlier, such as his claim of a 440% increase in handgun crime in Sydney. He also claims:

Ninety-five percent or so of the time, simply brandishing a gun was sufficient to stop an attack.

It is appalling that rather than admit to being wrong, Lott continues to spread false information about what works when defending yourself with a gun. This

could actually endanger people’s lives by giving them false impressions about what works and what doesn’t work.
John Lott, on radio Free Republic, May 8, 2003 (At 1:08:40 of audio).

Lott’s 6/13/03 entry on his blog links to a letter from David Mayer printed in the Columbus Dispatch replying to a letter from Donohue. Mayer asserts:

The recent letter by Stanford law professor John Donohue (June 7) nicely illustrates the propensity of gun-control advocates to play games with statistics and to engage in ad hominem attacks. In this case, Professor Donohue unfairly attacks economist John Lott, whose research has helped dispell the myths about guns that anti-gun fanatics continue to propogate.

Apparently Mayer is unaware what an ad hominem argument is. Attacking Lott because he impersonated a 115 pound woman on the net would be an ad hominem argument, but Donohue does not do this. Nor are Donohue’s attacks unfair. Lott’s data contains coding errors and Lott continues to refuse to admit this.

Mayer continues:

Professor Donohue’s own study, which purports to show an increase in crime after concealed-carry laws are enacted, is itself “fatally flawed,” to use his own terminology. John Lott’s new book, The Bias Against Guns, on pages 235-39 discusses the problems with the way Donohue and his co-author, Ian Ayres, have manipulated the statistics. As Lott shows, even Donohue’s and Ayres’ own results show that violent crime rates fall after right-to-carry laws are adopted.

As I pointed out earlier, Lott has just cherry-picked one result frm their paper and pretended that that is Ayres and Donohue’s entire argument. It seems that Mayer has not even read their paper. And Lott’s graphs on pages 235-39 are incorrect, since they are based on the miscoded data.

As well as posting Mayer’s letter, Lott asserts:

One point that I would add is that while Ayres and Donohue have now written several pieces between them on right-to-carry laws, none of their papers have appeared in refereed journals.

Now, Lott is well aware of

Ian Ayres and John J. Donohue III
Nondiscretionary Concealed Weapons Laws: A Case Study of Statistics, Standards of Proof and Public Policy,
1 American Law and Economics Review 436 (1999)

From the web site of the American Law and Economics Review:

The Review is a refereed journal, published twice a year.

Earlier I observed that Lott had claimed that a paper by Cummings et al that found a significant decline in juvenile accidental gun deaths following the introduction of safe storage laws was

widely discredited because the researchers never factored in that accidental gun deaths have been falling everywhere for decades.

When I pointed out that their paper clearly stated that they had controlled for national trends by using fixed effects, Lott responded with:

We had been unable to replicate their claimed results using fixed effects and the only way we could get something similar was without fixed effects. It really shouldn’t have been that difficult for us to confirm what they found since we were used their dates for the laws. Unfortunately, Cummings, Grossman, Rivara, and Koepsell were unwilling to give us their data when we asked for it. I asked for the data from Cummings and one other coauthor. Possibly we should have made a big deal of yet more academics who refused to share their data, but we decided that the more straightforward approach would be to simply say what we found. Alternatively, we could have simply stated that we were unable to confirm their results.

Firstly, I note that Eugene Volokh had no difficulty obtaining Cummings’ raw data. More importantly, in an article Reexamining the association between child access prevention gun laws and unintentional shooting deaths of children by Webster and Starnes I find that:

1. Webster and Starnes gathered their own data:

   “Data on the annual number of unintentional gunshot deaths among children 0 to 14 years old and on population demographics were obtained for each state and the District of Columbia from the Compressed Mortality Files of the National Center for Health Statistics for the years 1979 through 1997

2. Webster and Starnes included fixed year effects:

   “Year dummy variables were also used to control for temporal variation observed across states, irrespective of their CAP law status, attributable to unmeasured factors (eg, prevalence of gun ownership).

3. Webster and Starnes replicated Cummings’ results:

   “Using the same data and methods used by Cummings et al, we first replicated their findings.

Lott’s original claim that Cummings did not use fixed year effects is false. His new claim that Cummings’ results could not be replicated is also false.

The most likely explanation of all this is that in his book Lott made a mistake about Cummings. Perhaps he carelessly confused their paper with some other paper that didn’t use fixed effects. When I pointed out his error, rather than admit to making a mistake, Lott made up the story about only being able to replicate Cummings without fixed effects.

Lott has an update to his 6/13/03 post where he responds to this post. He writes:

An e-mailer asks about whether the Ayres and Donohue piece in the American Law and Economics Review was refereed. While the original papers in that journal are indeed refereed, their piece was a review article and my understanding from Ayres was that it was not refereed.

It seems that Lott is unable to admit to even the smallest error. He claimed that they had not published it in a refereed journal, when in fact they had. Instead of admitting to the error he pretends that his original claim was different.

More importantly, this very minor point has been the only response by Lott to my last two weeks of postings where I have raised some very important issues. When will Lott say something about the coding errors?

The centrepiece of Lott’s The Bias Against Guns is the story he tells about the shootings at the Appalachian Law School. According to Lott, after killing three people Peter Odighizuwa was almost out of ammunition and was on his way to his car to get more when he was confronted by two armed students, Tracy Bridges and Mikael Gross. When Bridges aimed his gun at Odighizuwa Odighizuwa dropped his gun and was tackled by students. Lott opines that Bridges and Gross “undoubtedly saved many lives”. Lott says that the biased media mostly suppressed this story, with only four stories out of 208 in the week afterwards mentioning that the students were armed. He also tells us that he spoke to reporter after reporter who knew about Bridges and Gross’ actions but did not report them.

Lott’s claims were picked up by others, with commentators such as Larry Elder, James Eaves-Johnson and Donny Ferguson all seeing it as proof of media bias. In a Houston Law Review article Eric Luna thought it demonstrates

the willingness of certain individuals or groups to skirt the truth or disregard all other considerations when issues of guns and gun control are at stake.

Unfortunately, Lott’s counting methodology is flawed, his count missed half of the stories that mentioned the armed students, his version of what happened deliberately omits important facts and omits contradictory accounts from other eye witnesses and his version contains details that appear to have been invented by Lott.

First, let’s check Lott’s work in counting news stories.

I did my own search on Factiva of all the stories appearing in the week following the shootings. I found eight stories (twice as many as Lott found) that mentioned the students having a gun. Next, in his 208 stories, Lott has counted the same stories over and over again. Many papers printed the Associated Press story by Chris Kahn and Lott has counted it each time it appeared. He has also counted stories about completely different aspects of the matter—there is no reason for a story about one of the funerals to have mentioned a defender’s gun.

To address this problem I did my own count by reporter (so all stories by a particular reporter were only counted once), and only counted stories that mentioned how Odighizuwa had been apprehended.

The result? 28 different reporters wrote about the shootings and 8 mentioned the defender’s gun. There was also a striking pattern—stories on the 17th of January (the first day of coverage) tended not to mention a defender’s gun, whereas later stories did. In fact, four of the reporters who did mention a gun later also wrote stories on the 17th of January which did not mention the gun. Clearly those reporters did not leave out the gun in their first story because of a bias against guns—they simply did not know about it yet.

If you examine stories written on the 18th of January or later, there are ten stories, six of them mentioning a gun. That leaves only four stories that might have been expected to mention a defender’s gun. Let’s look at each of them:

1. Mike Oduniyi All Africa News
   This story seems to have been written for Africa News because Odighizuwa was Nigerian. Oduniyi looks to have written his story from the news stories of 17 Jan and doesn’t seem to have talked to anyone himself. No evidence that Oduniyi knew about a defender’s gun.
2. Alfonso A. Castillo. Newsday
   This story was about the murdered dean. Castillo just mentions in passing the police report of how Odighizuwa was apprehended. No evidence that Castillo knew about a defender’s gun.
3. Paul Dellinger Roanoke Times & World News
   This story was about the memorial service. Dellinger talked to Mikael Gross about one of the murder victims and identified Gross as one of the students who tackled Odighizuwa. No evidence that Dellinger knew about a defender’s gun.
4. Maria Glod The Washington Post
   Talked to Gross about the shootings. Did not mention that he was armed.

So there’s your bias. One reporter who didn’t mention the defender’s gun when she probably should have. Maybe she is biased, but you can’t conclude that from just one example.

Now let’s look at the facts Lott deliberately left out of his account of what happened: Rick Montgomery in an article in the Kansas City Star wrote:

The Star recently interviewed two students involved—Bridges and Besen. They gave differing accounts.

Bridges repeated that he pointed his weapon at Odighizuwa and ordered the suspect to put his own down, which he did.

According to Besen, the first student to tackle the suspect, nothing of the sort happened. He said Odighizuwa set down the gun and raised his arms—”like he was mocking everyone: ha, ha, what are you going to do now?”—before the students confronted him.

The two armed students had not yet arrived at the scene, Besen said: “Peter had no knowledge anyone had a gun.”

Virginia State Police confirmed Odighizuwa’s weapon was empty by then.

Police spokesman Stater said the armed students did assist after Besen and another student, Todd Ross, tackled the gunman. Bridges sat on the suspect while Gross, also armed, provided handcuffs he had gotten from his car.

But to Stater’s view, the biggest heroes were Besen and Ross—the unarmed men who lunged at Odighizuwa.

Alas, they weren’t the focus of attention when a writer and photographer for an NRA magazine came to the campus to interview the armed students.

Bridges said they took his picture; NRA spokesman Gregory said, “It was nobody from our staff.”

It’s all gotten way too political for Besen.

“I’m a gun advocate, but it really irritates me that people are trying to use this as a (political) plug,” he said. “The NRA is minimizing the tragedy that happened here. I don’t appreciate it.”

Lott is well aware of these facts because he selectively quotes from this very article. He knew that Besen contradicted Bridges account but did not mention this fact at all

And not only does he deliberately suppress inconvenient facts, he invents new ones to make his argument more compelling. Even though the Kansas City Star and many other stories report that Odighizuwa’s gun was empty, Lott specifically claims that Odighizuwa was not out of ammunition. That lets him claim that armed citizens “undoubtedly” saved many lives.

Lott also prints a quote from Kent Markus, taken from a Legal Times article:

“The gun lobby, without much sensitivity or attention, has distorted   what actually happened for their own political benefits, I think it   is a shameful exploitation of a tragedy.”

Lott writes:

  “However, when I called up Markus to find out exactly which facts he   was referring to, he was unable to provide me with any details.”

I contacted Markus to find out if Lott had accurately reported their conversation. Like everyone else who I have contacted about conversations reported by Lott, Markus informed me that Lott had greatly distorted what was said. Markus actually told Lott exactly what facts he was referring to—that witnesses had told him that Odighizuwa was disarmed before the armed students returned. He also refused to tell Lott the names of the witnesses because he felt that they had suffered enough unwanted attention and he did not want Lott badgering them. Presumably this refusal was translated into Lott-speak as “unable to provide me with any details”.

So again we see that Lott was well aware that Bridges story about using his gun to disarm Odighizuwa was in dispute, but he did not mention this.

This also explains why the stories published on 17th January did not mention the armed students—none of the other witnesses were aware of their guns, either because their guns played no role in the story, or all of these witnesses somehow did not notice them.

This isn’t the first time that Lott has omitted important details of this story to help his argument. In his original op-ed on the shootings Lott left out the fact that the students were police officers.

The tale Lott spins: “Armed citizens save many lives; media suppresses this fact” seems to resonate with pro-gunners and many seem to accept it, turning off their powers of scepticism. Trouble is, his tale isn’t true.

My thanks to Tom Maguire for sending me the link to the Kansas City Star article and prompting this investigation.

Tom Spencer believes that I have essentially destroyed one of Lott’s core arguments and wonders why pro-gun people continue to support him.

There are two contradictary stories about what happened at the Appalachian Law School:

• Besen said that Odighizuwa set his gun and a clip on a light fixture about four feet off the ground before Bridges arrived.
• Bridges said that he aimed his gun at Odighizuwa and then Odighizuwa “throwed his weapon down”.

Note that they contradict each other about what Odighizuwa did with his gun. If Besen is correct and Bridges arrived after Odighizuwa put his gun down, then Bridges may not have seen what Odighizuwa did with the gun.

I asked Rick Montgomery (the Kansas City Star reporter) what the police report said about where Odighizuwa’s gun was found and he told me that it agreed with Besen’s account. This suggests that Bridges arrived after Odighizuwa had put his gun down and Bridges imagined or invented the details of what Odighizuwa did with the gun.

I also note that Besen doesn’t seem to have any motive to lie here, while Bridges’ story makes his role in the affair much more important.

So the most plausible theory is that the Bridges’ gun did not have a significant role in the affair, not that media bias suppressed its role.

Tom Maguire has an interesting post which collects some links to blogspace discussion about the Appalachian Law School shootings.

One interesting thing is that Lott and Kopel independently made the same error—they both claimed that the New York Times did not mention the defender’s gun when it did. Both errors were particularly egregious. Kopel quoted a sentence from the article but did not notice that the gun was mentioned in the very next sentence. Lott counted the New York Times story as one of the four that mentioned the gun, but also claimed that it did not mention the gun. I think this is evidence of bias on their part. They were so sure that the New York Times wouldn’t mention the gun that they didn’t notice that it had. (To be fair, both of them have corrected this particular error.)

Maguire doesn’t think things look good for Lott and wonders

whether some Lott supporters or gun enthusiasts have attempted to rally a convincing counter-argument. Presuming, of course, that there are Lott supporters (probable), and plausible counter-arguments on this incident (doubtful).

Mark Kleiman has taken the anti-Lott side. How do we rally the pro-Lotters? Are there any?

Pro-Lott bloggers seem to have given up defending him. William Sjostrom did say he was going to write something, but nothing has been forthcoming.

Brian Linse observes that while Eugene Volokh has criticized a study that found a link between gun ownership and homicide, he hasn’t said anything about Lott. Linse writes:

Bitching about “bogus” gun studies would impress me more if these same folks didn’t support Lott in a thesis that promotes gun tragedies involving children and other innocents. It’s shameful, and it erodes the credibility of otherwise thoughtful gun rights activists.

The study Volokh is criticizing found that people who lived in a house where a gun was kept were more likely to end up as homicide victims. This suggests that keeping a gun in the home causes you to be more likely to get killed. Volokh argues that there is an alternative explanation for the link—criminals are more likely to be murdered and more likely to own guns.

Now, this is a legitimate criticism, but Volokh overstates his case, claiming that the study is “badly flawed” and

says nothing about whether gun ownership really “increases the odds” that a law-abiding citizen will be killed

Any correlational result like this could be caused by some other factor, and it is never possible to control for everything. Results from such studies are never certain, but it is not true that they “tell us nothing”.

It is possible that the link seen came from gun ownership increasing the chance of being killed, or because of criminals being more likely to own guns and be murdered or some combination.

To completely dismiss this result because of the “criminal background” confounding factor it would be necessary to show that criminal’s are more likely to be murdered (Volokh has some good evidence here), and that they are more likely to own guns (Volokh has no evidence here) and that these factors were sufficient to explain the correlation.

The study provides some evidence that having a gun in your house make you more to be murdered, but the confounding factor that Volokh has identified weakens that evidence.

John Quiggin comments on this Gun Control Australia press release attacking John Whitley (see also Eugene Volokh’s comments). Ditto on Quiggin’s Voltaire/JS Mill allusions, but I think everybody is paying too much attention to this. I believe that they may have made their attack on academic freedom just to generate some publicity and controversy. You can read Whitley’s response to them here—note that he ignores that part of the press release.

Quiggin also writes:

Sadly for Whitley, I think anyone associated with John Lott is going to have a hard time getting ARC money or anything else that involves research cred. Lott’s disgraceful antics have been particularly damaging to his co-authors, including Whitley

Yes, Lott’s name isn’t even on the Stanford Law Review paper any more, so Plassmann and Whitley have been left with the responsibility for the coding errors in that paper. And despite months going by they have not addressed this problem. It is sad, and it is bad for Whitley’s career.

Cypren criticizes Fox News for presenting this Lott op-ed as if it were a news story.

Lott has an editorial arguing that the US army would be better off if it didn’t disarm Iraqi civilians. Kathy Kinsley agrees. What is notable about his piece is what he doesn’t cite. In arguing that American soldiers would be better off if more Iraqi civilians had guns he doesn’t cite his own research on concealed carry. This is quite possibly because while he won’t admit to making the coding errors, he knows that when they are corrected his results go away.

Instead he cites Mustard’s paper that found a correlation between concealed carry laws and reduced police deaths. Unfortunately for Lott, the model Mustard used is similar to Lott’s and suffers from the the same flaw of lumping all the states together. Most likely if Mustard’s model is disaggregated or extended to include the latest crime figures the results will go away, just as they did with Lott.

Lott also claims:

The states that polls show as having the biggest increases in gun ownership are also the ones that have experienced the biggest relative drops in violent crime.

This is not correct—his analysis used data from polls that were totally unsuited for his purpose and contradicted by all other polls including those he used in subsequent research. More details are here.

Terry Krepel writes about biased reporting from CNSNews.com. Krepel observes that their coverage of Lott exhibits bias by omission with two stories about Lott carefully avoiding mentioning the mysterious survey or Mary Rosh.

After I linked to a posting by Cypren that attacked Fox News, Lott read the posting, construed it as an attack on himself and complained about it. Cypren’s post seemed to me to quite clearly be an attack on Fox rather than Lott, but I guess Lott’s is so used to being attacked that he saw it as an attack on himself.