John Lott’s unethical conduct

John Lott is embroiled in several controversial affairs:

  • he almost certainly fabricated a mysterious survey and certainly behaved unethically in making claims for which he had no supporting data
  • he presented results purporting to show that “more guns” led to “less crime” when those results were the product of coding errors
  • he pretended to be a woman called “Mary Rosh” on the internet in order to praise his own research and accuse his critics of fraud.
  • he probably was the person who anonymously accused Steve Levitt of being “rabidly antigun”

The Mysterious Survey

John Lott has claimed that he conducted a survey over three months in 1997 that found that in 98% of defensive gun uses the defender merely has to brandish the gun to break off an attack. Strong doubts have arisen as to whether he ever did a survey because:

  • nine published surveys give numbers ranging from 21% to 67% as to how often defenders shoot, far more than the 2% Lott claims
  • Lott claims that his survey found defenders firing in 2 out of 28 cases, which is 7%, not 2%. Nor is it possible that the weighting scheme Lott now claims to have used turned 7% into 2%.
  • no evidence that Lott ever conducted such a survey can be found
  • Lott has repeatedly changed his story about the source of the 98% figure, variously attributing it to “national surveys” and some particular polls, only publishing the story about the survey in 2000.
  • Lott made the 98% claim on Feb 6, 1997, well before his survey was completed.
  • Lott changed his story about the survey.
  • Despite extensive coverage on the net, in many papers such as the New York Times, the Chicago Tribune and the Washington Post, an ad in the alumni magazine and in the best-selling book Freakonomics, none of the eight students Lott claims conducted the survey have been heard from.
  • Lott claims to have “replicated” his survey with a new one that gives 95% brandishing, when in fact the new survey found only seven gun users (one of whom fired), far too small a sample to yield a meaningful brandishing number. (And if you correct his arithmetic you get 91% anyway.)
  • Lott was caught lying when he claimed “I have not participated in the firearms discussion group nor in the apparent online newsgroup discussions”.

Best writing by bloggers on this is by Marie Gryphon, Kieran Healy and Mark Kleiman. Best articles by journalists are those by Tim Noah and Michelle Malkin. An annotated copy of Lott’s response to Malkin is here. More details about the problems with Lott’s claims are here. Also recommended are Otis Dudley Duncan’s comments on scientific ethics and on Lott and surveys.

All of my posts on the mysterious survey.

Coding errors in More Guns, Less Crime data

In 1998, John Lott published a book entitled More Guns, Less Crime. In that book he presented statistical evidence that concealed-carry laws were associated with lower crime rates. My critique of his book is here.

In 2002, Ian Ayres and John Donohue analysed a more extensive data set and found that, if anything, concealed carry laws lead to more crime. Lott that using even more data confirmed the “more guns, less crime” hypothesis. Ayres and Donohue’s response (April 2003) was devastating—Lott’s data contained numerous coding errors that, when corrected, eliminated the results and, this was the second time these sorts of errors had been found in Lott’s data. My posting here has more on the coding errors and Mark Kleiman has a nice summary.

For months Lott did not respond on this important matter, neither admitting nor denying the errors. Instead he would change the subject when the matter is raised, often alleging unfair treatment by the Stanford Law Review. Finally on Aug 20 he admitted that there were a few hundred errors, but claimed that even after correcting them there were still significant reductions in crime. After correcting his coding errors (which eliminated his results), Lott had changed his model in order to bring his results back. When I asked him to explain why he changed his model, he did not offer any explanation, instead making a clumsy attempt at covering up what he had done. Chris Mooney has an excellent article describing how Lott changed his model. Lott’s response to Mooney is covered here.

All of my posts on the coding errors.

John Lott’s Fabricated Identities

Over a couple of years John Lott made hundreds of postings on the Internet, using the false identity “Mary Rosh”. He used the Mary Rosh identity to

Lott lied when he claimed

“I have not participated in the firearms discussion group nor in the apparent online newsgroup discussions”.

while continuing to use the “Mary Rosh” identity.

Lott was unmasked by Julian Sanchez, who also wrote an excellent summary of the affair. Also of interest is this Washington Post article on the fabrication.

On Jan 22 he confessed to using the Mary Rosh sock puppet and wrote “I shouldn’t have used it”. The very next day he was back posting again, using another pseudonym, “Washingtonian“.

Using a mixture of pseudonymous and anonymous postings Lott posted numerous reviewsto Amazon.com, giving his own books fifteen five-star reviews, and panning rival books.

There is a collection of some of Rosh’s postings, and also a page containing an extended discussion “Rosh” had with Ed Huntress.

All of my posts on John Lott’s sock puppets.

John Lott attacks Steve Levitt

Lott is widely believed to have been the anonymous person who accused Steve Levitt of being “rabidly antigun” in this NRO article. In The Bias Against Guns Lott uses the NRO article to support his claim that Levitt is biased. Mark Kleiman has written a nice summary of the affair.

All of my posts on Lott’s attack on Levitt.

Lott’s bias

John Lott’s favourite example of the “Bias Against Guns” is the story of the shootings at the Appalachian School of Law. Lott performed a superficial analysis of the news stories about the shootings and found that very few of the stories mentioned the fact that two of the students involved in apprehending the killer were armed. Lott concluded that reporters deliberately left out this fact because they were biased, but a more careful analysis finds that the first stories published did not mention the guns because the reporters did not know about them, while the later stories were about different aspects of that matter.

All of my posts on the shootings.