Deltoid

(more…)

After discussion has simmered along via email, Usenet and mailing list, Marie Gryphon has posted a nice summary on her blog.

Several blogs have picked up on this: Julian Sanchez, Jim Henley, Jane Galt, Kevin Drum and Thomas M. Spencer.

Clayton Cramer has posted a letter from John Lott on his blog. Some highlights:

“The overwhelming majority of the survey work was done at the beginning of the period over which the survey was done. It has obviously been a while, but my recollection is that the small number of people surveyed after the first four or five weeks (mainly January 1997) did not include any more defensive gun uses.”

In an email to Lindgren Lott wrote “I am willing to bet that I don’t start mentioning this [98%] figure until the spring of 1997. If I use it before I said that I did the survey, I will say that they nailed me.” Lott used the figure on Feb 6, 1997 and got caught. Does he say “they nailed me”, like he promised? No, he changes his story yet again. Instead of three months, the survey is now supposed to have taken one month. This is the fourth version (1. “national surveys”, 2. “Kleck”, 3. “three month survey”, 4. “one month survey”) of the source of the 98% figure Lott has now given.

“I did another survey over 10 days this past fall and it will be discussed in a book coming out in a couple of months. The results of the survey are very similar to those previously reported.”

It is surprising that with nine published surveys giving numbers ranging from 21%-67% for the percentage firing, Lott could conduct two surveys that give far lower figures. The questions and methodology of this second survey need to be examined carefully to seek an explanation. [There used to be some comments here about the results of Lott’s new survey. Clayton had posted that Lott’s new survey found a “bit more than 4%” firing. He informs me that he removed this because it was a mistake and could not post the correct figure because Lott does not want the results made public. Consequently I have deleted my comments on this figure.]

“I am not sure that I understand why things should be weighted by household size since I was asking questions about individual experiences.”

Because the sampling unit was the household. Let me demonstrate with a small example. Suppose that half the household have two adults and half have one adult. Suppose further that adults in small and large households are equally likely to experience a defensive gun use (DGU), but adults in large households are more likely to shoot, 30% (large) vs 12% (small), say (because they are defending a spouse and children as well as themselves).

OK, now, since 2/3 of people and hence 2/3 of DGUs are in large households, it follows that 2/330+1/312 = 24% of DGUs involve shooting. However, since Lott is sampling by household, he will find half of his DGUs in small households and half in large households and hence 1/230+1/212 = 21% of his DGUs involve shooting.

Notice, however, that even in this contrived example the difference is small, so this can’t possibly explain Lott’s anomalous 2% shooting figure. I also did an analysis on the 1992-2001 NCVS data set, and it gives 21% shooting whether you weight by household or don’t weight by household.

Next, Clayton gives me too much credit. Otis Dudley Duncan deserves the credit for uncovering this mess. As part of my critique of Lott I listed his 98% claim in a long list of errors in More Guns, Less Crime, but without expressing any doubts that he actually conducted a survey. Duncan contacted me, suggesting that the survey might never have been carried out. While I was initially sceptical, Duncan was able to persuade me that there was a real issue here. (The key was Lott’s failure to respond to a reasonable request from Duncan for evidence that the survey had actually been carried out.) In September 2002, I raised the issue in Eugene Volokh’s firearmsregprof mailing list. The ensuing discussion involving Clayton, myself, Glenn Reynolds and several others prompted James Lindgren’s investigation that he reports on here.

Finally, I should comment on the overall significance of this question. Lott’s 98% claim takes up just one sentence of his book. Whether or not it’s true, it doesn’t affect his main argument, which is about alleged benefits of concealed carry laws. I don’t think any fuss would have been made if Lott hadn’t repeated the claim numerous times on TV shows, on radio shows, and in opinion pieces. I have a list containing 52 examples. By suggesting to millions of people that all you have to do defend yourself is wave a gun around when all the evidence suggests that more serious action is often required, Lott could cause people to act in ways that could get them injured or killed.

Lindgren has updated his report. Main changes are the inclusion of a reply from John Lott and a dissection of Lott’s new “Did I say three months? I meant one month. Yeah, that’s the ticket!” claim.

Lots more people have blogged on this: Glenn Reynolds, Pejman Yousefzadeh, skippy, Ken Parish, Roger Ailes (twice), Atrios and Guy Cabot. And Marie Gryphon, Julian Sanchez, Jane Galt, Kevin Drum and Thomas M. Spencer have updates or new comments.

Glenn Reynolds and Thomas Spencer mention Bellesiles, but from opposite sides. Glenn states that “Lott’s critics want, rather too obviously, for this to be another Bellesiles affair” while Thomas asks “Where are Bellesiles’ critics?”. Uh, guys, the set of Lott critics and the set of Bellesiles critics has an intersection called James Lindgren. So the answer to Thomas’ question is “front and centre!”. And, uh, Glenn, Lindgren really didn’t want this to be another Bellesiles affair. Lindgren deserves our thanks, again, for doing an unpleasant job. As for myself, I’m certainly critical of Lott, but this is not something I wanted—it’s like winning a race because your opponent is disqualified for cheating at the starting line.

Glenn also argues that even if Lott is found to have fabricated data, it would only be something akin to the Jon Ellis affair. This is patently false. Ellis lied about his past, but his dishonesty did not extend to his published work. Lott published his claim about conducting a survey on page 3 of More Guns, Less Crime and repeated it in the Criminologist. Now, the claim is only a very small part of More Guns, Less Crime, but the probate data that got Bellesiles into so much trouble was only a small part of Arming America. Lott went on to make the issue important by publishing his 98% figure in: the Los Angeles Times (three times), Investor’s Business Daily, Wall Street Journal (four times), The American Enterprise, Chicago Tribune (twice), American Bar Association Journal, Consumers’ Research Magazine, National Review, Washington Times, Insight on the News, American Experiment Quarterly, Intellectual Capital, Rocky Mountain News, Detroit News, Christian Science Monitor, National Forum, San Diego Union-Tribune (twice), Boston Herald, Dallas Morning News, Philadelphia Inquirer, New York Post (twice), Chicago Sun-Times, and the Detroit News. He also made the claim in testimony for the Nebraska Committee on Judiciary LB465 and the US House Judiciary Committee. He’s done it on TV—Uncommon Knowledge, a TV pilot hosted by John Stossel and Hardball. He’s done it on radio—South Dakota Public Radio, NPR’s ’s “Justice Talking”, the Zoh show and Radio Liberty. He’s done it in public talks—to the Independent Women’s Forum, at the 2000 Gun Rights Policy Conference and at the Eagle Council Forum XXVIII. And in all these cases he used the figure to argue against gun control. It’s hard to imagine a case more unlike that of Jon Ellis.

Finally, I should comment on the suggestions that Lott can somehow salvage things with his new survey. While I have it on good authority that he really did conduct a survey this time, it is really very easy to alter the interview records to give any result you want. Unless some trusted third party has verified all the records and results of the survey as being unaltered, there will be a big question mark over this survey too.

John Lott has emailed Jim Henley with another response to Lindgren’s report. I’ll comment on this below. Julian Sanchez also got the Lott email. I didn’t.

Comments from Jesse Walker at reason.com, skippy, Guy Cabot, Kevin Drum and Steve Verdon. Brief mentions by Iain Murray, Charles Murtaugh, Tapped, John Quiggin, Natalie Solent, Gene Hoffman, Jr., jeff and Bruce R.

In Lott’s latest response, he changes his story again. He originally told Lindgren that hadn’t discussed the survey with anyone at the time. Now he has recalled the name of an economist he discussed it with at the time. Unfortunately, when Julian Sanchez contacted this economist, he was unable to recall those discussions. He also originally told Lindgren that the survey was conducted by “several University of Chicago undergraduate volunteers” (a survey of the size he claims to have conducted would have required at least ten students, and much more than that if almost everything was done in one month as he is now claiming). Now he seems to be saying that only two students from U of Chicago were involved, and the rest were from other places. This latest story change from Lott is truly disturbing because it appears to be designed to stymie Lindgren’s plan to settle the issue. This was Lindgren’s plan:

“If someone were to email the 1997-98 Chicago college alumni twice, I remain virtually certain that at least one of those who did the study for Lott would come forward, if the study were actually done. If no one came forward after two attempts emailing the entire class, the evidence would tend to point strongly to the probability that the 1997 study was never done.”

If there only two U of Chicago students involved, then, given the frequency at which people change email addresses, it would not be at all surprising to get no response from the mass email. Lott has changed his story and changed his story until he has ended up with one that cannot be proven or disproven.

Julian also has some comments where Lott claims that the difference between the 2% shooting figure he got and the 21% to 67% shooting numbers from all nine published surveys is not statistically significant. He mentions that in his current survey he found 13 defensive gun uses, and in only one of them (8%) was the gun fired. We can calculate a 95% confidence interval for the percentage (via Wilson’s method for those interested in the technical details). It’s 1.4%-33%. (If you’re not sure what a 95% confidence interval means, it’s the same thing people are talking about when they say that a survey is plus or minus 3%.) Since the interval includes 2%, this new survey is consistent with what he alleges he found in his first survey. Trouble is, since it also includes 21%, 24%, 27% and 28% it is also consistent with most of the big published surveys. (These had hundreds of defensive gun uses in the sample.) The trouble is that the sample size of 13 defensive gun uses is much too small to say anything meaningful about the percentage who fire their weapons. Most statisticians would not even print a percentage based on a sample size of 13, instead they would print an asterisk, with a footnote stating that the sample size was too small for the number to be meaningful. Lott’s claim that his new survey confirms his previous one is nonsense.

As for Lott’s original survey, he hasn’t told us how many defensive gun uses were found in that survey (maybe he has forgotten?) so we can’t determine if his 2% shooting number was significantly different from the numbers in the other surveys, but there are only two possibilities:

1. it wasn’t, in which case he should not have repeated the statistic 50 times when there were numbers from other, vastly more statistically reliable surveys available
2. it was, in which case Lott’s latest explanation is wrong, and he still has to explain why his survey gave such a radically different number.

Jim Henley has an excellent blog roundup with thoughtful comments and even ratings.

Julian Sanchez has an update where he includes a comment from an economic researcher that repeats what I wrote below: if his new survey only has 13 defensive gun uses, then this sample is far too small to say anything useful about the frequency with which defenders shoot, and certainly can’t be regarded as “confirming” the alleged 1997 survey.

Thomas Spencer has a post that is critical of some of the folks who were all over Bellesiles for downplaying Lott’s sins.

Marie Gryphon (who broke the story into the blog world) also got the Lott email. I feel so left out.
Brief comments from Jane Galt, skippy, Alan Schussman and jeff. Ted Barlow even has a Lott light bulb joke.

Mark Kleiman has an insightful post that is definitely worth reading. Mark observes that even though Lott’s work on concealed carry laws does not rely on his alleged survey, destroying his credibility means that people cannot trust the results of his concealed carry laws. Actually, there is actually no reason to trust the results of his research on concealed carry laws in any case, since it has been superseded by the work of Ayres and Donahue who repeated it using more data and better models:

“Those who were swayed by the statistical evidence previously offered by Lott and Mustard to believe the more guns, less crime hypothesis should now be more strongly inclined to accept the even stronger statistical evidence suggesting the crime-inducing effect of shall issue laws.”
Ayres, I. & Donohue, J. 2002. “Shooting Down the More Guns, Less Crime Hypothesis” Stanford Law Review

Mark also suggests observes that permit holders very rarely commit crimes, so carry laws could not cause increases in firearms crime. However, carry laws could cause criminals to carry more frequently and increase crimes that way.

I want to comment further on the email Lott sent to some bloggers. Lott states that he has been responsive and implies that he was somehow ambushed on this issue.

Duncan first asked Lott for evidence for his survey last June. Lott did not respond. I sent him the first version of my report last September, and regularly sent him updates. He has not responded to any of them. I’ve tried my best to be open, while Lott has acted like someone with something to hide. For example, this new survey was done months ago, but Lott kept it secret until Dec 26. The results, which Lott claims are “very similar” are still secret.

Here’s a simple question that Lott could respond to if he is going to be responsive: “Why, if it came from your survey, did you attribute the 98% figure to “national surveys” and to Kleck?”

I’ll let Lott have the final word in today’s post:

“The reluctance of gun-control advocates to release their data is quite widespread. In May 1997 I tried to obtain data from the Police Foundation about a study that they had recently released by Philip Cook and Jens Ludwig, but after many telephone calls I was told by Earl Hamilton on May 27. “Well, lots of other researchers like Arthur Kellermann do not release their data.” I responded by saying that was true, but it was not something that other researchers approved of, nor did it give people much confidence in his results”
John Lott, page 291 of “More Guns, Less Crime”

Steve Verdon comments on the Ayres and Donahue study I cited yesterday. Unfortunately he doesn’t seem to have understood what their conclusions were. Try reading the abstract, Steve:

“Estimating more statistically preferred disaggregated models on more complete county data, we show that in most states shall issue laws have been associated with more crime and that the apparent stimulus to crime tends to be especially strong for those states that adopted in the last decade.”

Now this study is correlational, so it certainly doesn’t prove that carry laws cause more crime, but if you believed that Lott’s correlational study proved that more guns caused less crime you should also believe that Ayres and Donahue have now proved that more guns caused more crime.

Otis Dudley Duncan has sent me these comments, which draw attention to a key point that almost all bloggers have missed:

“There are two distinct issues in this case.

1. Lott repeatedly made erroneous statements about the findings of other researchers. None of the national surveys that he cited by name actually had any figure at all for merely brandishing or firing. One of them, the Roper survey (which was mentioned in the Feb. 6, 1997, Nebraska testimony) never even did any survey on defensive gun use. Of the polls that did collect data on firing, none of them obtained a figure anywhere near Lott’s 2 percent. So everything he has said about surveys on this topic done by others is utterly, totally false. There is no room even for reasonable doubt.
2. Did Lott himself carry out the survey that he described in his “Reply” of Sept./Oct. 2000? It appears that he will not be able to produce compelling evidence that such a survey was done. But the internal evidence indicates that no survey with the results he described can have produced responsible statistical estimates of population frequencies: sample size = 2,424; population estimate of 2.1 million defensive uses in a population of about 200 million American adults as of 1997 (rate = about .01); 98% of the uses involving merely brandishing, 1.5% firing warning shots, 0.5% firing at offender. An event (firing in self defense) with a probability of about 1 in 5,000 (.01 x .02) cannot be honestly estimated with a sample size of 2,424, let alone the 1 in 20,000 probability of firing at an offender.

“Thus, while it is important to continue the investigation Lindgren has begun into whether the survey itself was actually faked, there can be no serious question as to whether Lott is guilty of fabricating evidence in his reports on what others found. And at best he has convicted himself of gross incompetence/culpable negligence in his report on his own purported survey. As to the first of these charges, Lott has never seen fit to comment in any way on why, knowing as early as Spring 1997 that the brandishing figure of 98% was obtained in his own survey, he chose to attribute it to surveys done by others. Nor has he explained why the 2000 edition in which the claim about his own survey first appeared there was no explanation of the revision of the statement on page 3.”

Jim Henley gets another email from Lott. It’s not fair!

Roger Ailes asks “how reliable is a self-reporting survey of ‘defensive gun use’ in the first place?”. You’re opening a real can of worms here, Roger. Survey estimates of the frequency of defensive gun use range from 80,000 (NCVS 87-92) to 23,000,000 (NSPOF). There is heated debate in the criminology literature as to what the correct number is. You can, for example, read Hemenway (”100,000 DGUs”), Kleck (”2.5 million DGUs”), Smith (”both sides are wrong”) or my own endless writings on this topic. However, despite such huge differences in the frequency of DGUs, these surveys agree about the percentage of defenders that fire their guns. Since they can’t all be right, it is a mystery how they could agree so well about the firing percentages. All this makes it less credible that Lott could have conducted a survey that came up with such a radically different shooting percentage.

Oh, and another thing. It is possible that all of the published surveys are wrong and the shooting percentage really is 2%. This still wouldn’t let Lott off the hook, since he claims to have conducted a survey that gave 2% firing. The relevant question as to whether his claim to have conducted a survey is credible, is what people tell you in surveys, not what the correct number is.

Julian Sanchez finds evidence that Lott lost data because of a computer crash. I’m afraid that he hasn’t discovered anything new—his time would have been better spent reading Lindgren’s report:

“I talked with one of Lott’s co-authors on another paper, Bill Landes, and received emails from David Mustard, another co-author, and Gregory Huck, Lott’s editor at the time at the University of Chicago Press. With varying degrees of certainty, all give circumstantial support to Lott’s story of a sudden loss of data and text on projects, requiring delays and regeneration of work.”

Marie Gryphon, who broke this story into blogspace has posted Lott’s latest response.

In his latest response, Lott accuses both Lindgren and Duncan of making false statements. Some of the points in dispute involve what was said in a conversation between Lindgren and Lott. Here there is no direct evidence who is right, so readers will have have to judge for themselves who is the more credible witness. I can offer one piece of evidence on this question. I went over both Lindgren’s “Fall from Grace: Arming America and the Bellesiles Scandal” and Lott’s “More Guns, Less Crime” looking for holes. Here is the long list of errors I found in Lott’s work. I don’t have a list for Lindgren’s work because I couldn’t find any. Lindgren’s work was written with meticulous care, Lott is careless with his facts.

On the other points it easy to find out who is telling the truth. For example, Lott claims: “My response in the Criminologist also discussed other incorrect claims by Duncan.”. Let’s see then. In his critique Duncan writes:

“The ‘fifteen national polls’ evidently include the thirteen summarized by Kleck and Gertz (1995), which is referenced by Lott, and by Kleck in Kates and Kleck (1997). The summary in Kleck (1997) adds to these the NSDS and NSPOF results, to make fifteen altogether.”

In his response Lott writes:

“Duncan (p. 5) is concerned about where the reference to ‘fifteen national polls’ comes from, when the Kleck and Gertz (1995) piece provides a table that only lists twelve national and three state level polls. What is not mentioned by Duncan is that the twelve polls do not even include Kleck’s own work and do not include subsequent surveys by Cook and Ludwig as well as my own research, thus producing a total of fifteen surveys.”

So does the table in Kleck and Gertz (1995) contain thirteen polls as Duncan says, or fifteen polls as Lott says? Well, I counted thirteen, but you don’t have to take my word for it—you can follow the link and count them yourself.

Lott writes:

“As to the attribution of sources, look at the complete context of the quote Lindgren mentions:

Polls by the Los Angeles Times, Gallup and Peter Hart Research Associates show that there are at least 760,000, and possibly as many as 3.6 million, defensive uses of guns per year. In 98 percent of the cases, such polls show, people simply brandish the weapon to stop an attack.—August 6, 1998, Chicago Tribune and August 14, 1998, Washington Times”

OK, I looked at the complete context and is perfectly clear. The phrase “such polls” clearly refers to “Polls by the Los Angeles Times, Gallup and Peter Hart Research Associates”. Lott has attributed the 98% figure to “Polls by the Los Angeles Times, Gallup and Peter Hart Research Associates”. He hasn’t attributed it to his own study. And none of the Los Angeles Times, Gallup, or Hart polls even asked if the gun was fired. Lott still refuses to admit that there is anything wrong with his attribution above. Lott writes:

As to so-called technical problems, I am have always acknowledged that these are small samples, especially when one breaks down the composition of those who use guns defensively. Even the largest of the surveys have few observations in this category.

Readers can easily check for themselves that on the over 50 times that Lott has cited the 98% statistic he never once stated that it was based on a small sample (a mere 25 incidents it seems), or that it was statistically unreliable. What’s worse is that on several occasion he has claimed that 3/4 of the shots were warning shots. Here the sample size is the number of incidents where the gun was fired. Which was 1/2. Yes, that is not a typo, Lott put forward an estimate based on a sample size of less than one. Furthermore is not true that “Even the largest of the surveys have few observations in this category. Kleck’s survey had 222 defensive gun uses, the NCVS has even more. The best case scenario for Lott is that on over 50 occasions he put forward a number based on a sample size he knew was to small to be statistically reliable, from a survey he knew that had been lost, and which was markedly different from the numbers produced by vastly more reliable surveys. In the best case he is guilty of scientific misconduct.

Lindgren has released a new version of his report. It’s long, but it’s an absolute must read. If you’ve read the earlier version, you can skip to section 4 to read Lindgren’s description of how Lott is trying to change his story.

The disgraceful way that Lott has behaved towards Lindgren fits into the pattern of behaviour he has displayed—Lott absolutely can not, will not admit to making a mistake. If he wanted to change his story about the Chicago students, all he had to do was say “Did I say they were all from U of Chicago? Oops. That’s not what I meant to say. I should have said that the only affiliation I recall for any student was the U of Chicago. Sorry.” Instead he insists that Lindgren’s account was not accurate. Well, by doing this, Lott has made things simple for us. We’re not talking about the events of 1997, where memories could be hazy, but a conversation last September about a matter of extreme seriousness. The choice is stark. Either Lott is giving a true account or Lindgren is. Choose.

Julian Sanchez has pointed out that yesterday I missed the point of his post about a confirmation from Lott’s editor that a section or chapter of More Guns, Less Crime was scrapped. He’s right. I got an email from Lott’s editor in September, confirming that Lott’s disk had been damaged and I didn’t notice that Julian’s had a bit more information. The September email has a few more tidbits, so here it is:

“I was John Lott’s editor at the University of Chicago Press for his book, More Guns, Less Crime, published originally in 1998 and then in a second edition in 2000. John has asked me to confirm for the record an incident that occurred in the summer of 1997, just as he was preparing the final version of the manuscript for submission to the Press. At that time, John reported to me that a bookshelf had fallen on his computer, seriously damaging his hard disk containing not only all his files and data for More Guns, but also work on some other projects as well. I recall that much of what was on the disk was lost and could not be recovered. We did have hard copy of most (but I think not all) of the book manuscript, however, and were able to proceed with that.”

Steve Verdon has responded to my critique of More Guns, Less Crime.

(more…)

If you haven’t looked at the new section 4 in Lindgren’s report, you should.

In his latest response Lott asks:

“There is also a question as to why people have waited so long to ask for this additional information when people have known about the lost data for years.”

A few people have known about the lost data since the Sep/Oct 2000 edition of The Criminologist came out. I only heard about it when Duncan contacted me in August last year. Most people probably only heard about the problem when this story broke.

In the summer of 2002 Duncan asked for more information about the survey because he discovered that Lott had continued to put forward the 98% figure over and over and over again.

It is, I suppose, conceivable that Lott might have forgotten that his 98% figure was not properly supported, but a responsible scientist would have withdrawn the figure as soon as was reminded of this. Of course, a responsible scientist would have also mentioned the other surveys that contradicted his in the first place. And a responsible scientist would not have advanced the figure in the first place, since the the sample size it was based on was too small for a meaningful estimate. Instead, even after his admission that he had no proper support for the 98% figure, Lott continued on his merry way, citing the 98% figure again and again and again. His only concession seems to have been occasionally adding the qualifier “up to”. (Reading this page will make you up to 98% smarter!)

[This is an email sent to John Lott that Lott posted to the firearmsregprof mailing list on January 20, 2003.]

I believe that in my emails to you and in my conversation with James Lindgren, I have stimulated just about all of my memories and impressions that I’m going to have concerning the survey that I took, without having any substitutions and additions by any possible suggestion, express or implied. I do not wish to be seen as a partisan in this matter, for I am not. I also am unwilling to speculate. My recollections and impressions of the survey that I took are now closed to prevent any accusations that I am having “convenient” recollections. I simply refuse to get in the middle of this any more than I already have. I am a witness who has come forward; I don’t have a dog in this fight. (more…)

From: John Lott
Sent: Monday, January 20, 2003 1:58 PM
To: Eugene Volokh
[Critical Commentry by Tim Lambert.
This is a copy of a January 20 email by Lott to the firearmsregprof list. My comments appear in italics like this.]

Response to Lindgren’s January 17th posting:

Lindgren’s role in this process seems to be that of a prosecutor. [This is not correct. Lindgren is careful to present any evidence that supports Lott. ] He claims that he “never heard of a professor doing anything of that size with no funding . . . .” Well, I did the 2002 survey paying students $10 per hour out of my (more…)

Julian Sanchez posts some comments from someone who believes he was surveyed by Lott. Lott is in error when he states that there were no other gun use surveys at that time, but once these have been eliminated, we can regard it as established that Lott conducted a survey in 1997.

James Lindgren writes:

I am very pleased to be able to say that my informal inquiry has been brought to a successful conclusion. I had a long substantive interview with David Gross Sunday night and a short discussion tonight (Monday). I will write up my Sunday interview sometime in the next 5 days. It has some details not in [his] post to this list, including the kinds of questions he recalls as being on the survey and his explanation of how he happened to come forward.

The bottom line is that I found [him] credible. Overall, this is the direct evidence that I had been seeking from the beginning and that I always said had to be out there if a study had been done. I offered to act as an honest broker between Lott and his critics and seek out evidence that would tend to support the existence of the 1997 study if it could be found, and, in that capacity, I said that all trace of a national study does not just disappear because a computer crashes. Well, it doesn’t. For any large study actually done, I believed that someone would eventually come forward, though I expected it to be on the asking rather than the answering side.

The inquiry, though necessary painful, was basically successful—and better, I think, than the even messier alternative. Suspicions about whether Lott ever did the 1997 study have been hanging around for over 3 years. If we hadn’t had this effort, the suspicions that he had entirely fabricated a study would still be hanging over him.

As to the coincidence of someone coming forward, if 100,000 people knew of the concerns about the Lott study, and if those 100,000 were randomly selected (which they weren’t), we would expect one person out of the 100,000 to have been surveyed in a national survey of 2,424 people.

In the next week, I will write up the interview, add it to the report, and change the conclusion from an agnostic one to a more positive one. I will also probably respond to Lott’s latest missive posted on FireArmsRegProf. Then I will very happily put aside this unpleasant task. The main concern that I attempted to help people resolve was whether John Lott ever did the 1997 study. Secondary and legitimate concerns remain over the quality of the 1997 work to support the conclusion he reached, Lott’s odd and so far unexplained attributions of the 98% figure to other people’s studies rather than his own, and his shifting stories about what he told me. Some of these may never be cleared up, but the last of those might well be attributed to the extraordinary pressures that this affair has put him under.

Remaining Questions:

1. Why did Lott repeatedly make false claims that the 98% figure came from other studies and from Kleck?
2. Even Lott cannot possibly be sure that the correct result of his survey was 98% since there is no way to check his calculations. Why did he repeat the figure over and over again?
3. Lott has conceded that the size of the defensive gun use sample in his survey was very small. Too small, in fact, for the result to be statistically reliable. Why did he never even mention the markedly different results obtained from the other surveys with vastly greater sample sizes?
4. Why did he make his 98% claim well before his survey was completed? (And without attributing it to his survey.)

Tim Lambert’s questions:

1. Why did Lott repeatedly make false claims that the 98% figure came from other studies and from Kleck?
2. Even Lott cannot possibly be sure that the correct result of his survey was 98% since there is no way to check his calculations. Why did he repeat the figure over and over again?
3. Lott has conceded that the size of the defensive gun use sample in his survey was very small. Too small, in fact, for the result to be statistically reliable. Why did he never even mention the markedly different results obtained from the other surveys with vastly greater sample sizes?
4. Why did he make his 98% claim well before his survey was completed? (And without attributing it to his survey.)

From: John Lott
This is an edited version of what I sent Kleiman on 1/22.

I haven’t read whatever you are referring to by Lambert. As (more…)

If you don’t know who Mary Rosh is, you might want to read “The Mystery of Mary Rosh“. Also of interest might be the blog post that unmasked Mary, and the latest Mary Rosh news. [Editor’s note: Most of these postings were made to Usenet. Some were made to comment sections on blogs, two are comments on www.freerepublic.com and one is a review posted to Amazon.com. I collected them here for easy reference. ]

1999-08-18

Mary Rosh:

SAVE YOUR LIFE, READ THIS BOOK — GREAT BUY!!!!
Reviewer: maryrosh (see more about me) from Philadelphia
If you want to learn about what can stop crime or if you want to learn about many of the myths involving crime that endanger people’s lives, this is the book to get. It was very interesting reading and Lott writes very well. He explains things in an understandable commonsense way. I have loaned out my copy a dozen times and while it may have taken some effort to get people started on the book, once they read it no one was disappointed.If you want an emotional book, this is not the book for you. If you want a book with the facts, a book that tells you the benefits and risks from protecting yourself and your family from crime, a book that will explain the facts in a straightforward and clear way, this is the book to get.This is by far the largest most comprehensive study on crime, let alone on gun control. Professor Lott examines crime rates as well as accidental gun deaths and suicides for all 3,056 counties in the United States by year for 18 years. By comparison, the previous largest study on gun control examined 170 cities within one single year 1980. Lott examined 54,000 observations and the previous largest study looked at 170 observations. Lott used all the FBI data that was available from the first year that they released the county level data to the last year that they had put it out when he wrote his book. Unlike other studies, Lott used all the data that was available. He did not pick certain cities to include and others to exclude. No previous study had accounted for even a small fraction of the variables that he accounted for.

[Editor’s note: This review has been deleted from Amazon.com] (more…)

Oh, and in amusing side note, Julian Sanchez uncovered evidence that determined Lott defender Mary Rosh is actually John Lott, and Lott actually confessed. Atrios, Roger Ailes, Kevin Drum and Tom Spencer also seem amused. I’ve had some discussions with “Mary” on Usenet. Her argument style is that if the facts disagree with Lott, then so much the worse for the facts. You can see an example here. You can read her Amazon review of More Guns, Less Crime here. And check this posting out, where “Mary” defends John from a criticism of his 98% brandishing number and rips into Bellesiles for repeatedly changing his story.

Well, after yesterday’s revelations, blogspace seems to be split. On the one hand, we have Clayton Cramer, Steve Verdon, Jane Galt, Glenn Reynolds and Marie Gryphon who think Lott has been exonerated. On the other hand Kevin Drum and Tom Spencer are not yet convinced. On the gripping hand we have Jim Henley and John Quiggin who think that it has been established that Lott conducted a survey, but the small sample size means that Lott could not properly use it make his “98% brandishing” claim. Julian Sanchez’s friend also makes the point about the sample size being too small.

I appreciate the concerns of those who think that one person saying that he had been surveyed is not sufficient to settle the matter. After all, anybody in the USA could have come forward and say that they had been surveyed. However, Lindgren questioned him carefully about what questions he was asked and they corresponded pretty well to what Lott said they were (with one major discrepancy which I will comment on in the next paragraph). The only way this could have happened is if he had conspired with Lott, and given the blundering way Lott has defended himself, I just don’t buy it.

The major discrepancy is that since Lott was only counting defensive gun uses that occurred in the year before the survey and this gentleman’s one was four years before the survey, it should not have been counted. Lott himself argues that the explanation for this is that his student interviewer screwed up! Well, if a sample size of 0.5 is good enough for Lott to conclude that 75% of firings are warning shots, a sample size of one should be more than sufficient for us to conclude, using the Lott method, that 100% of Lott’s interviews were not conducted correctly.

Anyway, I now believe that Lott did conduct some sort of survey. The next question is, did the 98% figure come from that survey? The evidence suggests that it did not. Lott put forward the 98% figure well before the survey was complete. He repeatedly attributed the figure to other surveys or Kleck, only changing his story after Duncan pointed out that the other surveys either said nothing on the subject or gave radically different results. (And note that Lott actually still denies he has changed his story about the attribution of the figure.) And even with his small sample size, it is still extremely unlikely that a survey could have come up with a brandishing figure as high as Lott’s. (Though this last one could be explained by arithmetic errors on Lott’s part.)

Furthermore, even if the 98% did come from his survey, Lott is still in trouble. Proper scientific conduct requires that if you make a claim, you had better have the evidence to back it up. By his own admission, Lott does not. He should have withdrawn his 98% figure long long ago instead of repeating it over 50 times.

And, even if by some miracle Lott found all the missing data from his survey and was able to recreate all his calculations, he would still be in trouble. As others have noted, the sample size of his survey (25 DGUs it would seem) is just too small to give a statistically reliable estimate. Other surveys, such as Kleck’s have a much larger sample of defensive gun uses, but Lott never ever mentions the brandishing number from Kleck (unless you want to count the times when he falsely attributes the 98% brandishing number to Kleck). Lott’s excuse here (from his 14 Jan response is

“As to so-called technical problems, I am have always acknowledged that these are small samples, especially when one breaks down the composition of those who use guns defensively. Even the largest of the surveys have few observations in this category.”

Meanwhile Kleck and Gertz write about their survey:

While estimates of DGU frequency are reliable because they are based on a very large sample of 4,977 cases, results pertaining to the details of DGU incidents are based on 213 or fewer sample cases, and readers should treat these results with appropriate caution.

No similar caution appears in any of the over 50 times Lott advanced his 98% figure based on a mere 25 DGU cases.

Kevin Drum suggests that the large scale of the Lott/Rosh deception suggests that Lott maybe could have carried off a conspiracy with this witness. Sorry, but I still don’t buy it. Lott’s a liar, but he’s a clumsy one. He could have saved himself most of the embarrassment of this Mary Rosh affair, if he had lied and had “Mary” admit to being Lott’s wife. Kevin also comments on Lott’s fishy statistics.

Tom Spencer also doesn’t think Lott is off the hook and seems rather unimpressed by the silence of Clayton Cramer and Glenn Reynolds. Over in the comment section at Hit and Run Glenn explains why: he doesn’t think that Julian’s revelations are “actual news”.

And people are still enjoying themselves with Mary Rosh: At Electrolite, soundbitten, Roger Ailes, Eschaton (twice), PostPolitics, The Liquid List and godofthemachine.

Yesterday I described the Mary Rosh escapade as a side-note, but on reflection I realize it isn’t. Posting under a pseudonym is perfectly OK. Inventing a few personal details to explain why Mary was so familiar with Lott’s work I also consider fair game in maintaining the pseudonym. The fact that “Mary” would follow me round on Usenet, follow-up to my postings and accuse me of dishonesty has had me rolling around on the floor, laughing. But, and here is the important point: when Lott wrote

“I have not participated in the firearms discussion group nor in the apparent online newsgroup discussions”,

he was lying. And not about something major. It was really a minor point in his response. He could have left it out if wanted to be truthful. Instead, he lied because he thought he wouldn’t get caught. He lied for some trivial advantage because he thought he would get away with it.

Julian Sanchez drops another bombshell: the person who came forward saying that he believed that he had been surveyed is a pro-gun activist. Julian’s revelations prompted John Quiggin to change his mind—he thinks Lott’s employer should fire him.

Ok, so what about this witness being a pro-gun activist? Well, he made it perfectly clear to me that he was strongly pro-gun and greatly admired John Lott in the long message that was sent to firearmsregprof. I should have mentioned it that day, but I knew all of the detail would be in Lindgren’s long report so I didn’t think to say something. In hindsight, I realize it made it look as if this guy might have concealed that he was a big fan of Lott’s from us. He didn’t. I apologize for the misleading impression I created.

Lindgren has sent me a long comment on this topic:

Things happen so fast in the blogosphere that people may have been jumping to some unwarranted conclusions. I didn’t want this to come out piecemeal, but I don’t have time to go through the whole thing even now. I had a manuscript due yesterday and a committee report due today, which are still commanding almost all of my time:

My Monday night post to FireArmsRegProf responded to an earlier post to the list from David Gross, the man who came forward and said that he thought that he was surveyed by a student working for John Lott. In his post, Gross disclosed to the list that he contacted Lott through Joseph Olson (who is a pro-gun rights academic widely known on the list to be one). In his post, he mentions his close ties to Leroy Pyle, a pro-gun rights activist. Further, he makes clear how much he likes John Lott’s work and how it resonates with him. So I wrote my post to a group that already understood that Gross had pro-gun ties and sympathies. I take it that many people in the wider world either did not read David Gross’s post or did not understand the possible significance of the people mentioned. Further, Gross in his post expressed his admiration for the work of Milton Friedman and his contacting Dan Polsby (a former Northwestern professor also known to support gun rights). If that weren’t enough, Gross made a statement about the necessity of his defensive uses and described his tape of a speech by John Lott, which he cared enough to purchase at the event and keep. Only by missing all these clues in Gross’s own post (or not seeing his post) would you conclude that he had no special interest in gun rights. His post says that he has no stake in the outcome, but he openly admits his attachment to Lott’s work.

After saying that I had a long interview with Gross, I wrote only 2 quite oblique sentences about what he said:

It [the interview] has some details not in Gross’s post to this list, including the kinds of questions he recalls as being on the survey and his explanation of how he happened to come forward.

The bottom line is that I found Gross credible.

So really, I said nothing specific Monday other than I found Gross credible. I did not say, nor in the context of the list to which I posted would anyone sophisticated believe, that Gross lacked a strong attachment to John Lott’s work, since Gross confessed that he did. I assumed that my readers on FireArmsRegProf either knew or assumed this from Gross’s own post. I think the confusion arose mostly among those who did not see the earlier email to which I was responding.

I also promised to reveal how he came forward, which I will do now. According to Gross, he received an email from a gun rights/police list that Leroy Pyle runs or is involved with. That email, which he read part of to me, informed readers that people were attacking John Lott, questioning whether he did a study. Gross then called Joseph Olson, who put him in touch by email with John Lott, who forwarded Gross’s email to me and some friends and bloggers. I called Gross fairly quickly and Gross said that he had not spoken to Lott. Contrary to some accounts, I did not discover Gross and he came to Lott (apparently by email) before he came to me.

Also, even before I posted my brief announcement that Tim Lambert posted on his website, I emailed Lambert and informed him explicitly that a “pro-gun” rights guy had come forward to say that he was surveyed in a fashion that fit Lott’s account in most, but not all, respects. I further told Lambert that Gross had come forward because of an email from a gun rights/police list. So Lambert knew some of Gross’s pro-gun rights orientation when he first posted his views on the respondent Gross.

During my interview with Gross, he did not hide his pro-gun activities, telling me at length about his legal representation of a gun club. He mentioned contacts with the activities of Mr. Pyle and told me that he was friendly with Joseph Olson, who introduced him to Lott at the end of Lott’s January 1999 speech. Gross is a lawyer and former prosecutor.

For the terminally curious (as most blog readers are), Lambert has a little bit on discrepancies between what Lott said and what Gross said on his website. Please don’t interpret my silence on that issue at this time to suggest that there are none. Further, I will explain in more detail about what in Gross’s responses made me think that he was more likely than not telling the truth. I am not vouching for Gross; I just found him a credible witness. Bloggers who want to form their own opinions might call Gross and make up your own mind (as I suggested on Wednesday afternoon to Julian Sanchez). Last, I hope people will remember (as my current report covers in tedious detail and as I also noted in my earlier post on Gross):

Secondary and legitimate concerns remain over the quality of the 1997 work to support the conclusion he reached, Lott’s odd and so far unexplained attributions of the 98% figure to other people’s studies rather than his own, and his shifting stories about what he told me. Some of these may never be cleared up.

Bloggers just can’t get enough of Mary Rosh. We have Andrew Conway, Tim Dunlop and Jane Finch. G. Beato even has a picture of Mary.

Tom Spencer comments on Glenn Reynolds curious reluctance to mention Mary Rosh, apparently because it is not “actual news”. Well, actual people have changed their minds about Lott because of his Mary Rosh deception and the associated lying. One of them is John Quiggin. Yesterday I said that Julian’s revelations had caused him to change his mind. Actually it was just the Mary Rosh thing. My error.

And, from the surely-it-can’t-get-any-weirder department: After John Lott made people promise not to reveal the questions in his new survey, Mary Rosh posted the questions from the new survey to Usenet. I’m imagining it went something like this:

Lott:

Nasty tricksy bloggers. We hates them, we hates them all.

Rosh:

No, some bloggers are nice. It’s Lindgren we hates. He hurts us with his cruel report.

Lott:

I’ll show them with my Precious survey questions. Keep them secret, we can escape, even from Lambert, eh? Grow strong, eat fish every day.

Rosh:

No, we should post survey questions. Bloggers see we be good, good as fish.

Lott:

But He will see, He will know, must keep Precious secret!

Rosh:

[posts survey questions]

Lott:

Nooo! My Precious!!

Atrios has a disturbing report on David Gross. He is an extremely active pro-gun activist and was involved in a dirty trick that stole the name of a pro-control organization. Ouch. Well, I may start getting many more emails from Nigeria, but I’m still inclined to believe him. However, I imagine that lots of other people won’t, so we can’t consider the question of whether Lott did a survey to be closed. We really need one of his student surveyors to come forward. The Washington Times whitewash may well have hurt Lott, by discouraging people from coming forward.

Robert Stacey McCain has a disgraceful whitewash of the affair in the Washington Times. The most important thing about this affair has always been whether Lott’s 98% brandishing claim is true. By any objective reading of the existing evidence it is not. On the one hand we have nine surveys that say the number is much lower, on the other hand we have Lott’s new survey, where even Lott concedes that the sample size is so small that it does not contradict the big surveys. By advancing this 98% figure over 50 times Lott is giving people dangerously misleading advice. Advice that could get them injured or killed. “Just wave a gun around. Doesn’t even have to be loaded.” he implies. McCain’s article implies that the 98% number is correct: “Lott said 98%, he’s been vindicated, there you go.” Beside this horribly misleading implication the rest of the flaws in the article seem insignificant:

1. I get labeled as a “gun control advocate”, but Polsby and Gross don’t get labeled as “pro-gun advocates”.
2. It’s not about a single sentence; there are over 50 sentences involved.
3. Again we see the utterly false claim that this is some sort of payback for Bellesiles. James Lindgren is angry about the treatment of Bellesiles? Ridiculous.
4. How can McCain possibly say that “Lott cited no source for that statistic in the first edition of More Guns, Less Crime“? Lott cited a source, clear as day, and McCain even quoted him: “National surveys”. This is the sneakiest dodge of all, because it enabled him to avoid discussing what “national surveys” actually said, which was something very different from what Lott claimed.
5. McCain selectively quotes Lindgren to make sure his readers do not learn that there are still unresolved questions about Lott’s claim.

Kieran Healy makes an absolutely first class post on this topic. Kieran writes:

In any event, it seems trivially obvious to me that you shouldn’t make claims on the basis of data you don’t have. This is especially true when your claims are inconsistent with all of the other available data on this issue.

Exactly. What is it going to take to get Lott to behave in a professionally responsible manner and withdraw his 98% claim?

From: John Lott
Sent: Saturday, January 25, 2003 2:49 PM
Please post

Claims by archpundit (apparently reflecting similar claims made in other places): (more…)

Julian Sanchez is on the case again. This time he has a bit more detail from Mustard. The key point is that Mustard is “fairly confident” that Lott told him in 1997 that he had done a survey. This suggests that Lott didn’t invent the survey in 1999 to explain his 98% figure. Well, this makes me lean more towards Lott having done a survey, but it’s still not conclusive. Mustard isn’t sure about being told in 1997. All this back and forth is making me dizzy. I’m not going to express another opinion on whether he did a survey until I see Lindgren’s new report.

Kevin Drum gives us one, two three postings on the weighting, lack of IRB review and some more implausibilities. For what it’s worth, I enquired about the IRB thing last year and was told that folks in Law schools often ignore it, so it was no big deal. Ted Barlow outdoes him with four postings. Ted makes a good point about the sheer quantity of paper you get with even a small survey and has a good summary of the problems in his top posting. ArchPundit costs Lott’s survey and also has two postings with a nice list of possible explanations of what Lott actually did, all of which are bad for Lott, though some are less bad. Lott says he got student volunteers to do it, so it would not have cost as much as ArchPundit estimates. Tom Spencer posts twice. Meanwhile Atrios only posts once. Well, actually he posts a bajillion times, but only one is about Lott.

Steve Verdon comes round and agrees that Lott should withdraw the 98% figure but thinks I should lay off now that I’ve won. But, Steve, Lott has not withdrawn the 98% figure. In fact, he just sent a message to firearmsregprof where he says:

Would one want even larger samples for brandishing so as to get even tighter confidence intervals? Sure, but I have limited personal resources and the point estimate gives us the best guess that we have for the rate of brandishing. I do not believe you can point to anything that has me claiming more for this result than was appropriate. The sentence in the second edition (2000) even added a cautionary phrase at the beginning of the sentence.

Look at that, he insists that 98% is the “best guess that we have for the rate of brandishing.” And look at what he says about the change between the first edition and the second edition. Here’s the sentence in the first edition:

“If national surveys are correct, 98 percent of the time that people use guns defensively, they merely have to brandish a weapon to break off an attack.”

and here it is in the second:

“If a national survey that I conducted is correct, 98 percent of the time that people use guns defensively, they merely have to brandish a weapon to break off an attack.”

He didn’t add a cautionary phrase at all. He changed the attribution of the 98% figure. And he won’t admit to doing it.

From: John Lott
Sent: 1/26/03 1:44 PM
Subject: Response to Lindgren’s “Lott’s Tax Arguments”
Response to Lindgren’s “Lott’s Tax Arguments”

My wife, who took care of our taxes, has a discussion below about how the forms were completed. The main point is that all payments to research (more…)

If you are interested in the question of whether Lott’s claim that passing concealed carry laws reduce crime, you might want to read this article, which reports that a new book, Evaluating Gun Policy will be published in February contesting Lott’s claim. The book is partly based on the research in the Ayres and Donahue paper I mentioned a few days ago. I also located Lott’s response to that paper.

James Lindgren makes some interesting points in the comment section to this Jane Galt post. First, he comments on this Lott claim about his tax returns:

As to deducting these costs on my income taxes, my 1997 tax form, which I have shared with many others, shows that $8,750 was deducted for research assistants (the heading was under “legal and professional services”). We do not keep the supporting documents past the three years required by the IRS and the $8,750 does include the expenses for other projects. On the other hand, I am sure that I did not keep track of all of my expenditures so the $8,750 is a sizeable underestimate of what I spent.

Lindgren observes that:

John Lott implies that his 1997 tax returns support his claims of having done the 1997 survey. They do not. I have not opened his main return (since no one has suggested that it has any relevant information). I did look at Lott’s 1997 Business expense schedule and there are no reimbursements for phone calls listed. Some commentators have also implied that the return shows payments of wages to research assts, which it does not, nor should it for the 1997 survey because Lott says that they were unpaid volunteers. Of course, Lott might have forgotten about the supposed phone research charges when he filed his return, but there are none listed, which tends to undercut his claim to have done the 1997 survey rather than support it. The return does support Lott’s contention that he spends large amounts of his own money on business travel, computer expenses, journal subscriptions, etc., so it is not hard to believe that he would be willing to fund a large project himself. But there is no evidence in the business deduction schedule on his 1997 return that he did the 1997 survey unless the expenses are mislabeled.

As to his business tax schedule, John Lott just posted to a discussion list that amounts he paid research assts in 1997 were listed on his schedule C under “Legal and professional services.” Of course, research assts should have been paid under “wages,” which lists no deduction, not “Legal and professional services,” which is for the professional services of firms and independent contractors like lawyers, book agents, and brokers.

But this is a moot point since he has made clear that he did not pay his volunteers on his 1997 survey.

Second, he hoses down the criticism of Lott over the IRB issue. As I mentioned yesterday, the IRB rules are routinely ignored at U of Chicago Law, so folks should not make a big deal about it.

Glenn Reynolds has a long post on the Lott affair. He attempts to frame things so that only question that matters is whether Lott did a survey or not. He wants to completely ignore the important question of whether Lott’s 98% figure is correct because he is “not qualified to judge the statistical merit of this stuff”. But you don’t need a mathematics degree to realize that if you don’t have the data to support your claims you should withdraw them. And as for the the question of sample size, why not listen to what Lindgren says?

Unless John Lott can come up with a sensible explanation for why his rates could possibly be justified with only a 2,424 person sample, it is my opinion that Lott should withdraw the 98% figure as probably erroneous and, in any event, too unreliable to form the basis of an estimated rate. Perhaps he has an explanation that doesn’t appear yet. If not, withdrawing the 98% figure in some appropriate way would be a simple matter of good social science.

Glenn then complains that many are trying to make this into a Bellesiles-payback case. Who? Who are these people? I’ve tried to read and link to every bit of coverage of this in blogspace and maybe it’s been too subtle for me, but I just don’t know who he is talking about.

If we must have a Bellesiles analogy here’s one. Suppose that Bellesiles had been claiming that only 2% of early American households had guns based on a random sample of 25 inventories. Suppose further that he had lost all the data and records supporting his claim and that there was no record of him even having visited the places where the records were stored. Then, someone (who happens to have been a director of a gun control advocacy group) comes forward to confirm that Bellesiles had indeed visited one of the places where the inventories were stored. Would Glenn have said “Case closed” and moved on?

Glenn also seems to have missed something very important about the Mary Rosh thing: Lott was caught making a blatant lie:

“I have not participated in the firearms discussion group nor in the apparent online newsgroup discussions”

Greg Beato also has some more comments about the significance of Mary for Glenn.

Atrios takes a swipe at the publisher of Lott’s new book and it looks like Mary Rosh has started a blog.

Lott has made some more responses to some of the questions asked and comments made. First, he has responded to my remaining questions I asked a while ago. Let’s see how he went:

1. Why did Lott repeatedly make false claims that the 98% figure came from other studies and from Kleck?

   Lott says:

   As to attributing things, in op-eds or talks I simply don’t go through and explain where every statistic that I mention comes from.

   This isn’t an answer at all. I didn’t ask why he didn’t give a source, I asked why he gave an incorrect source many times.
2. Even Lott cannot possibly be sure that the correct result of his survey was 98% since there is no way to check his calculations. Why did he repeat the figure over and over again?

   Lott says:

   Is there any evidence to suggest that I can’t figure out a weighted average?

   Well, yes there is. Lott’s 98% figure is markedly different from that obtained by nine published surveys. There is more to getting the number than just calculating a weighted average, since the weights must be computed, the interviews coded and so on. And while Lott has experience in analysing data, this was the first survey he ever conducted, and he admits that he did not get help from people experienced in the area.
3. Lott has conceded that the size of the defensive gun use sample in his survey was very small. Too small, in fact, for the result to be statistically reliable. Why did he never even mention the markedly different results obtained from the other surveys with vastly greater sample sizes?

   Lott just ducks this question.

4. Why did he make his 98% claim well before his survey was completed? (And without attributing it to his survey.)

   Lott effectively claims that the 98% number from his survey did not change after January because almost all the interviews were completed in January (at least 98% if he didn’t get more gun defenses after then). Fine. But on Feb 6, 1997 he couldn’t have known that the number wasn’t going to change. Why go to all the effort of weighting everything to get a preliminary estimate? And then not saying that it was a preliminary estimate? In fact, not saying that it came from his survey at all?

Second, he has a reply to some questions ArchPundit asked and also included his new survey. ArchPundit has fired back with one two three postings where he is rightly critical of Lott’s methodology. Jacob T. Levy is not impressed either.

And, third, he has a response to Lindgren’s comments on his tax returns yesterday. In this response Lott has changed his story again (see Section 4 of Lindgren’s report for some of the other changes he has made.) In September Lott told Lindgren:

The survey was done by phone by several University of Chicago undergraduate volunteers in their junior or senior years in 1997, so there are no financial employee records.

Now, he says:

While I no longer have records for what was paid them, it is my recollection that I did pay them something for their work and I did tell Lindgren that I had given them something.

Now maybe he has recalled something that he didn’t remember earlier, but he certainly did not tell Lindgren this in September.

Tom Spencer has some more information on David Gross and explains that he is one of the mysterious people Glenn was complaining about yesterday. It seems to me that if you think someone is behaving badly you should explain why and link to them so anyone interested can check for themselves, but I’m a total newb here and Glenn has been blogging forever, so I’m sure he knows what he is doing.

Steve Verdon observes that both versions of the famous sentence are conditional. OK, but my point is that Lott will not admit that he originally attributed the 98% figure to “national surveys”.

Mark Kleiman has a excellent summary of the recent developments. Mark ends with a plea to gun-rights folks —they should consider cutting Lott loose. (I would add, either that, or defend him—right now, Lott’s side is losing the argument by default.)

Archpundit has reply from Lott to his earlier critique of Lott’s methodology. ArchPundit makes some excellent points when he compares Lott’s survey to Kleck’s. Just one small point: I analysed Kleck’s data and found that the percent firing in the 1 year recall frame was almost identical to that in the 5 year frame.

Ted Barlow gives a nice simple explanation of what is wrong with the Lott’s 98% figure.

A reader sends in the following comment about yesterday’s post:

I think “question 3″ from your last update—significantly, the one he dodges—is the key, because many of Lott’s defenders seem to think you’ve shifted to a “different issue” now, when in fact it’s very much the same issue. Glenn et. al. seem to believe that the only question about Lott’s honesty is whether he told the truth about conducting a survey, when for a trained statistician, doing a slipshod survey with a meaningless sample size, then reporting the result without mentioning that any expert would consider the number sheer garbage, is dishonest. I mean, if he were citing some other survey, at least people could have checked the primary source and raised questions about how well it supported his claim. Here he was using the fact that his survey (which we’ll assume for now really happened) was lost to hide the fact that it didn’t back his statements—something that would have been obvious to everyone with stats training if the data had been preserved.

Reynolds wants to claim that this is now merely about the accuracy of Lott’s claims, and no longer about his honesty. But the kind of deceptive use of numbers we’re talking about is clearly fraudulent for someone who knows better—which an economist would. This, I think is a point you may want to stress, because it’s clear from the non-controversial data we already have. Force ‘em to explain why this kind of behavior doesn’t count as academic fraud.

I wish I could have put it that well.

Kevin Drum has persuaded Lott to consider doing an interview. I think that is an excellent idea. With no-one in blogspace defending him, Lott needs a way to get his point of view across.

Atrios points his readers at Mary Rosh’s blog, which has been updated again. (Though with only a few new postings.)

ArchPundit continues to