Deltoid

One more quote from yesterday’s Chronicle of Higher Education article:

Mr. Lott also points out that because the claim of coding errors appears in a law review, it has not been subject to review by third-party scholars, as would have been the case in a peer-reviewed economics journal.

It has been weeks since Lott saw the claim of coding errors. It would have taken him a few minutes to check for the existence of the errors and not much longer to see if correcting the errors reverses his results as Ayres and Donohue claim. He must know full well whether or not the claim is true. So, who cares if the claim has been reviewed by third-party scholars? It’s been reviewed by Lott. Does he concede the claim or is he going to deny it?

A gun control group has set up a “fan” site for Mary Rosh.

Still nothing from Lott on whether he concedes or denies the charge of coding errors.

In the mean time, let’s examine his other claim: “Ayres and Donohue have simply misread their own results.” This is a remarkable claim. Lott is saying that crime went down but somehow Ayres and Donohue read a decrease as an increase. If you look through the Lott/Plassman/Whitley (I’ll just abbreviate this to Lott) paper to find the basis for this, you find they are referring to Ayres and Donohue’s figures 3a to 3e. Figure 3b shows murder, with virtually no change in the first two years after the law, then a big decline for the next decade which bottoms out at 13 years after the law, and then an even bigger increase over the next four years.

Here’s what Lott says about this graph:

Their state level regressions indicate that murder rates were rising in the three years prior to the law being passed and then falling over the next thirteen years. Only one state, Maine, had had the law in effect for more than 13 years. The increase during years 14, 15, 16, and 17 thus solely reflect changes in Maine s murder rate and since this is state level data each coefficient represents only one data point. … The increase between years 13 and 14 is also more apparent than real. The real increase is actually not due to any sudden change in Maine s crime rates, but due to the fact that other states are included in calculating the crime rate for year 13, while only Maine is used for year 14.

Gee, how could Ayres and Donohue have “misread their own results” so badly as not to have noticed that decrease? Let’s look at what they wrote about that graph:

To underscore the message that the observations from the adopting states at the two ends of the time span, either well before passage or well after passage, are causing mischief when estimating a single aggregated impact of the shall-issue laws, it may be helpful to illustrate this point graphically. … Figure 3b shows the results for murder and it should immediately be apparent that the period from eight years before to three years after passage evidences relatively little movement in this crime category…. However, outside this time frame (especially in the last years), one sees large swings in the estimated effect of shall-issue laws on murder. Of course, the thought that shall-issue laws caused crime to drop by almost twenty-five percent in the thirteenth year after passage and then caused it to increase by almost twenty-one percent in the fourteenth year is obviously untenable. These wild swings are caused not by any true impact of shall-issue laws but by the selective dropping out from the individual year estimates of states that adopted the law more recently, leaving only the shrinking number of earliest adopters to identify the particular annual impact.

Oh, Ayres and Donohue did notice the decrease. They specifically pointed to the wild swing to show that it is incorrect to do an analysis that assumes that the law had the same impact every state. This graph appears in the first half of their paper—they go on to do analyses that do not rely on the incorrect assumption.

So what Lott did was take the part of their graph that showed a decrease, ignore the reason why the graph was included (except that he selectively used that reason to justify ignoring the increase), ignore the next sixty pages of their paper and claim that they “misread their own results”. I think he has set some sort of world record for cherry picking.

To summarize, Lott’s rebuttal of Ayres and Donohue consists of 50% cherry picking, and 50% miscoded data.

Mark Kleiman has written a must read post covering the recent developments and concluding:

1. defenders of gun rights should stop citing Lott as an authority
2. the University of Chicago Press should conduct a formal enquiry into the existence of the 1997 survey
3. the AEI should conduct an enquiry into Lott’s professional ethics

Mark both spoke to Lott and posted a long email. Yet again, Lott does not admit to making any coding errors. In fact he comes close to denying making such errors when he writes:

Ayres and Donohue’s attacks on the quality of our data are not only misleading, but it should be noted that these authors have not been equally forthcoming in sharing their own data.

In his email to Mark Kleiman, Lott accused Ayres and Donohue of lying:

However, the Stanford Law Review allowed Ayres and Donohue to add an addition to their piece commenting on all this. They said that:

“It is important to note that what we now refer to as the PW response has already been widely circulated as a draft, whose first author is John Lott. Moreover, Lott has repeatedly told the press and/or publish to the Internet that Ayres and Donohue have simply misread their own results. But after seeing this Reply to the original Lott, Plassmann, and Whitley paper, Lott asked the Stanford Law Review to take his name off the work. We hope that this indicates that the arguments in our Reply have caused the primary proponent of the more guns, less crime hypothesis to at least partially amend his views. We note that to this day, legislators are still voting for the adoption of concealed-carry laws while citing Lott’s work.”

Ayres and Donohue know that this is false because the ultimatum issued by the Stanford Law Review was made only at their request. Ayres and Donohue’s attacks on the quality of our data are not only misleading, but it should be noted that these authors have not been equally forthcoming in sharing their own data.

Now, it is clearly not true that the ultimatum was made at Ayres and Donohue’s request. Horwich denies this and the emails that Lott includes provide absolutely no support for this claim. Removing your name from a paper shown to based on faulty data seems an entirely appropriate thing to do, while removing your name to try to prevent a one word correction seems to be a rather unlikely thing to do, so Ayres and Donohue’s speculation as to the reason seems warranted.

However, Horwich knew that the speculation was in error, so I asked him why he didn’t ask Ayres and Donohue to correct it. He replied:

The most significant reason is that I read A&D’s new language as a debater’s point: There are two sentences of note:

“But after seeing this Reply to the original Lott, Plassmann, and Whitley paper, Lott asked the Stanford Law Review to take his name off the work. We hope that this indicates that the arguments in our Reply have caused the primary proponent of the more guns, less crime hypothesis to at least partially amend his views.”

The first sentence is entirely correct if you interpret “after” to be a temporal description, and not a causal description. In light of the second sentence, maybe it’s more of a causal “after,” but, frankly, that sort of debating tactic pervades this area, and so I didn’t think much of it. Moreover, “we hope” acknowledges that the statement was speculation. So the second sentence is literally correct, though mistaken.

Second, and related to this first point, is that I discounted it as a debater’s point because I had a different view of events. That might have been myopic.

Third, I wanted to move things as quickly as possible—remember, the ultimatum was only ever about getting the issue out so as not to hold up other authors’ pieces. That may be a reason, but not an excuse.

Fourth, I had only ever read the new language as impacting Lott alone, assumed that he expected as much, and had weighed it in his decision to withdraw. It’s not my job to consider the optics of other people’s decisions. That said, I do regret not having considered how the statement would be read with respect to Plassmann’s and Whitley’s continued participation. They do not deserve the small cloud of suspicion that it introduced over the work, and I’ve been taking every opportunity possible to disabuse folks of the notion that Lott’s withdrawal was on substantive grounds and therefore speaks ill of Plassmann and Whitley. That is simply not so, and I know all three stand behind the conclusions of the paper.

In sum, then: It was the right decision at the time insofar as the language was literally true and/or ambiguous; and I didn’t want to delay publication further still. It was the wrong decision, in hindsight, but only because of the subtle and unfortunate effects on Plassmann and Whitley.

I hope that is enough to put the “removing his name from the paper” thing to rest and Lott will now focus on the miscoded data question. Does he concede that he miscoded the data or will he deny it?

Brian Linse comments on Mark Kleiman’s post and suggests that people should write to the University of Chicago Press and ask them to investigate Lott. Tom Spencer also comments on Kleiman’s post, as do Kevin Drum and Matt Yglesias.

Lott has just published an op-ed in the Minneapolis Star Tribune entitled “Gun control advocates’ credibility on line”. This op-ed is largely recycled from one published a couple of weeks ago in the Columbus Dispatch. He cites research by Olson and Maltz that he alleges shows that concealed carry reduced gun carrying by criminals:

Other research, by David Olson at Loyola University and Michael Maltz at the University of Illinois, found that when law-abiding citizens carried concealed handguns, criminals were much less likely to carry guns. In fact, they found gun murders fell by 20 percent.

Maltz replied in a letter to the editor (I don’t know if it was published):

In an effort to promote laws permitting the carrying of concealed weapons, in the April 24th Dispatch John Lott supports his own position by citing research I did with David Olson, purporting to show that these laws reduce homicide. In doing so, Lott bends the truth so much that he breaks its back. Specifically, Lott is well aware of a paper Joseph Targonski and I published in September of last year that points out that the data that we used in that study was problematic and should not be relied upon. The conclusion of our more recent paper is that Lott’s data (and ours) were so error-laden that they cannot be used with any degree of reliability.

What is all the more galling is that Lott has used a pseudonym (”Mary Rosh“) on the Web to vilify my research and that of others. He also used this same pseudonym to write an enthusiastic review of his own book on Amazon.com and to inform others that he was an excellent teacher; in fact, “Mary Rosh” (Lott) wrote of himself that “I have to say that he was the best professor that I ever had.”

Moreover, he claims that he did a survey in 1997, with phone interviews of over 2000 people, for which he has no evidence, saying that the crash of his computer wiped out all of the data. Having gone through the same misfortune, I can well understand how this could devastate one’s research. However, he has no record of how he obtained the sample, of what questions were asked, of who made the phone calls, of how or whether he paid the interviewers, of how they were supervised and their results validated, indeed, of anything that would support his claim that he actually conducted the survey. In short, Lott is not a person whose research I would trust; he is obviously a gun advocate, but he claims to be an unbiased researcher, which he clearly is not. Rather than using research to clarify important policy issues like this one, he instead selects only those “facts” that support his own political position. Worse, he misrepresents the work of other researchers to further his own unsupported claims.

(more…)

Archpundit has some thoughtful comments on Mark Kleiman’s post. Ryan Barrow has a short comment. Glenn Reynolds would like someone else to check to see if the Ayres and Donohue are correct about the coding errors. I think the way that Lott persistently ducks the question gives a strong indication where the truth lies.

Tom Spencer comments on Maltz’s rebuke of Lott.

William Sjostrom gamely defends Lott against the charge that he anonymously accused Levitt of being “rabidly antigun”.

Julian Sanchez has an article in the May issue of Reason on the role blogs played in the investigation of the Lott affair.

So what was the one word correction that prompted Lott to remove his name from his paper? In their draft, Ayres and Donohue wrote (my emphasis):

On the other hand, the temporal pattern, that states adopting shall-issue laws in the late 1980s did better while those adopting in the 1990s did worse, may simply reflect the influence of a time-varying factor (the crack trade?) that caused sharp rises in crime for many states in the late 1980s, and then greater-than-average price declines in the 1990s.

Now the word “price” here makes no sense. Why contrast a crime increase in the 80s with a price (of what?) decrease in the 90s? The context of the sentence is all about crime increases and decreases. It seems obvious that they meant to say “crime” rather than “price”.

Not to Lott, however who felt that they were saying that there a relationship between crack cocaine prices and crime. From his draft:

While they don’t mention the use by Lott and Mustard of this variables, Ayres & Donohue, supra note 5, at 51, even imply a relationship between crack cocaine prices and crime when they mention “the greater than average price declines in the 1990s.”

Lott also puts scare quotes around the word “error” when mentioning their error, implying that it wasn’t an error, but what they meant to say.

Mark Kleiman has posted some comments from John Donohue about the Stanford Law Review controversy. Donohue isn’t even sure what the changed word was that caused Lott to withdraw his name. (Details are here if you are interested.) And like the rest of us, Donohue is puzzled as to why Lott has no direct response to the serious allegation of coding errors.

I also have some comments from Donohue responding to claims in Lott’s The Bias Against Guns. Donohue writes:

The figures on pages 237-239 of Lott’s new book are the same as Figures 4a-4f in the LPW reply (”Weighted least squares estimates using county-level data from 1977-2000″ for various crime categories). These are Figures that the Stanford piece shows to be wrong because of his mis-coding, but since they are now out in book form, I guess Lott doesn’t want to address the fact that they are all wrong.

Just before the figures above, Lott implicitly accuses me of being somewhat deceptive: “Donohue doesn’t use simple dummies for each year. Instead they [I guess now he’s talking about both me and Ian, even though he’s referring to the Brookings paper and we didn’t do this in the Stanford piece] aggregate years into two-year groupings…This is an unusual approach, but they do it to obscure in their results the large drop that occurs between the year of passage and the first full year that the law is in effect for their sample period.” But the Stanford piece does contain the year-by year dummies, which don’t support Lott in any event.

Lott then goes on to say “Despite all the work that has been done on the topic of concealed handguns, it is remarkable that no academic study has found a bad effect from these laws.” Not true if you take the aggregated state-specific effects on crime (which are modestly positive — ie. pernicious).

Kleiman also has another, interesting post on the mechanisms by which concealed-carry laws could cause crime increases or decreases. He can see mechanisms by which it might cause violent crime increases or decreases, but not property crime increases, suggesting that the property crime increases that Ayres and Donohue found were caused by something other factor. This other factor could well have caused the violent crime increases.

Well, there is a mechanism by which the laws could have caused property crime increases. In fact, Lott and Mustard proposed it in their initial paper—criminals are deterred from robberies into property crimes. Of course, Ayres and Donohue found that robberies went up as well, but that could be explained by some criminals being deterred and switching to property crimes, while others decide to carry guns, leading to an increase in impulsive robberies that more than makes up for the decrease caused by some robber switching to property crime. [Update: Kleiman points out that the increase in the number of property crimes is too large to be explained by any plausible decrease in robberies, so this mechanism doesn’t work.]

Now, while this mechanism is possible, I don’t think that the carry laws caused the crime increase. My reason is the same as the reason why, before Lott’s results were refuted, I didn’t think that carry laws caused a crime decrease. The carry laws did not cause any significant increase in gun ownership. Nor were there significantly more guns carried in public places. And nor were there more guns used to defend against crime. There may be a very small number of crimes committed by permit holders, and a very small number of criminals might take up gun carrying after encountering a permit holder, but the crime increases this would cause would be smaller than those observed by Ayres and Donohue. So it is possible that the carry laws increased crime, but I would expect the increase to be very small.

However, some people have ignored or dismissed these issues and citing Lott’s work, advocated passing such laws to reduce crime. To be consistent, such people should now advocate repealing the carry laws to reduce crime. To my knowledge, not one carry law advocate has done this.

Glenn Reynolds comments on Kleiman’s post, stating than in his area the carry law made little difference in gun carrying.

Brian Linse responds to William Sjostrom’s attempt to defend Lott.

Three recent papers that contradict Lott’s “More Guns, Less Crime” theory:

Duwe, Kovandzic and Moody, “The Impact of Right-to-Carry Concealed Firearm Laws on Mass Public Shootings” Homicide Studies Journal, 6:4 pp 271-296 (2002). Duwe et al find no statistically significant impact of carry laws on mass public shootings, contradicting Lott’s claims in his new book and an earlier paper with Landes. (Lott does not even mention this paper in his book.) Even when they tried to replicate Lott’s results they could not find a significant effect.

Helland and Tabarrok in Using Placebo Laws to Test “More Guns, Less Crime” : A Note tested the theory by comparing Lott’s results with regressions run where instead of the actual carry laws being used, placebo laws were assigned to random states. This gives a better idea on whether Lott’s results could have arisen by chance than the standard tests. Carry laws were found not to have had a statistically significant on crime rates. The only thing that could not be explained by chance is the combination of a decrease in violent crime and an increase in property crime, but as Ayres and Donohue point out, the decrease in robbery is far too small to explain (by substitution) even a small part of the increase in property crime.

Kovandzic and Marvell in Right-to-carry Concealed Handguns and Violent Crime: Crime Control Through Gun Decontrol? looked at how the number of concealed-carry permits in each county in Florida related to changes in crime. No significant effects were found and there were about an equal number of increases and decreases.

The Journalist’s Guide to Gun Policy Scholars and Second Amendment Scholars is a directory of pro-gun scholars. It is grouped into sections by specialty. So who are the experts with special expertise in Women and Gun Issues?

Women and Gun Issues
Dr. John Lott	American Enterprise Institute
Dr. Helen Smith	Southeastern Psych. Servs.
Prof. Mary Zeiss Stange	Skidmore College
Prof. Carol Oyster	U. of Wisconsin Psych.

So, apart from pretending to be one, what expertise does Lott have on women and gun issues? Well, he wrote this NRO article on women and guns. It was widely linked by bloggers, who felt that the key statistic was this:

“The probability of serious injury from a criminal confrontation is 2.5 times greater for women offering no resistance than resisting with a gun.”

Lott makes the same claim in More Guns, Less Crime, in The Bias Against Guns and in op-eds and speeches and on radio and TV shows. Along with the “98% brandishing” it is one of his favourite statistics. It shares another characteristic with the “98% brandishing”—no serious researcher in the area advances the figure and it does not appear in the peer reviewed literature.

So where does it come from? More Guns, Less Crime (published in 1998) gives the source as

Lawrence Southwick, Jr “Self-Defense with Guns: The Consequences,” Managerial and Decision Economics (forthcoming), tables 5 and 6;

The Bias Against Guns, published in 2003, gives the same source. In fact, it was published in the Journal of Criminal Justice, 28(5):351-370, 2000, and table 6 gives the following number of incidents in the National Crime Victimization Survey (NCVS) involving female crime victims:

	Used gun	No action
Injured	1	228
Not Injured	79	7163

1.25% of the gun defenders were injured, while 3.08% of the passive women were, 2.5 times as much. So far so good. However, what should be obvious to someone skilled in statistics is that the ratio is not statistically significant. You can check this yourself if you wish by using an online calculator for the Fisher exact test. It gives a p value of 0.5, meaning that if there was really no difference in the injury rates, 50% of the time you would get a difference as large as this from a random sample.

Note that although Lott takes this result from Southwick’s table, Southwick does not report it, since he is careful to report only the results that are statistically significant. In fact, Southwick notes that the difference between men and women in his table 6 is not statistically significant. Lott, however, writes:

Men also benefit, but the benefit is smaller because there is, on average, a smaller difference in strength between violent criminals, who are almost always men, and male victims than for female victims. For men, passive behavior is 1.4 times more likely to result in serious injury than resisting with a gun.

The “1.4 times” difference is not statistically significant either.

If you check the American Statistical Association’s Ethical Guidelines for Statistical Practice you will find the following, which Lott did not follow:

Report the limits of statistical inference of the study and possible sources of error.

Even newspaper reports typically report margins of error for surveys, which Lott did not do.

Nor can Lott claim that he mistakenly thought that the results were statistically significant. After a Usenet discussion I had with David Friedman in 2001, he pointed out to Lott that it was wrong to make the “2.5 times” claim. This did not stop Lott from repeating the claim in his NRO article and in his new book. (Buried in an endnote in his new book he admits that the difference is not statistically significant, but then falsely claims that the difference is statistically significant if men and women are pooled together.)

As well as making the highly misleading claim that women are 2.5 times as likely to be injured if they offer no resistance than resisting with a gun as I discussed yesterday, in The Bias Against Guns Lott claims (page 99):

Carrying a gun is also the safest course of action when one is confronted by a criminal.⁶

Endnote 6 states:

Gary Kleck and Don Kates (288-290) present the most recent data from the Department of Justice’s National Crime Victimization Survey from 1992 to 1998 and also indicate that the risk of serious injury from a criminal attack is lowest when one resists a criminal confrontation with a gun.

However, what we actually find on page 291 of Armed: New Perspectives on Gun Control by Kleck and Kates is (my emphasis):

Consequently, while defensive gun use is generally safe, it does not appear to be as uniquely safe among self-protection methods as data from earlier NCVS data suggested.

Lott says that Kleck says the opposite of what he actually says.

The reason why the newer NCVS data changed Kleck’s conclusion is that the NCVS starting asking when injuries occurred—whether people were injured before or after their self-protective actions. Obviously, injuries that happened before a defender took action could not properly be attributed to that action. Here’s an extract from Kleck’s Table 7.1:

	Robbery		Assault		Burglary
		Post-SP		Post-SP		Post-SP
SP measure	Injury	Injury	Injury	Injury	Injury	Injury
Any SP with gun	13%	8%	28%	4%	10%	2%
All incidents	30%	5%	57%	6%	14%	2%

You can see that although the overall injury rates are lower when a gun is used for defence, the post-self-protection injury rates are about the same, so the lower overall injury rates cannot be properly credited to the gun use.

Note that the circumstances of gun defenders might be different from other defenders, so after controlling for circumstances it might turn out that guns are more effective or it might turn out that guns are less effective at preventing injury. We really don’t know which, but at this stage the best guess would be that they aren’t any better or worse than other methods.

For an example of a pro-gun writer properly describing Kleck’s results see GunCite.

[Note: This is a copy of a document found at this link on John Lott’s website on May 13, 2003. I have added critical commentry, written in italics like this.
Tim Lambert ]

With some recent attacks on me in a variety of places from the Washington Post to the Chicago Tribune to numerous other places, I thought that I should send out some responses for those who might be interested. (more…)

Jeff Johnson of CNSNews.com writes a very pro-Lott piece on the dispute between Lott and Ayres and Donohue. Probably the most notable feature is what is not mentioned—there is nothing about the coding errors Lott made. We can be sure that Donohue mentioned the problem to Johnson, but Lott had nothing to say on the matter and Johnson chose not to mention it. [Update: I checked with Donohue and he told me that he didn’t get to mentioning the coding errors. My mistake. It is still true that Lott did not use this chance to dispute the allegation of coding errors.]

Anyway, Johnson writes:

The pair [Ayres and Donohue] examined monthly, rather than annual, crime data and operated under the assumption that - if passage of concealed carry laws truly reduced crime - there would be a “straight line drop” in crime rates from the date of enactment forward. When the crime rate dropped slower than this assumption predicted it should, Donohue and Ayres referred to the difference as an “increase” in crime.

This is wrong from beginning to end. Firstly, they do examine annual data. Secondly, they considered and reported dozens of different specifications, only one of which was the “straight line drop”. Thirdly, a slower than expected drop would just make the “straight line drop” smaller, it would not show up as an increase.

“It’s only when they use this kind of ‘artificial specification’ that simplifies this do they get a bad result,” Lott explained. “A better way of doing it is by looking at the crime rates year by year, for one year after the law, two years, three years - and when you do that, even their own results get an immediate drop that continues to fall after that.”

This is just another variation of Lott’s claim that “Ayres and Donohue have simply misread their own results.” This is completely untrue, as I explained earlier. Lott is pulling one graph out of the middle of their paper and pretending that this graph is all of their results.

Donohue and Ayres also use varying definitions of “crime” in their analysis of Lott’s research.

“Lott claimed that the 10 states that enacted shall-issue laws between 1985 and 1991 experienced declines in murder and other violent crimes relative to the crime trends observed in other states that did not pass shall-issue laws,” Donohue wrote in a press release promoting the book. “In contrast, Donohue contends that the 13 states that enacted shall-issue laws after 1992 experienced relative increases in crime.” [Emphases added.]

Ayres and Donohue do not change their definition of “crime”. It is perfectly obvious if you look at any of their tables that they considered each of the UCR crime categories separately, as well as the aggregate categories of violent crime and property crime.

But Lott never argued that all crime was reduced by passage of concealed carry laws, only violent crime.

“That’s the finding that people have seen all along,” Lott told CNSNews.com. “You have some people who were engaging in robbery in order to get money previously and, when people are able to carry concealed handguns to protect themselves, you have some criminals [who] stop committing crimes, but some switch into other crimes.”

Most often, Lott said, that switch is from robbery, where criminals come into direct contact with their victims and face a newfound risk of getting shot, to burglary and property theft “because it’s relatively less risky.”

Ayres and Donohue’s most general results (shown in table 13 of their paper) show increases in robbery in more states than decreases.

Those crimes are possibly committed with greater frequency because they are also less lucrative than robbery, explaining the increase in overall crime committed while experiencing a decrease in violent crime.

Except that there was an increase in violent crime.

“I think the thing to do is just put it in context of all the other people who have looked at [my work],” Lott said. “Nobody has found a bad effect except for this one section of [Donohue and Ayres’] paper, and even then, it’s just a temporary one.

“Everybody, including this paper, finds that the crime rate falls the longer the laws are in effect,” he continued. “I think that’s pretty strong evidence.”

This is also not true. Lott is pulling one graph out of Ayres and Donohue’s 120 page paper and pretending that that one graph is the whole paper.

Glenn Reynolds comments on the CNSNews article. Despite Ayres and Donohue’s best efforts, Reynolds is all agnostic on the Lott question, but fortunately he has an opinion on the study by Ludwig and Cook (who Reynolds calls “antigun researchers”):

What’s most striking to me, though, is another study, by antigun researchers, that tries to measure gun ownership by suicide rates. (And it’s not mentioned here, but I believe there was another that tried to use subscriptions to gun magazines as a proxy.) This seems rather bogus to me, and I can only imagine the general derision if this kind of proxy were employed by researchers whose work supported gun ownership.

Why imagine? Kleck used the percentage of suicides with guns as a proxy for gun ownership (see J Quant Crim 9:249-88). There doesn’t seem to have been any derision.

While people throw stones at Lott, whether deservedly or not, it’s worth remembering that the anti-gun side has been throwing out utter bilge disguised as “research” for years without a peep from the usual guardians of scientific rigor.

I asked Reynolds to explain what “anti-gun” bilge he was referring to, and he said was thinking primarily of Kellermann. Let’s see how accurate his criticism of Kellermann is:

In their famous “rabidly antigun” article Kopel and Reynolds assert that a 1993 study by Kellermann et al that found that gun ownership was associated with a three times greater risk of becoming a victim of a homicide in the home was “junk science”. They argue that

hardly any of Kellermann’s murder victims were killed with a gun from their own home, and a significant number of the murder “victims” were lawfully killed by police, and the whole factoid disappears once you account for the true rates of gun ownership among the “control group” of people who weren’t murdered

All of these claims are false. Enough of the victims were murdered by the gun in the home so that all of the extra risk from guns was associated just with gun homicide and homicide by people with access to the home gun. Only four (out of 400) were killed by police. You can speculate that the correlation may have be caused by gun ownership being more underreported in the control group, but that does not make the speculation true.

Kellermann’s paper has been subject to many erroneous criticisms. This article by Steve Kangas corrects many of them.

In Lott’s appearance on KSFO he talked about the Appalachian Law School shooting and described the two armed off-duty police officers who apprehended the shooter as “two students, one with a former law enforcement background”. Lott knows full well (as this thread demonstrates) that both of them were current police officers. And they didn’t stop the shooting—the shooter had run out of ammunition.

Lott also claimed that British gun control in the 20s, 50s and 90s was followed by an increase in crime in each case. I have data for homicide from 1857 to 1993 and violent crime in the 90s. There was a modest decrease in the 20s, no apparent change in the 50s and a marked decline in the 90s. I’m not saying that gun control should get credit, but the results were pretty well the opposite of what Lott claimed.

Tom Spencer thinks that Reynolds is “making wild evidence-free claims about the nature of other scholarship in the field just to distract you from the trouble his man Lott is in.” It is certainly odd that Reynolds doesn’t provide any evidence that the research he criticizes is “utter bilge” while insisting that criticism of Lott is

“best done by those who … have done actual work, and have actual evidence relevant to the matter at hand.

Lott has an op-ed in The Plain Dealer where he continues to mislead:

My new book, “The Bias Against Guns,” examines multiple-victim public shootings in the United States from 1977 to 1999 and finds that when states passed right-to-carry laws, these attacks fell by 60 percent. Deaths and injuries from multiple-victim public shootings fell on average by 78 percent.

Lott does not mention here, or in his new book, this paper: Duwe, Kovandzic and Moody, “The Impact of Right-to-Carry Concealed Firearm Laws on Mass Public Shootings” Homicide Studies Journal, 6:4 pp 271-296 (2002). Duwe et al find no statistically significant impact of carry laws on mass public shootings. Even when they tried to replicate Lott’s results they could not find a significant effect.

People’s reaction to the horrific events displayed on TV is understandable, but the more than 2 million times each year that Americans use guns defensively are never discussed - even though this is five times as often as the 450,000 times that guns are used to commit crimes over the last couple of years.

Lott is cherry picking numbers from different surveys. Surveys that have measured both offensive and defensive gun uses (NCVS and Hemenway’s survey) find that offensive gun use is much more common. Otis Dudley Duncan’s paper “As Compared to What?” has much more on the comparisons between offensive and defensive uses.

Annual surveys of crime victims in the United States continually show that, when confronted by a criminal, people are safest if they have a gun.

The surveys do not show that.

Studies, such as one conducted recently by Jeff Miron at Boston University, which examined 44 countries, find that stricter gun control laws tend to lead to higher homicide rates.

The measure of gun control used in Miron’s study was a crude three point scale: No gun laws/Some controls/Complete ban. By this measure the US and the UK have exactly the same amount of gun control. Miron’s study provides no information about whether the US or the UK’s gun control regimes are preferable. And a correlation between gun control and homicide is also explained if more homicides lead to gun control laws.

ArchPundit has a thoughtful analysis of the latest from Reynolds and Lott.

Shorter dsquared: If you use some data to construct a model, then to test it properly you need new data. Lott’s approach is a little different. The model that was given greatest prominence in the original Lott and Mustard paper (the results of this model where given in the abstract and whenever Lott summarized his paper) showed that there was a 3.5% decline in the violent crime rate associated with the carry laws. In the second edition of More Guns, Less Crime Lott has a chapter where he gives the results he obtained by adding four more years of data. Except that he uses a different model. Nowhere does he present the results you get with the original model. Why not? A clue to the mystery appears in Ayres and Donohue’s paper. Table 10 shows that with five extra years of data, Lott’s original model indicates that the change in the violent crime rate associated with the carry laws was 0.0%.

Jim Henley reflects on the British crime statistics I mentioned a couple of days ago. He is concerned that I am too trusting of the British Crime Survey. As far as I can tell they have used standard practices for victimization surveys and since many people are involved I don’t see how the data could be cooked without someone finding out. I would also expect victim surveys to be more accurate than police reports because the police are directly responsible for controlling crime, while the statisticians and surveyors who run the BCS do not.

Tom Spencer thinks that Lott’s days are numbered now.

Jim Henley is still concerned about the possibility that the BCS figures might be cooked. I still consider this possibility extremely unlikely because it would require some sort of conspiracy between the statisticians (who don’t have an interest in crime figures showing some particular result) and the top level people (who do have such an interest). With this many people involved, it would be quite likely that someone who blow the whistle and a major scandal would result. Also, if they were cooking the figures, they would have cooked the police statistics as well. (Recall that these show an increase.)

He also concerned about the explanation the report gives for the increase in police recorded crime. My understanding of their explanation is that the abrupt jump was caused when new offence categories were added to the definition of violent crime, and the increase since then was because of the gradual adoption of new counting rules. Chapter 3 of the report has more details for this, as well as a table showing results once the change in recording has been allowed for.

On pages 36-37 of The Bias Against Guns Lott attacks Tom Smith:

A few years ago, while I was doing research at the University of Chicago,I had lunch with Tom Smith, who is the director of the General Social Survey at the National Opinion Research Center (NORC).  This private organization conducts many important national surveys for the government as well as other clients. During lunch Tom mentioned how important he thought the General Social Survey was.  He felt the large drop in gun ownership implied by his survey would “make it easier for politicians to do the right thing on guns” and pass more restrictive regulations. His surveys have traditionally shown one of the lowest gun ownership rates among any of the surveys: for example, almost 20 percentage points lower than    recent polling John Zogby. After Tom made his comment about    politicians, I didn’t ask him whether he had deliberately phrased his questions in such a manner to obtain an artificially low gun ownership rate. But the question certainly crossed my    mind. Possibly Tom is still right and Zogby and others are wrong.

1. So Lott believes that if a survey gives a result that differs greatly from other surveys and in a direction that favours the views of the designer it is reasonable to suspect that that the survey was contrived to give that result.
2. Lott seems to think that the GSS is run in the same haphazard fashion as his own survey, with Smith writing the questions himself without consulting any experts. In fact, Smith did not write the GSS questions on gun ownership—they date back to 1973, before Smith was associated with the GSS.
3. Tom Smith did not tell Lott that his survey would “make it easier for politicians to do the right thing on guns”. I asked him about it and he replied:

   I have no public views on gun control and am in no way pro or anti-gun nor pro or anti-gun control. To the extent that his statement implies that I favor gun control or have any position on changes in public policy regarding guns, it misrepresents my position. It possible that I stated as a fact that lower levels of gun ownership would strengthen the pro-gun control political position, but I did not and do not either favor or oppose such a development.

4. The GSS does not give a household gun ownership rate 20 points lower than other surveys.  The rate is actually much closer (Zogby 46%, 2002 GSS 36.5%). Lott has confused personal gun ownership rates with household gun ownership rates. The GSS also asks if the gun is personally owned, and the 2002 GSS found that 26% of people personally owned a gun. The Zogby poll asked “Do you or anyone in your home own a gun” (my emphasis). Lott ignored the “or anyone in your home” part of the question. On page 83 of The Bias Against Guns he multiplies the 46% household gun ownership from the Zogby poll by the number of adults in the US to come up with an erroneous estimate of 94 million gun owners in the US. The GSS is only 20 percentage points lower because Lott has mistakenly compared household ownership from Zogby with personal ownership from the GSS.

So, in one paragraph Lott managed to misrepresent Smith’s position, misunderstand the survey data in question, mistake the author of the GSS questions and mount a personal attack on Smith.

Jeff Johnson of CNSNews.com describes an AEI event to publicise The Bias Against Guns. Lott repeats his version of the Appalachian Law School shootings, as usual not mentioning that the shooting stopped because the shooter ran out of ammunition and not mentioning that the armed students were off-duty police officers.

Lott also atttacked University of Chicago professor Mark Duggan who published a paper, “More Guns, More Crime” in the Journal of Political Economy (CIX p 1086-1114):

He pointed to a recent paper that used subscription to the third-most popular gun magazine in the U.S. as a measure of gun ownership. When subscription rates for the most popular and second-most popular magazines were used instead, the findings of the research were altered dramatically.

“If I was a referee, I would ask, Why only look at one magazine here? Why not the largest or the fifth largest?” Lott said. “The fact that it had not would make me pretty suspicious and unlikely to go ahead and publish the paper.”

Lott insinuates that Duggan cherry-picked his data to contrive his result, but in his paper Duggan explained his choice of magazine:

“In contrast to the three gun magazines with greater circulation (American Rifleman, American Hunter, and North American Hunter), sales data for this magazine are available annually at both the state and the county levels. More important, Guns & Ammo is focused relatively more on handguns than these other three magazines. Because handguns are the weapon of choice in the vast majority of firearms-related crimes and are more likely to be purchased for self-defense purposes than rifles or shotguns, this magazine is a more appropriate one for analyzing the dynamic realtionship between crime and gun ownership.”

Also at the AEI Event, Carl Moody criticized Lott for bypassing the normal peer review process in publishing results in his book, but said that this was more than compensated because:

“He makes the data available, which means he is probably not cheating. I’ve checked him out; he’s not cheating, and he uses all the requisite controls. “He does it right, and so, I tend to believe the results that John has published in the back of the book”

However, because he made the data available, Ayres and Donohue were able to find that Lott’s results were the product of systematic coding errors. These results were published in the back of the book and Lott is still ducking discussion of the errors.

Tom Spencer comments on Lott’s attack on Tom Smith. John Quiggin writes about data mining and Lott.

Brian Linse writes:

The fact that former supporters (and some current supporters like NRO) haven’t been outspoken and firm in their denunciations of Lott’s irresponsible and unethical behavior is destined to do additional harm to the cause of gun rights.

Posts by d-squared and John Quiggin on data mining and Lott reminded me that Lott accused his critics of data mining in a response to Webster:

The Black and Nagin paper excludes Florida after they have already excluded the 86 percent of the counties with populations fewer than 100,000. Eliminating Florida as well as counties with fewer than 100,0000 does eliminate the significance in the one particular type of specification that they report for a couple of crimes, but the vast majority of estimates were unaffected from this extreme data mining and they ignore that doing this actually strengthens some of the results.

and in a Reason interview:

I wanted all the data that were available….I didn’t pick and choose, and when somebody drops out 86 percent of the counties along with Florida, you know they must have tried all sorts of combinations. This wasn’t the first obvious combination that sprang to mind. And it’s the only combination they report….If, after doing all these gymnastics, and recording only one type of specification, dealing with before-and-after averages that are biased against finding a benefit, they still find only benefits, and no cost, to me that strengthens the results.

So, how accurate is Lott’s claim that Black and Nagin were doing “extreme data mining”?

Well, Lott’s comment about dropping 86% of the counties is a red herring. Black and Nagin also got similar results if they included the small counties and just dropped Florida. They wrote:

“Nor is this result a function of our use of the large-county sample. Without Florida in the sample, the estimation of Lott and Mustard’s model, which is given by equation (1), for all counties provides no evidence of an impact of RTC laws on homicide and rape.”

Even if Lott failed to notice this sentence, he must have known that dropping the small counties didn’t matter, since he said he reran all the regressions without Florida. Lott’s accusation of “extreme data mining” was deliberately misleading.

Lott’s 86% figure is also misleading. Maltz and Targonski noted problems in the county crime data that Lott used, with about 13% of counties having significant under-reporting. In Lott’s reply, he argued out the regressions were weighted by population, so the size of the problem was best measured by the percentage of the population in the problem counties, which was only 6.8%. And yet when he criticized Black and Nagin he used the percentage of counties that they dropped (86%), rather than the percentage of population in those counties (about 30%).

Is it data mining to check to see if the results depend on the inclusion of a particular state? No, that is a legitimate test of the robustness of the result. It does not prove that the carry laws did not reduce crime, but it strongly suggests that something is wrong with the model.

Lott has written an op-ed where he criticizes the New York Times for a “pattern of deceit”.

Metafilter has a thread on Mary Rosh, while David Schimke also comments.

In chapter 7 of The Bias Against Guns, where Lott argues that “safe storage” laws cause increases in violent crime, he quotes from an op-ed:

Jessica Lynne Carpenter is 14 years old. She knows how to shoot … Under the new “safe storage” laws being enacted in California and elsewhere, parents can be held criminally liable unless they lock up their guns when their children are home alone … so that’s just what law-abiding parents John and Tephanie Carpenter had done…. [The killer], who was armed with a pitchfork … had apparently cut the phone lines. So when he forced his way into the house and began stabbing the younger children in their beds, Jessica’s attempts to dial 9-1-1 didn’t do much good. Next, the sensible girl ran for where the family guns were stored. But they were locked up tight…. The children’s great-uncle, the Rev. John Hilton, told reporters: “If only (Jessica) had a gun available to her, she could have stopped the whole thing. If she had been properly armed, she could have stopped him in his tracks.” Maybe John William and Ashley would still be alive, Jessica’s uncle said.

David Friedman summarizes Lott’s account of this story given in one of his talks: (You can hear the story at 32:50 in this Real Audio recording of another Lott talk.)

But the most interesting thing in the talk was information, not about firearms but about reporting bias. There was a news story a few months back from (I think) Merced California, about someone who broke into a house containing five children, killed two (with a pitchfork) and wounded two others. Apparently the original news story from the local paper, carried by the wire service, included the fact that while he was breaking in the eldest child, a fourteen year old girl with experience in target shooting, went to her parents’ bedroom, got out their handgun–and was unable to use it because of the trigger lock that her father had put on in obedience to a recent state law.

The interesting point was that, according to John, that part of the story was cut out by every newspaper in the state, aside from the Fresno Bee (I think) which is where the original appeared.”

Later, however, Friedman discovered that Lott’s story was not true and corrected it in this Usenet posting:

It’s a good story, but as far as I have been able to determine it isn’t true. After I heard it , I tried to locate the news stories. As far as I can tell, the account my source had given (in a public lecture) and had gotten from someone else confused two different news stories.

The original story on the shooting had nothing about the girl trying to get at her parents’ handgun, and it sounded from the sequence of events as though that would have been impractical. A later story, based on an interview with a relative, put some of the blame on gun control laws, I think specifically safe storage laws. So the other newspapers were not cutting out information from the original story–merely repeating what the original story said without adding anything from the later story. And what was in the later story was a lot less damning than in the account I heard.

Now, when Lott first presented the false story a couple of years ago, he might have just been very careless with his facts and honestly believed it to be true, but he must know by now that it is untrue. Why did he present it again in The Bias Against Guns?

Tom Spencer finds Lott’s complaints about a “pattern of deceit” at theNew York Times rather hilarious.

Dr Limerick thinks Lott isn’t washed up, because the AEI and similar institutions will always have a place for him.

Andrew Chamberlain tells us why character matters:

By Lott’s total disregard for norms of honesty, he’s revealed a deep character flaw that runs to his core. And people always become what they have in their core.

I keep having conversations where people argue the Mary Rosh scandal has no impact on Lott’s scholarship. This is absurd. The scandal demonstrates that Lott is a man with no integrity or respect for the truth. Like it or not, this casts doubt on all his work, since all scholarship depends fundamentally on a tacit trust between author and audience.

Because intellectual resources are scarce, not every datum in papers can be justified, or every claim checked for accuracy. We have to trust scholars to some extent. And anyone who trusts John Lott after the Mary Rosh debacle is a damn fool. Or in the case of many conservatives, a blind ideologue.

I found a copy of the Fresno Bee story that Lott claimed

“included the fact that while he was breaking in the eldest child, a fourteen year old girl with experience in target shooting, went to her parents’ bedroom, got out their handgun—and was unable to use it because of the trigger lock that her father had put on in obedience to a recent state law.”

Of course it doesn’t include that “fact”, which would appear to be a fabrication.

Jan Haugland writes about how blogs have been bad news for people named Lott:

Even if Lott is not an outright fraud (which the evidence so far suggests he is), he is certainly not a scholar, and he is a using unethical methods and sloppy research.

Also commenting on blogs and Lott are Greg Vassie and Dan Gillmor.

Also in the Fox News article we have:

But Lott counters that the number of gun accidents among law-abiding citizens is remarkably low given that about 90 million Americans own firearms.

As I explained earlier, Lott’s estimate of the number of firearm owners is much too high because he misunderstood the question in the poll that he used for his estimate.

Lott has started a blog and responded to the questions I raised about his claims about the Merced pitchfork murders:

Fox News interviewed the father of the dead children and reported the following:

“Lott cited a Merced, Calif. family whose guns were put away because of the state’s safe storage law. John Carpenter, who lost two children in an attack in 2000, said a gun would have stopped the man who broke into his home with a pitchfork. ‘If a gun had been here, today I’d have at least a daughter alive,’ Carpenter said.”

It doesn’t appear that Fox News interviewed the father at all, but rather that they interviewed Lott and Lott claimed that the father said that. This quote also appears to be a fabrication. None of the newspaper reports written at the time of the murders has this quote and it doesn’t make sense—three of his daughters survived, while the quote makes it sound like all his daughters were murdered. [Update: Lott has posted a Fox transcript that shows the quote is genuine.] [Further update: Kevin P interpreted the above as an accusation that Lott fabricated that quote. That was not my intent. I was suggesting that he was repeating another fabrication, not that he was the source. I apologize for my careless wording.]

It is also rather odd for Lott to cite himself to support one of his claims. Anyway, Lott goes on to write:

On top of this I appeared on a radio show with Rev. John Hilton, children’s great-uncle, who repeated what he said in the quote used by Suprynowicz. In addition, there were two articles that were published in a local California newspaper that discussed this case.

In the newspaper article that Lott refers to, Hilton does argue that if Jessica had had a gun she could have stopped the murderer:

Their father, John Carpenter, kept a gun in the home. His children had learned how to fire it. But he kept it locked away and hidden from his children.

“He’s more afraid of the law than of somebody coming in for his family,” Hilton said. “He’s scared to death of leaving the gun where the kids could get it because he’s afraid of the law. He’s scared to teach his children to defend themselves.”

However, this is just another example of the sort of tactics that Lott has used to attempt to defend himself on the question of the missing survey—there, since he can’t provide evidence that he conducted a survey in 1997, he keeps providing evidence that he had a hard disk crash, even though that is not being questioned. Here, he tries to make it seem that the Hilton quote is being questioned when what is being questioned is Suprynowicz’s claim (quoted by Lott in The Bias Against Guns) that:

the sensible girl ran for where the family guns were stored. But they were locked up tight.

and Lott’s claim that the Fresno Bee story

“included the fact that while he was breaking in the eldest child, a fourteen year old girl with experience in target shooting, went to her parents’ bedroom, got out their handgun–and was unable to use it because of the trigger lock that her father had put on in obedience to a recent state law.”

and his claim that

that part of the story was cut out by every newspaper in the state

All these claims are false. The Fresno Bee story is quite clear—after she found that the phone in her bedroom was dead, Jessica climbed out of her bedroom window to go and get help—she didn’t go and get her parent’s locked handgun. Instead of admitting to his mistakes, Lott has once more tried misdirection.

The Fox News story that Lott cites contains some other falsehoods:

For several years, gun control advocates have been quoting a study that reached a very different conclusion. University of Washington doctors claimed that in a dozen states which had safe storage laws, 39 children’s lives were saved.

But the study has been widely discredited because the researchers never factored in that accidental gun deaths have been falling everywhere for decades.

The author of the article does not say where he got the claim that the study was “widely discredited” because they didn’t factor in national trends, but presumably it was Lott since Lott says something similar on page 313 of The Bias Against Guns:

The Cummings et al. (1997) research provides evidence of a 23% drop in juvenile accidental gun deaths after the passage of safe storage laws. Juvenile accidental gun deaths did decline after the passage of the law, but what Cummings et al. miss is that these accidental deaths declined even faster in the states without these laws. While the Cummings et al. piece examined national data, they did not use fixed year effects which would have allowed them to test whether the safe storage states were experiencing a drop relative to the rest of the country.

However, the study did in fact control for national trends and it did use fixed year effects and it did test and find that the safe storage states were experiencing a drop relative to the rest of the country. From the study:

To control for national trends over time in firearm mortality rates, all states were included in the analysis, and 15 indicator variables were used to represent each calendar year. Categories of age, sex, and race were examined as potential confounders.

The Washington Post zings Lott for throwing stones at the New York Times from his glass house. Matt Welch also comments, while Greg Beato thinks that the New York Times has hit rock bottom when even John Lott is denigrating its integrity.

Andrew Chamberlain invites readers to join in an online debate about Lott and scholarly integrity.

Tom Spencer comments on the errors that Lott made on his blog.

Lott has a new entry on his blog where he posts a transcript from Fox News that apparently has the father of the murdered children saying:

“If a gun would have been here today, I’d have at least a daughter alive.”

I was mistaken when I suggested that the quote was a fabrication. The quote implies that all his daughters were murdered when they weren’t, but perhaps it didn’t imply this in its original context.

In any case Lott has not responded to the rest of my post, instead concentrating on one minor point. In two blog postings now Lott has neither supported nor admitted errors in the claims that

1. “the sensible girl ran for where the family guns were stored. But they were locked up tight.”,
2. that she was unable to use her parents gun because it had a trigger lock on it,
3. that every newspaper in the state suppressed this fact,
4. that the Cummings paper did not use fixed year effects

Unfortunately, this behaviour is not new. He still has not responded to the serious charges that on two occasions his data contained systematic coding errors that happened to support his thesis.

Tom Spencer mentions the Washington Post’s criticism of Lott. ArchPundit finds Lott’s criticism (in this interview)of poorly done gun control research ironic. Arie discovers the John Lott story.

Lott has blogged for the third time about the Merced murders:

Taken together, the different articles in these various posts indicate that the gun was locked; it was placed in a way that was not accessible by the children; both the father and the great-uncle, the Rev. John Hilton, believed that if the gun had been accessible children’s lives would have been saved; and these moves were done because of fear of the California state law.

And for the third time he has neither supported nor admitted as false this claim from his new book:

“the sensible girl ran for where the family guns were stored. But they were locked up tight.”

This is not some minor quibble. Lott’s account makes it sound like the only thing stopping her from using her parents’ gun was that it was locked up. In fact, she locked herself in her bedroom and tried calling 911. The killer banged on her door. If, for the sake of argument, we assume that the gun was unlocked and she had come out the door and tried to get the gun, there would have been a fair chance that she would have been pitchforked to death instead of climbing out her window and getting help (which is what actually occured).

Lott’s false version of the story avoids this problem by having her get to where the guns are stored and being stymied by the gun lock.

In Lott’s latest entry he has given up trying to support his claim that

“the sensible girl ran for where the family guns were stored. But they were locked up tight.”

and responded to this post, where I pointed out that Cummings et al clearly stated that they controlled for national trends, but Lott none the less dismisses their results, claiming that they did not control for national trends. Lott now writes:

We had been unable to replicate their claimed results using fixed effects and the only way we could get something similar was without fixed effects. It really shouldn’t have been that difficult for us to confirm what they found since we were used their dates for the laws. Unfortunately, Cummings, Grossman, Rivara, and Koepsell were unwilling to give us their data when we asked for it. I asked for the data from Cummings and one other coauthor. Possibly we should have made a big deal of yet more academics who refused to share their data, but we decided that the more straightforward approach would be to simply say what we found. Alternatively, we could have simply stated that we were unable to confirm their results.

Yes, it would have been better to state something that was true instead of claiming that Cummings et al did not use fixed effects when they clearly stated that they did

Lott has knowingly misled the readers of his book. Someone who reads his book without carefully crosschecking it with Cummings et al is left with the impression that Lott’s work trumps Cummings because Cummings did not control for national trends. If Lott had instead made true statements about Cummings’ paper, readers would be aware that researchers using similar data and methodology had gotten different results and that the issue was in dispute.

David Glenn’s article on academic blogging in the Chronicle of Higher Education mentions the role of blogs in the investigation of Lott’s conduct.

William Sjostrom writes “Lott always releases his data.” But Lott has not released the data for his 1997 survey.

Several people have commented on the irony of Lott attacking the New York Times for a “Pattern of Deceit”, but let’s look at what he says in his article:

As an example, take the major 20,000 word series on “rampage killings” the Times published during 2000.

The paper declared that the evidence they compiled “confirmed the public perception that they appear to be increasing.” Indeed, the Times found that exactly 100 such attacks took place during the 50 years from 1949 to 1999, 51 of which occurred after the beginning of 1995. Their conclusion: “the nation needs tighter gun laws for everyone.”

Observed a Flaw

Having done a lot of work on this topic (together with Bill Landes at the University of Chicago), I immediately noticed that the Times noted virtually all the cases during the second half of the 1990s, but omitted most of the cases prior to that.

While a side bar to one of the articles briefly cautions that the series “does not include every attack,” the omissions are so extremely skewed as to produce a nine-fold increase between the 1949 to 1994 and 1995 to 1999 periods.

The Times claimed that from 1977 to 1994 there was an annual average of only 2.6 attacks where at least one person was killed in a public multiple victim attack (not including robberies or political killings). Yet, what we found was an average of 17 per year.

However if you look at what the Times actually said (payment required), you find that they did not say that there were “exactly 100 attacks”, but rather

the database does not include every attack of this type over the last 50 years.

They did not say that there was a nine-fold increase in attacks and they did not say that “from 1977 to 1994 there was an annual average of only 2.6 attacks”, but rather

Yet there is a strong impression that they have become more common. In an effort to confirm the trend, The Times analyzed F.B.I. reports of all homicides since 1976. Each year there were 15,000 to 22,000 homicides, but very few involved three or more victims.

That universe shrank even more, to just a few dozen, when The Times weeded out those involving robbery or gang violence, and those in which the primary victim was a family member.

What is left is the closest thing there is to a census of rampage killings — about one-tenth of one percent of all killings.

And it shows that in the 1990’s, they increased.

Their number remained fairly consistent from 1976 to 1989, averaging about 23 a year, only once going above 30. But between 1990 and 1997, the last year for which data was available, the number averaged over 34, dipping below 30 only once, in 1994.

‘’In the early 90’s, for some reason, it increased, and seems to have a different level since,'’ said Steven Messner, a criminologist at the State University of New York at Albany, who reviewed the numbers at the request of The Times.

The New York Times article considered two different data sets. The one they used for trends as in the quote above was based on FBI data on all homicides. The other contained 100 cases where detailed information was obtained from media reports, and did not contain all cases of rampage killings (as they made clear). The cases that were in it were skewed towards the more recent ones. Lott pretended that they used the second data set to analyse trends even though it is quite obvious from the article that they did not.