Comments on Questions
About John R. Lott’s Claims Regarding a
1997 Survey
Personal Note,
In September 2002 I offered to look into a question raised
about John Lott’s work. I thought that
offering such help to Lott and to the profession was the responsible thing to
do when serious questions were raised, and I thought it would be exceedingly
simple to establish that a survey of 2,424 people had been done. While I recognized that it is extremely easy
to lose data in a computer crash, I had not anticipated that Lott would claim
to have done a large national survey without discussing the sampling design
with anyone, leaving any financial or other records of the study, or
remembering anyone who had worked on it.
I had never heard of a professor doing anything of that size with no
funding, paid support, paid staff, phone reimbursements, or records (though
there are probably precedents for such an unusual method). As it stands now, unless someone comes
forward to verify working on the study--as I still hope occurs--we may never
know with any certainty whether the 1997 study was done. Although I strongly favor emailing 1997-98
University of Chicago college graduates to see if any remember any classmates
working on the study, John Lott now raises serious questions about how complete
the University’s alumni records are, rendering that approach a less reliable
route to an answer than I had anticipated.
I recently contacted Otis Dudley Duncan, who first raised questions about Lott’s claim that 98% of defensive gun uses involved the mere brandishing of a weapon. He agrees (as I do) with this statement of Tim Lambert on Lambert’s website:
“I should comment on the overall significance of this question. Lott's 98% claim takes up just one sentence of his book. Whether or not it's true, it doesn't affect his main argument, which is about alleged benefits of concealed carry laws.”
So there agreement even among those
who have raised questions about Lott’s work that his 98% claim is not central
to his book, More Guns, Less Crime.
Both
I find recent developments in this affair personally
troubling. I carefully recorded what
John Lott told me and now Lott has changed the story he told me in several
specific ways--most of them minor. They
are discussed in my revisions to this report.
I have no research interests in
this subfield and no ideas for further efforts to get to the bottom of this
inquiry beyond surveying graduates and Lott’s looking at picture books of
former students. This project detracts from my other scholarly efforts. Accordingly, my part in this affair is
essentially done, at least if John Lott will stop changing his stories about
our conversation. If not, then I suspect
that I will have to stay in it a little longer, at least to respond to comments
on this report.
For those who have been following the dispute over Lott’s
1997 study, other than a few fairly small changes, the portions of this report new
on January 17, 2003 are this Personal Note, Sections 4 and 5(“Comments on John Lott’s Response to this
Report” and “Conclusion”), and Appendix 3 (John Lott’s email responding to this
report).
James Lindgren
Professor of Law
Northwestern University
Comments on
Questions
About John
R. Lott’s Claims Regarding a 1997 Survey
Revised
Sections
4-5 added and Appendix 3 added on
1. Background
and John R. Lott’s Claims
a. Different Sources Cited for the 98% of Defensive Gun Uses
Being Mere Brandishing
On
Here
is Lott’s statement in the (first) May 1998 edition of More Guns, Less Crime,
which attributes the number to “national surveys.”
"If national surveys are correct, 98 percent
of the time that people use guns defensively, they merely have to brandish a
weapon to break off an attack." More Guns, Less Crime (University
of Chicago Press, 1998), p. 3.
The
year before, in the July 16, 1997 Wall Street Journal, Lott appeared to
attribute the 98% figure to one or more of three specific survey organizations:
“Other research shows that guns clearly deter
criminals. Polls by the
The same language (other
than typesetting conventions) appears the following year in two articles by
Lott on the same topic for the Chicago Tribune and the Washington
Times. John R. Lott Jr., Prime Suspect: Gun-Lock Proposal Bound to Misfire,
Starting in January 1999, Otis Dudley Duncan began
writing a series of letters to Lott questioning aspects of his work, including
the 98% figure. In May 1999, Duncan
informed Lott that he was writing an article calling the 98% a “rogue number”
and then sent him a draft of an article containing these words, “The '98
percent' is either a figment of Lott's imagination or an artifact of careless
computation or proofreading."
Lott then called
“I am a great admirer of Gary Kleck's work, and I
think that he has done a great deal to advance the study of crime. Few academics have his integrity and courage. His numbers are a little higher in terms of
the total number of defensive uses that I have found and the frequency of
brandishing is lower than I have found. The
information of over 2 million defensive uses and 98 percent is based upon
survey evidence that I have put together involving a large nationwide telephone
survey conducted over a three month period during 1997. Follow up telephone calls were made to ensure
that the questions were answered by those who we attempted to contact. The survey was not as detailed as several
other surveys, but it did try to include a couple initial questions to ensure
accuracy and screen out any problems and then focus exclusively on defensive
gun uses. I plan on repeating the survey
again during the next year to year and a half. I will be happy to inform you what the results
of that survey are after I have conducted it.”
Letter from John Lott to Otis Dudley Duncan, dated
Note that Lott makes no mention of having lost the data for the 1997
study. According to
In
the second 2000 edition of More Guns, Less Crime, Lott gives the same
98% figure, but (as in his letter to
"If a national survey that I conducted is
correct, 98 percent of the time that people use guns defensively, they merely
have to brandish a weapon to break off an attack." More
Guns, Less Crime, second edition (
In an email to me on
Then,
in February and March 2000, Lott gives the same 98% figure, but this time attributes
it to Gary Kleck, an authority that he had disavowed as the source for the 98% figure
in 1999. In his March 2000 piece on gun
locks, Lott wrote:
"Guns clearly deter criminals. Americans use
guns defensively over 2 million times every year--five times more frequently than
the 430,000 times guns were used to commit crimes in 1997, according to
research by
Lott used almost identical
language in a version of the same article on
Then in his comment in the September/October 2000 Criminologist, Lott returned to claiming
that he got his 98% number, not from Kleck, but from his own 1997 study. Otis Dudley Duncan raised questions about the
98% figure in The Criminologist (Jan./Feb.
2000), after exchanges between Lott and Duncan that occurred in 1999. In response to
“The survey that I oversaw interviewed 2,424 people
from across the
That means that the source
that Lott gave for the 98% figure has shifted over time:
1. In the 1998 edition of More Guns, Less Crime, John Lott attributes the 98% figure to
“national surveys.”
2. Elsewhere in 1997 and 1998, Lott appears to
attribute the 98% figure to “such polls” as the “Los Angeles Times, Gallup and Peter Hart Research Associates.”
3. In a
4. In almost
identical
5. In the Criminologist (Sept./Oct.
2000), Lott switches back to claiming that the 98% figure came from Lott’s own 1997 study, not from Kleck, which is
where things stand as of this report.
Note that by using the
plural “national surveys” and “such studies” Lott is stating that there are
more than just one study showing the 98% figure, yet he now insists that the
98% figure came from his own study, not Kleck’s (and Kleck’s study does not
support the 98% figure). Indeed, as
discussed in the next section, because the 98% figure is supposed to be based
on his own study, not those done by others, Lott says that his critics will
have “nailed” him if they find that he began talking about the 98% figure
before he says he did his study in 1997.
Yet, if Lott really based the 98% figure on his own study alone, it
seems strange that he would attribute the 98% figure to such plural entities as
“national surveys” or “such studies”--until
c. A Study Done “over 3 months during 1997”
In the Criminologist,
Lott also wrote: “My survey was
conducted over 3 months during 1997.” Lott called me on the telephone and repeated that he
had conducted the study over several months during 1997. If he spent 3
months doing it in 1997, as he claims, the earliest that he could have
completed it would be early April. Further, in an email to me Lott wrote,
“I am willing to bet that I don't start mentioning this [98%] figure until the
spring of 1997. If I use it before I said that I did the survey, I will
say that they nailed me. But if I only started using it about the time
that I said that I did the survey, I think that it would be strong evidence the
other way.”
Lott’s
first mention of the 98% figure located by Dudley Duncan is
"There are surveys that have been done by the
Los Angeles Times, Gallup, Roper, Peter Hart, about 15 national survey
organizations in total that range from anything from 760,000 times a year to
3.6 million times a year people use guns defensively. About 98 percent of
those simply involve people brandishing a gun and not using them."
Page 41, State of
I have not verified this
transcript, but if accurate, Lott’s
After
this discrepancy was noted in the first draft of this report, on
“The
overwhelming majority of the survey work was done at the beginning of the
period over which the survey was done. It has obviously been a while, but
my recollection is that the small number of people surveyed after the first
four or five weeks (mainly January 1997) did not include any more defensive gun
uses.”
While
again this story is certainly possible, Lott himself gave spring 1997 as the
time before which he should not have been discussing the 98% figure. Additional matters bear mentioning. It hardly matters whether all of the
defensive gun uses were found in the first 4-5 weeks of the study, since Lott
could not have known that at the time he spoke about the results unless data
collection were complete. If data
collection were partial, the precise percentage of defensive gun uses would
have been higher with partial data.
Collecting so much data in 4-5 weeks would have been unusual for unpaid
volunteers who were full-time undergraduate students at the
There
are other, more ambiguous contextual clues that Lott had contrasted his main
work, which was done on county-wide data, with surveys done by others, which
involved household surveys. For example, in the Washington Times
in 1999, Lott wrote:
“Indeed, about 450,000 crimes, including 10,744
murders, were committed with guns in 1996. But Americans also use guns
defensively over 2 million times a year and 98 percent of the time merely
brandishing the weapon is sufficient to stop an attack.
In my own recent research on gun ownership across
states and over time, I found that states with the largest increases in gun
ownership rates had the largest drops in crime rates.” John R. Lott,
Lethal Handgun Fears,
In
a long 1999 Chicago Tribune Magazine story, after speaking to John Lott,
Linnet Myers twice contrasted Lott’s county-wide work with Kellermann’s
household study:
Bolstering the other side is Dr. Arthur Kellermann,
of
indicates that owning a
gun is far more dangerous to a homeowner than it is to potential intruders.
. . .
Lott didn't examine home protection, but he did
study the impact of armed self-defense. In his book, "More Guns, Less
Crime" (
citizens who want to
carry handguns. Twelve states allow permits in certain cases. Seven, including
…
[Tom] Smith points out that while the two
researchers clearly support opposing sides in
Lott didn't study gun use at home, but looked at
the impact of laws that allow guns to be carried outdoors. Even so, Lott said
that in most cases of self-defense, "people merely need to brandish a gun
. . . less than 2 percent are fired." He said guns particularly help
women, who become more "equal" to men when they're armed. "Women
who behave passively are 2.5 times more likely to end up being seriously
injured than women who are able to brandish a gun when confronted by a
criminal," said Lott. Linnet Myers, Go Ahead Make Her Day With Her Direct
Approach And Quiet Confidence,
If
this newspaper account is accurate (and newspapers often aren’t), it is odd
that Lott would try to answer the reporter’s claims about the Kellermann
household study without pointing out that he had done a big household study
himself. Although this contextual evidence is less telling, it does tend
to fit the pattern that, until Lott replied to
2. Technical Problems
Another
problem (mentioned by Tim Lambert and Dudley Duncan) is the small and almost
impossible numbers of respondents on which Lott would have based his claim
about the rate of firing v. merely brandishing. According to Lott, the
study found that 98% of the defensive uses of a gun involve mere brandishing,
and that 3/4ths of that 2% involve firing warning shots, the other 1/4th
of that 2% involves firing at a person threatening the shooter. With
Lott’s estimate of 2.1 defensive gun uses a year and
about 200 million adults, that would mean that about 25-26 respondents reported
defensive gun uses out of his 2,424 people surveyed. Thus, only ½ of a
person (2% of 25 people) reported firing a gun--and that ½ of a person
breaks down further into 3/8ths of a person firing warning shots and
1/8th of a person firing at someone.
While
one can get fractions of a person if one weights respondents by their numbers
in the general population, getting 8 times more people in a sample group than
ideal would be rare (the number needed to justify a weight of 1/8th).
Further, how can one generalize three different rates (firing, firing a warning
shot, and firing at a person) from at most only one (or two) people, people who
are so unrepresentative that they are weighted as 1/8th or 3/8ths
of a person?
It
is possible that a multi-year window was inquired about, though even with a
five year rate, one still has only 5/8ths of a person and 15/8ths of a person
to support his reported rates. Further, Lott never gives the rate as
being for a multi-year window such as 1993-97, sometimes implying that the rate
is for 1997. Even if a five-year window were used, the numbers of
respondents would be still be so low (e.g., 5/8ths of a person) as to be
unreliable for reporting a rate.
Unless
John Lott can come up with a sensible explanation for why his rates could
possibly be justified with only a 2,424 person sample, it is my opinion that
Lott should withdraw the 98% figure as probably erroneous and, in any event,
too unreliable to form the basis of an estimated rate. Perhaps he has an
explanation that doesn’t appear yet. If not, withdrawing the 98% figure
in some appropriate way would be a simple matter of good social science.
After
this report was written, Lott disclosed that he had done a new survey of about
1,000 respondents in the fall of 2002.
Daniel Polsby has spoken to enough people involved and seen enough
records to determine conclusively that this new study was done. Even more than with the earlier study,
however, I don’t see how one can get an estimate of something that Lott says
happened to about 1 out of every 4,800 people each year (2% of 1.05%) with a
sample size of just over 1,000 people, asking about their experiences over the
last year.
3. The More Serious Issue: Was the 1997 Study Ever Done?
a.
Circumstantial Evidence of a 1997 Computer Crash
The
more serious question, which Tim Lambert has raised, is whether Lott ever did
the 1997 survey giving rise to the 98% figure. As I posted in September,
all evidence of a study with 2,400 respondents does not just disappear when a
computer crashes. Having done one large survey (about half the size of John
Lott’s) and several smaller surveys, I can attest that it is an enormous
undertaking. Typically, there is funding, employees who did the survey,
financial records on the employees, financial records on the mailing or
telephoning, the survey instrument, completed surveys or tally sheets, a list
of everyone in the sample, records on who responded and who declined to
participate, and so on. While all of these things might not be preserved in
every study, some of them would almost always be retained or recoverable.
Just to get a representative list of the
According
to Lott, he lost all of his data on his hard drive when it crashed in June of
1997. I talked with one of Lott’s co-authors on another paper, Bill
Landes, and received emails from David Mustard, another co-author, and Gregory
Huck, Lott’s editor at the time at the University of
Chicago Press. With varying degrees of certainty, all give
circumstantial support to Lott’s story of a sudden loss of data and text on
projects, requiring delays and regeneration of work. Further, Mustard
recalls hearing about the 1997 study, though when Lott told him about it is a
little unclear from Mustard’s email to me:
John told me that he had conducted a survey in
1997. I did not participate in the survey--it was after our concealed carry
paper had been published (Jan 1997) and was after I was on the job market and
while I was finishing my dissertation and then moving to
John had some major computer problems in 1997 or
1998--I am not sure of the exact timing, but I think I was already here in
John has always impressed me with his willingness
to give out his data; to anyone who requests them. To my knowledge John has
always released his data to anyone who asks of it. In fact, we gave out our
data about 4 months before the article even came out in print. We have now
given our data out to about 75 people from around the world; perhaps more[.] As I understand the survey situation, John does not
release the survey data because he no longer has it, not because he is unwilling
to do so.
Mustard also is not sure
when the data loss occurred. At first he
uncertainly placed it after he moved from
“As
to the _date_ of John's computer crash, it could have happened in June 1997. My
previous response would more accurately be that I _sent him the data_ after I
was at
David Mustard, like Bill Landes,
Russell Roberts, and Daniel Polsby, also went out of his way to note Lott’s
scrupulousness in sharing data on other projects.
b. No Direct Evidence of a 1997 Survey
As
for obtaining direct, rather than circumstantial, evidence that the study was
done, I did not fare as well as might be expected. Lott called me and
told me the following:
1. Lott had no funding for the project; he paid for expenses himself.
2. The survey was done by phone by several
3. The calling was done by the undergraduates from their own
phones. Periodically, they would bring over their phone bills and Lott
would reimburse them out of his own pocket--either in cash or by check.
Asked whether he retained his checks, Lott said that he destroyed them after 3
years. I did not think to mention it, but the phone research expenses should
have been deductible if they were not reimbursed by his employers.
4. Lott does not remember the names of any of the undergraduates who did
the calling for him.
5. Lott had no discussions with any samplers about his sampling design.
6. Lott did not weight his sample for household size and did not ask how
many adults were in the household (as is standardly done, thus rendering his
results too heavily influenced by small households). [In response to Lott’s
recent question to me about why weighting for household size is necessary, if
the unit of interest is households rather than people, then you do not have to
weight responses by the number of adults in the household. If, however, the unit of analysis is people
rather than households, then you should weight for household size, otherwise
you will over represent people who live alone and under represent those who
live in large households. For example, if the average number of adults in
households were 2 people, then you would normally weight the result for a
person living in a household with 4 adults by 2, and weight the result for a
person who lives alone by ½. I asked
this question of Lott for another reason; it might have explained his small
apparent weights for some people if he had adjusted for household size, but he
didn’t. Sampling is a lot more
complicated than this simple example implies, which is why very few people
would attempt a national sample without consulting a sampling expert.]
7. For his list from which to draw the sample, Lott used a commercially
available CD-ROM with names on it. He does not remember where he got it
or now have the CD.
8. Lott does not remember how he drew his sample from the CD-ROM.
9. Lott does not have a copy of the survey instrument and doesn’t
remember the wording of the questions, though he was probing defensive uses in
more detail than other studies. He ended with a very few demographic
questions.
10. Lott weighted his respondents by demographic information taken from
his main national study in More Guns, Less Crime.
11. In his book More Guns, Less Crime,
Lott had planned to include a chapter on the 1997 study, a chapter that he had
not yet written, but decided not to do so after the data loss. He did not
end up publishing the 1997 study itself, just referring to it many times,
including a sentence about it in the second edition of More Guns, Less Crime.
12. Lott thinks that he did not retain any of the tally sheets, though he
is not certain. He reported that he
might have tossed out tally sheets or other evidence of the 1997 study during
one of his several moves over the years.
The
discussion of the tally sheets is possibly in conflict with what Lott wrote to
Dudley Duncan on AEI letterhead dated
With
the surprising lack of any of the normal indicia of having done a large
national study of 2,424 respondents, the key remains locating the
undergraduates who Lott says did the calling. The 1997 study was large,
extremely time-consuming, and very expensive in phone charges. Getting
2,424 respondents with refusals and callbacks would have required thousands and
thousands of phone calls. Students would have had to spend many hours
calling, which they and their friends would well remember. With John
Lott’s permission, I therefore contacted Saul Levmore, the Dean of the
University of Chicago Law School, requesting him to try to obtain from the
alumni office the email addresses of the 1997 and 1998 college graduating
classes at the University of Chicago, so that I could contact them asking
whether any of them or any of their friends had done any research for John Lott
during his fellowship at Chicago. Levmore declined to make such a request
for email addresses from the
Thus,
I have reached a temporary dead end. If someone were to email the 1997-98
4. Comments on John Lott’s Response to this Report
a. Several Changes in
Lott’s Story
On
THIS REPORT (above): “2. The survey was done by phone by several
LOTT’S RESPONSE (Appendix
3 below): “Lindgren does not accurately report my conversation with him about
how I paid people (in that I said that I possibly paid by check) . . . .
Incidentally, I told Jim that there were “two”
THIS REPORT: 3. “The
calling was done by the undergraduates from their own phones. Periodically,
they would bring over their phone bills and Lott would reimburse them out of
his own pocket--either in cash or by check.”
LOTT’S RESPONSE: “most of this next statement [“calling was
done by the undergraduates from their own phones “] is correct except the point
about the “possible” use of checks.”
The facts of my conversation with John are different
than he now remembers them. John Lott
called me shortly after I posted a notice to the FireArmsRegProf discussion
list on
What is most surprising about this claim is that Lott
emailed me on
As to the check issue, I
believe that the word "possible" was included, but you have to go on
what you remember. In any case, I am sure that I can get many people who
will say that I paid them with cash over the years.
As you can see, on Monday afternoon, Lott is “sure” on
one issue, but he only “believe[s]” that he included the word “possible” and
says that I have to go on what I remember.
Just a day later, on Tuesday, Lott is suddenly certain that I am wrong,
writing: “Lindgren does not accurately report my conversation with him about
how I paid people (in that I said that I possibly paid by check)” and “most of
[Lindgren’s] . . . next statement is
correct except the point about the ‘possible’ use of checks.”
In
September, I carefully listened to both Lott’s words and his demeanor, taking notes
on the more important parts of the long conversation. I wrote up those notes (which formed the
basis for what Lott told me listed in items 1-12 above), later revising them
into my report. I retain a vivid recollection of what Lott told me on many points,
including this one. One day, four months
after the conversation, Lott sent me an email saying that he “believe[s]” that
he said that he qualified his having paid students by check with the word
“possible,” but appears not to be sure (as he is about other instances of
paying cash on other projects). He tells
me that I have to go on what I remember.
Then the very next day, Lott is certain enough to accuse me of
“inaccuracy.”
Lott’s new assertion about
there being only two
Lott goes on to make less serious changes in his
story. For example:
THIS REPORT: “5. Lott had
no discussions with any samplers about his sampling design.”
LOTT’S RESPONSE: “I had
lunch Tom Smith during the fall of 1996. However, while I asked him many
questions about surveys, I did not tell him what I was planning on doing
because Tom works very closely with gun control organizations.”
A related LOTT RESPONSE: “Russell
Roberts is someone that I bounced the survey questions off of and he can
possibly talk to you about it, though he hasn't kept e-mails from 1997.”
One of the online
commentators is reassured that Lott now says that he discussed the survey
questions at the time with Russell Roberts of Wash. U. But Lott did not say this when I asked him in
September. He said that he regularly
asked Roberts for advice and that he might or might not have discussed his 1997
survey with Roberts. He didn’t remember
whether he did or not. I spoke with
Russell Roberts yesterday and Roberts said exactly what Lott told me in September--that
Roberts regularly discussed matters with Lott, but couldn’t remember hearing
about the 1997 study and couldn’t remember one way or the other whether Lott
had discussed it with him. Lott could
well have discussed the study, but Roberts didn’t remember him doing so.
Also, on
Lott’s recent change of
story--that he discussed specifically the survey questions with Roberts--might
well be true (Lott’s memory might have been jogged), but it is not what Lott
said to me in September (or even bothered to mention the day before he changed
his story when he reasserted his contacts with Roberts in an email). If Lott had told me in September that he had
discussed the survey questions with Roberts, I certainly would have called
Roberts right away, as I did yesterday. After
all, I was looking for ways to verify the study.
By the way, in Lott’s
defense I must point out that some online commentators have falsely claimed
that Lott never mentioned the study to anyone, but if you read my report
carefully above, David Mustard quite strongly confirmed Lott’s claim to have
discussed the 1997 study with him (and his data loss), but Mustard does not
remember when he first heard about it.
As to my statement that
“Lott had no discussions with any samplers about his sampling design,” I said
this because I asked Lott this question point blank and he flatly said
“No.” He did mention that he talked with
Tom Smith, but he said that he did not discuss his sampling design with him. One doesn’t just pull a national sample out
of one’s head. One usually either uses a
random digit dialing program or a national sample provided or designed by an
expert in survey sampling. A CD-ROM with
names on it designed for telemarketing is not the sort of thing academics
usually use if they want a representative sample, which is the reason I asked
whom he consulted on his sampling design.
Another change in story involved the CD-ROM and the
survey questions:
THIS REPORT: 8. “Lott does
not remember how he drew his sample from the CD-ROM.”
LOTT’S RESPONSE: “Not true. I told Jim that one of the
students had a program to randomly sample the telephone numbers by state. My
guess is that it was part of the CD, but on that point I can¹t be sure.”
THIS REPORT: “9. Lott does
not have a copy of the survey instrument and doesn’t remember the wording of
the questions, though he was probing defensive uses in more detail than other
studies. He ended with a very few demographic questions.”
LOTT’S RESPONSE: “It is
also not quite correct to say that “doesn¹t remember the wording of the
questions.” I told Jim that I don¹t remember the “exact wording” of the
questions, but I gave him the general outline of the questions.”
Once again, Lott’s memory fails him. I was listening closely to Lott’s answers to
these questions to see what details he could pull off the top of his head. I asked him how he drew the sample from the
CD. He said that he didn’t remember, but
assured me it was drawn randomly. I
remember being disappointed in his answer because I thought that a social
scientist would probably remember how he solved this problem of getting a
random sample (there are several solutions).
Also, I am absolutely positive that he did not mention pre-stratifying
the sample by state, which is a form of proportional sampling, not random
sampling. His current claim is
inconsistent with his September claim made to me that the sample was drawn
randomly from the list of names on the CD (though he didn't remember how he
drew that random sample); he seems to be claiming now that he drew the sample proportionally
by state then randomly within states. I
was paying close attention to any details he mentioned about sampling design
and he never mentioned breaking down proportionally by state first. I am not disturbed by the content of his
claim so much as that he would think he could just tell me one thing in
September (he did not remember how he drew the sample from the CD) and tell
people something else in January (“I told Jim that one of the students had a
program to randomly sample the telephone numbers by state”).
A
similar problem obtains for question wording.
I asked him directly whether he remembered the wording for any
questions. He said “No,” not their
wording, but he assured me that he was trying to probe defensive uses in more
detail than prior studies had and that he ended with a few demographic questions. He gave no details of his questions other
than this. The notion that he gave me
“the general outline of the questions” beyond what I faithfully reported is
just plain false. Drafting questions to
probe something you care about might (or might not) be just the kind of thing
that would stick in someone’s mind, so I was hoping that he could come up with
plausible approximate wording off the top of his head. If he had, I certainly would have noted
it. He couldn’t. I was looking for exculpatory evidence and was
disappointed to find not much more than good evidence of his exemplary pattern
of sharing data and very good circumstantial evidence that Lott had a major
computer crash in 1997.
b. Lott’s Attempt to
Put Things in Context
In
his response to this report, John Lott makes several odd statements other than
about our conversation. For example,
Lott claims: "I have told people directly (including Otis Duncan) from the
beginning that the data were lost.” Otis Dudley Duncan, who first raised
questions about Lott’s 98% figure in early 1999, however, says that he first he
learned of any data loss when Lott published his comment in the Sept./Oct. 2000 Criminologist.
“The
information of over 2 million defensive uses and 98 percent is based upon
survey evidence that I have put together involving a large nationwide telephone
survey conducted over a three month period during 1997. Follow up telephone
calls were made to ensure that the questions were answered by those who we
attempted to contact. The survey was not as detailed as several other surveys,
but it did try to include a couple initial questions to ensure accuracy and
screen out any problems and then focus exclusively on defensive gun uses. I
plan on repeating the survey again during the next year to year and a half. I
will be happy to inform you what the results of that survey are after I have
conducted it.” Letter from John Lott to Otis Dudley Duncan, dated
As you can see, while far
from conclusive on the point, Lott’s letter is consistent with
Lott
also claims that he has “always acknowledged” that the 98% figure is based on
small samples.
LOTT’S
RESPONSE (Appendix 3 below): “As to so-called technical problems, I am [sic] have always acknowledged that these are small samples,
especially when one breaks down the composition of those who use guns
defensively. Even the largest of the surveys have few observations in
this category.”
As pointed out above in the
section of my report called “Technical Problems,” in a sample of 2,424
respondents and a one-year window, Lott would find only about 25 respondents
reporting defensive gun uses, 2% of which (1/2 of a person) answering that they
had fired their gun. On the more than
four dozen occasions collected by Dudley Duncan and Tim Lambert in which Lott
mentioned the 98% figure, I do not see a single instance in which Lott
“acknowledged that these are small samples.”
When Lott says that he has “always acknowledged” this fact, he seems to
be in error.
Lott uses another style of argumentation that
I find troubling. Lott writes:
As to the attribution of sources, look at the
complete context of the quote Lindgren mentions:
Polls by the Los Angeles Times, Gallup and Peter
Hart Research Associates show that there are at least 760,000, and possibly as
many as 3.6 million, defensive uses of guns per year. In 98 percent of the
cases, such polls show, people simply brandish the weapon to stop an attack. --
References by Lindgren to things like the Linnet
Myers piece in the Chicago Tribune to provide evidence that I didn¹t do a
survey or that I have changed my statements over time are simply bizarre.
Attached below is an edited down version of the letter that was published by me
in the Tribune. Myers used her article to refloat claims such as my Olin
Funding, inaccurately reported exactly what the concealed handgun research
covered, and claimed that "others haven't confirmed (my)
findings." I no longer have the original letter to the editor, but as
I recall this is just a partial listing of her inaccurate statements. The
Tribune was not willing to run a longer letter, though the letter that they ran
was quite long.
Lott
says that he is going to give “the complete context” for a statement that I “mention”
from the Chicago Tribune and Washington Times. A fair minded reader would conclude that Lott
is actually quoting me, but he isn’t. I
didn’t quote the Chicago Tribune
version of the statement, but rather quoted the original version in the Wall Street Journal:
The year before, in the
July 16, 1997 Wall Street Journal, Lott appeared to attribute the 98%
figure to one or more of three specific survey organizations:
“Other research shows that guns clearly deter
criminals. Polls by the
The same language (other
than typesetting conventions) appears the following year in two articles by
Lott on the same topic for the Chicago Tribune and the Washington
Times. John R. Lott Jr., Prime Suspect: Gun-Lock Proposal Bound to Misfire,
Instead
of responding to the quotation I actually use from his 1997 oped in the Wall Street Journal, Lott instead quotes
an almost identical version of the same statement that he published in a later
Aug. 1998 oped in the Chicago Tribune,
fails to mention that the words in the Tribune
are under his byline, and then appears to provide "the complete context"
for his own statement by questioning an unrelated May 1999 Chicago Tribune story by a reporter. In the guise of providing “context” Lott omits
crucial information that is in my presentation above. Lott
omits that he first used the language in the Wall Street Journal in 1997 and presents the quotation as if these
are the words of the Chicago Tribune
or the Washington Times. Lott never mentions that these are his
words, published in an oped under his byline--a failure that tends
to undercut his claim that he is providing “the complete context.”
Lott never says why he chose to quote a Chicago Tribune version of his statement
rather than the earlier Wall Street
Journal version that I actually quote, nor does he say what his Tribune oped has to do with a later Tribune story by a reporter. Yet (whatever Lott’s intentions) I think a fair
minded reader might presume that one somehow gives context for the other--perhaps
one might conclude that I am wrong to mention a 1998 statement in the Chicago Tribune because you can't trust
their 1999 reporting on him. That might
appear to be a sound implication if one fails to realize that the 1998 Tribune statement he quotes is not the
version I quoted (I quoted the 1997 Wall
Street Journal), that the words he quotes are from his own 1998 oped in the
Tribune under his own byline (a fact
that he neglects to mention), and that he made a close version of the same
statement in his own oped in the Wall
Street Journal the year before (presumably untainted by any supposed Tribune bias). Of course, Lott never tells us what part of
the context he is providing for his 1998 Tribune
oped that I mentioned but didn’t actually quote.
In the course of this, Lott also criticizes my use
of a 1999 Linnet Myers story in the Chicago
Tribune. I did qualify my use of
that story with these words:
“If this newspaper
account is accurate (and newspapers often aren’t), it is odd that Lott would
try to answer the reporter’s claims about the Kellermann household study
without pointing out that he had done a big household study himself.
Although this contextual evidence is less telling, it does tend to fit the
pattern that, until Lott replied to
In the course of his criticism of my use of that story, Lott makes a potentially
damaging disclosure. Lott reveals that
he published a 1999 letter in the Tribune
complaining about errors in Myers’s story.
The text of his Tribune letter
is included in Lott’s email response in Appendix 3 below. Myers had written: “Lott didn't examine home protection, but he did study the impact of armed
self-defense. . . . Lott didn't study gun use at home, but looked at the impact
of laws that allow guns to be carried outdoors.” Linnet Myers, Go Ahead
Make Her Day With Her Direct Approach And Quiet Confidence,
In
his June 1999 letter to the editor published in the Tribune, Lott responded particularly to Myers’s sentence in which
she claims that Lott didn’t study gun use at home, but rather the impact of
laws that allow carrying guns outdoors:
“My book
analyzed FBI crime statistics for all 3,054 American counties from 1977 to 1994
as well as extensive cross-county information on accidental gun deaths and
suicides. This is by far the largest study ever conducted on crime, accidental
gun deaths or suicide. I examined not only concealed-handgun laws, but also
other gun-control laws such as state waiting periods, the length of waiting
periods, the Brady law, criminal background checks, penalties for using guns in
commission of crime and the impact of increasing gun ownership. The only gun
laws that produced benefits were those allowing concealed handguns. The
evidence also strongly indicates that increased gun ownership on net saves
lives.” John Lott, Letter,
Note that in his letter to
the Tribune, Lott makes no mention of
his 1997 study of households that he claimed to have done, even while
responding to a sentence that asserted that Lott didn’t study gun use at home. Lott lists seven things he looked at:
concealed-carry laws, waiting periods, waiting period length, the Brady law,
background checks, extra penalties, and gun ownership rates. In his published
letter, he omits to mention his 1997 telephone household study, even though
Myers twice says he didn’t do a household study. In Lott’s defense, Lott says that the letter
was cut; the original was longer. But Lott’s
published letter as it stands includes a long sentence listing the sorts of inquiries
Lott did, but fails to mention his 1997 study done at the household level,
rather than his other inquiries at the county level.
5. Conclusion
I think it prudent to
withhold judgment on the question whether the 1997 study was done until an
email inquiry of
I
remain hopeful that