A UC Berkeley graduate and doctoral student announced last week that they had found data irregularities in a widely cited study, which originally found that conversations with activists could change voters’ minds on same-sex marriage.
David Broockman, who recently obtained a doctorate from UC Berkeley, and campus doctoral student Joshua Kalla — along with Peter Aronow, assistant professor at Yale University — published a report showing a number of irregularities in a study published in journal Science in December. The study, spotlighted by major media outlets such as the New York Times, suggested that voters were likely to change their opinions after 20-minute talks with lesbian, gay, bisexual and transgender activists.
After seeing Broockman and Kalla’s evidence, Donald Green, the co-author of the original report and a Columbia University professor of political science, submitted a retraction letter to Science.
Broockman and Kalla, according to a report published last week, were impressed by Michael LaCour and Green’s research and had planned to expand the study on same-sex marriage in January. But the report said Broockman and Kalla’s initial doubts arose when they noticed that the study had a higher rate of test-retest reliabilities — a test of how consistent an experiment is over time under the same conditions — than in other panel survey data and that the response and re-interview rates were higher than expected.
These doubts became clearer after they failed to replicate LaCour’s data in a pilot survey, and according to the report, the survey firm purportedly involved in LaCour and Green’s study said it was not familiar with the project.
In Green’s retraction letter to Science, LaCour said that he accidentally deleted the data. But a representative from Qualtrics, an online survey platform used in the study, found no evidence of such a deletion.
LaCour said in an email that he will release a statement by Friday and that he still stands by his original study. Broockman and Kalla declined to comment.
Gabriel Lenz, an associate professor of political science at UC Berkeley, said Broockman and Kalla were “two of the stars” of the campus graduate program in political science.
“All these things (make it) hard to believe that (the original data) is true,” Lenz said.
Having known LaCour since he spoke with him at UCLA three years ago, Lenz said he was surprised when he first heard about the incident. He said that he thought LaCour’s past studies were well done and that he was impressed with LaCour’s analysis and with LaCour himself.
Alexander Coppock, one of the doctoral students at Columbia University who said Green asked him to check parts of the analysis of the study, said that the analysis was straightforward but that the primary data the study used was “a complete fabrication.” He expressed anger toward LaCour for wasting the time of the people involved in the study, including himself.
Lenz said that Green, though a co-author of the study, did not have access to the raw data automatically, as it identifies the respondents of the study, and that he could have sought approval from an institutional review board to see the data himself. Both Coppock and Lenz, however, said they have never heard of such a fabrication of data.
“I don’t think many people would have done it differently,” said Lenz, who described Green as “one of the most respected, terrific social scientist(s).”
Coppock said that it is important for researchers to establish credibility and that when the research is redone, it will be done in the right way.