A recent special issue of the journal Social Psychology is dedicated to an admirable effort to replicate 27 studies that have been cited numerous times in the scientific literature and attracted much media attention. Social psychology has been plagued recently by a number of scandals, but now, it seems, social psychologists lead the way to a new standard of scientific publication.

Science journals have been very reluctant, to say the least, to publish papers reporting replication of previous studies. More than 99% of the published studies report new results, many of which find their way into the public domain as unchallenged scientific truth. Replication is a fundamental part of the scientific process as it is the only way to falsify claims about reality. Yet, scientific journals prefer ‘positive’ findings over ‘negative’ ones that show that data are unrelated or theories unsupported. Journals, and the public media in particular, have a strong preference for spectacular new results.

In a paper in PLOS Medicine in 2005, John Ioannides observed:

There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance.
Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias.
[Abstract]

In BloombergView Megan McArgle gave a piece of advice to readers:

The more interesting the result, the more likely it is to be a product of random chance producing a publication-worthy outlier.

The special issue of ‘Social Psychology‘ has been reviewed by Michelle N. Meyer and Christopher Chabris in Slate Magazine. Here is the first paragraph:

Psychologists are up in arms over, of all things, the editorial process that led to the recent publication of a special issue of the journal Social Psychology. This may seem like a classic case of ivory tower navel gazing, but its impact extends far beyond academia. The issue attempts to replicate 27 “important findings in social psychology.” Replication—repeating an experiment as closely as possible to see whether you get the same results—is a cornerstone of the scientific method. Replication of experiments is vital not only because it can detect the rare cases of outright fraud, but also because it guards against uncritical acceptance of findings that were actually inadvertent false positives, helps researchers refine experimental techniques, and affirms the existence of new facts that scientific theories must be able to explain.

Read more …

Thanks to Pedro De Bruyckere for tip drawing my attention to this special issue.

References

John P.A. Ioannidis (2005). Why most published research findings are false. PLoS Med 2(8): e124.

Michelle N. Meyer and Christopher Chabris (2014). Why Psychologists’ Food Fight Matters. “Important findings” haven’t been replicated, and science may have to change its ways. Slate Magazine Blog post.