There has been a lot of talk about the importance of reproducibility of research e.g. in psychology (and don’t get me starting about replication in educational sciences) and via this post by Neuroskeptic that you just must read you’ll discover another reason why. Just look at this picture from this paper by David Shanks et al.Romance, Risk, and Replication on romantic priming:
Shanks et al. say that this is evidence of the existence of “either p-hacking in previously published studies or selective publication of results (or both).”These two forms of bias go hand in hand, so the answer is probably both.Publication bias is the tendency of scientists (including peer reviewers and editors) to prefer positive results over negative ones. P-hacking is a process by which scientists can maximize their chances of finding positive results.
I’ve been blogging about these issues for years, yet still I was taken aback by the dramatic nature of the bias in this case. The studies are like a torrent, rolling down the mountain of significance. The image is not so much a funnel plot as an avalanche plot.
If we want to have people to listen to science – which I think is important – and scientists, we need to get our act together, imho.
