Szollosi, Kellen, Navarro, Shiffrin, van Rooij, van Zandt & Donkin (2019) recently wrote the provocatively titled “Preregistration is redundant, at best”. They argue that
[t]he key implication argued by proponents of preregistration is that it improves the diagnosticity of statistical tests
and explain why there is
little reason to expect that preregistration will spontaneously help researchers to develop better theories
while they emphasize that
the diagnosticity of statistical tests depend entirely on how well statistical models map onto underlying theories.
Since preregistration cannot help with better theories, and better theories are a prerequisite for, well, informative statistical inference, they argue that preregistration is redundant.
As far as I can see they are right on target with everything except the conclusion. The reason why I disagree with their conclusion is most likely because they’re real scientists, and I’m not. In that sense their argument is very optimistic: it is based on the premise that developing better theory is what most articles in psychology are concerned with. I’m not persuaded that’s true, but would love to be wrong.
I wish we were at the point where improving our theories was the main concern. However, most theory in psychology is still qualitative: boxes and arrows that roughly predict that something is associated to something else, and it is not uncommon for articles to simply apply structural equation modeling to test a model with a slew of implicit assumptions that are never discussed and do not follow from the theory.
Some fields have progressed more than others, of course. But in some areas, the concerns of Szollosi et al. (2019) are almost utopic. If only this was the level at which theories are developed and evaluated! In my own field, for example, there was recently (relatively recently; I’m getting old I guess) a discussion about ‘who has the right to change a theory’. There’s a big gap between these two approaches to theory - and I think that gap partly explains why I agree with the arguments of Szollosi et al. but not with their conclusion.
Advantages of preregistration
First, let me list a number of advantages of preregistration that were not discussed by Szollosi et al. (probably because of the strict word limit).
Clear, explicit description of researchers’ expectations and biases
The value of trying to identify one’s own biases and making then explicit is acknowledged a bit better by researchers applying qualitative methods, but this is valuable for quantitative research as well. Such descriptions are best made explicit before data collection starts, as the results may change them (unconsciously).
Risks of preregistration
Afterthought
This preprint received oddly dismissive responses on Twitter. They’re linked to in this thread by John Sakaluk:
Cohen suggested that medium effect sizes and larger should be visible to the naked eye. So let's talk about what is visible to the naked eye in the thread that Jenny (yay Jenny!) draws our attention to: https://t.co/bKTb8DN3sK
— John Sakaluk (@JohnSakaluk) November 1, 2019
My experience is that when people respond dysfunctionally, it’s often because they feel threatened in some way. Of course, that might just be my optimistic disposition (i.e. in this scenario people aren’t simply dysfunctional), but if we bear with this interpretation, perhaps this can partly explain those responses to the preprint.
After all, this preprint is pretty confronting about the state of some fields in psychology: theory and testing are often not taken as seriously as Szollosi et al. (2019) clearly do. Then again, the deliberately provocative title maybe also didn’t help.