I just had my first pre-registered and results-free peer review paper published at BMC Psychology (Krishnan, Watkins, & Bishop, 2017, BMC Psychology). There’s more about why the journal is trying this format here. It’s a brave new world of open science out there, and I thought it’s worth trying to have some discussion about how this process differs from a standard submission, and whether it’s worth trying. I’ve consequently summarised my experience of this format, and a few takeaways for future pre-registered studies. This is also my view, it’s possible my co-authors would take a completely different perspective.
During my postdoc at OSCCI, I was keen to investigate how different methods of presenting information during learning could affect word learning outcomes. The two approaches I was keen to try (recall, reproduction) seemed to be quite successful in studies where adults learned written words, but spoken word learning hadn’t been investigated in any great detail. I thought if these methods did enhance learning, it could offer us some clear translational insights for children with language learning disorders, and the two approaches seemed to rely on different brain bases (a research interest of mine). A registered report was not an option as my position was only for a year. I therefore pre-registered my design on the OSF. As part of the pre-registration, I ran a power analysis, on the basis of which I doubled the number of participants I originally planned to test (N=36, instead of the usual norm of N=20). I did expect to find a real effect, especially as this study was meant to be a precursor to an fMRI study. I was quite disappointed when we didn’t find effective long-term differences for recall and reproduction. Despite this, I was keen to actually get the results published as I did think that the rationale and design were reasonably strong. However, past experiences of trying to publish null results made me a bit wary, and I knew I didn’t want to be spending a lot of time simply re-submitting the paper to multiple outlets. I was also in the middle of switching postdoctoral roles, so time was something I was short of. So I thought the results-free process advertised at BMC Psychology was a great way to go.
This process for a results-free submission is procedurally not too different from a registered report, but the major difference is that the study is already completed when the reviewers assess it. Authors submit their introduction and methods at the first stage. An editorial decision to publish is made on the basis of this submission, and if accepted in principle, the authors are expected to (quickly) add the results and discussion sections. The paper is reviewed again, but this time, the editor’s and reviewers’ role is mainly to make sure that the interpretation of the results is sensible and consistent with the hypotheses and proposed analysis plan.
For this paper, my first Stage 1 submission got rejected (with an invitation to resubmit but only if I could address some salient issues around novelty/ design). I emphasise this rejection for two reasons, 1) it’s not simply that all papers with open review or results-free submissions are accepted, and 2) my previous supervisor, Annette Karmiloff-Smith, once said that if reviewers didn’t understand something, it was likely that it wasn’t said very clearly (and that readers would probably act like reviewers). I rewrote parts of the introduction to emphasise how my paper was different from other work, and to emphasise why the design was still allowed us to address the question of interest. Annette was right, as this time the reviewers and editors did actually accept it in principle. We then added our results + discussion, and after a round of minor revisions, the paper was accepted.
So, do I think results-free peer review works? Here are the positives. We got excellent reviewers, who were real experts in the area, and their comments were really useful. It was clear from their comments that they thought the study would work and yield a positive result. This boosted my confidence, as it helped reinforce my belief that I had designed the study correctly, and there wasn’t a special ingredient or sauce I was missing. They also suggested some improvements to the design, and if I had their comments before I did the study, I would have made these (a limitation of results-free review relative to registered reports). We also managed to get an editor who was really interested in statistics and his suggestions helped to considerably improve our reporting of Bayesian statistics in the paper. Also, once the paper was accepted in principle, we were not asked by either the reviewer or the editor to do any other analyses (although we have included exploratory analyses we thought were interesting and worth following up on, such as correlations of learning with age, verbal memory etc). I think the last point is really important, as I think reviewers would have asked for multiple other analyses if they had known about the results alongside (this has happened in talks, and I think it’s completely natural, as scientists are by nature, curious!). For example some very sensible questions would have been, what were the learning slopes across conditions? Do they relate to individual differences in verbal memory? While these are interesting, they weren’t our primary questions, and now, the data is openly available should anyone have a burning desire to address these. I think better control of the publication timeline, a point that’s already been made by Dorothy Bishop, is an excellent reason to pre-register (especially as an ECR).
However, there were some downsides to the results-free review process. First, I think some of the issues that the reviewers raised were would not have come up if they could have seen the pattern of data we obtained. Knowing the pattern of data gave us a stronger platform to argue from for the resubmission, which we would not have had for a registered report. For a registered report, this could potentially be addressed by having pilot data, although there is always the question of how much data is enough. Having expertise in the specific area would have also helped with this issue. Second, the reviewers originally suggested an additional experiment, but having this included in the scope of the Stage 1 submission was simply not possible. This may be an issue for multi-experiment papers, and I’d be curious to see how these are handled. Third, despite the editor managing the process doing a stellar job at trying to get reviewer comments back to us quickly, and manage the process, we did effectively do two extra rounds of revision, so it did somewhat take longer than I expected (especially given that you would expect to circulate papers to co-authors for comments when changes are made). However, none of these are insurmountable problems.
The upshot is that I would definitely pre-register again, but I don’t think I will pre-register all the studies I expect to run (yet). For example, if the study had yielded the result we were expecting, I wouldn’t have been able to put together a full pre-registration for the ensuing fMRI study. Similar issues have been brought up by lots of people in the field. My view is that pre-registration is only feasible when there is a clear hypothesis and enough previous work to properly enable power analysis. I also think that I would plan future studies a little differently if I plan to pre-register them. My main takeaways from this OSF pre-registered/ results-free peer review experience were:
- Get pilot data. As much as possible. And ask people outside the lab to do your study!
- Power analyses are non-trivial. However, it is much easier if you know the field or have previously collected similar data – modelling means/ standard deviations can become much more obvious. Again, having pilot data really helps with data simulation.
- Pre-registering a study front loads the process of writing, building a plan for analysis, etc. You do most of the work before data collection. This means planning for this time is crucial. I only finished data collection a month before my post-doctoral position at OSCCI ended.
- Also, if there was time, I would write a registered report instead of choosing results-free peer review, as it allows for an opportunity to incorporate reviewer suggestions.
- Following a fully open science pipeline made me feel like there were a lot of things I could have done better (like my code, labelling my data etc). But even though my first attempt is far from perfect, doing it means I have a much clearer vision about what I’d like do better for my next project/ paper (for example, I’d like to write R code for my analyses at the point of pre-registration). Watch this space!
Comments very welcome.
Leave a Reply