Preregistration

2023-11-06 Mon

Rick Gilmore

Overview

Announcement

Last time…

Today

Preregistration

Preregistration

Nosek, B. A., Ebersole, C. R., DeHaven, A. C. & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences of the United States of America, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114

Abstract

Progress in science relies in part on generating hypotheses with existing observations and testing hypotheses with new observations. This distinction between postdiction and prediction is appreciated conceptually but is not respected in practice. Mistaking generation of postdictions with testing of predictions reduces the credibility of research findings…

(Nosek et al., 2018)

However, ordinary biases in human reasoning, such as hindsight bias, make it hard to avoid this mistake. An effective solution is to define the research questions and analysis plan before observing the research outcomes—a process called preregistration…

(Nosek et al., 2018)

Preregistration distinguishes analyses and outcomes that result from predictions from those that result from postdictions. A variety of practical strategies are available to make the best possible use of preregistration in circumstances that fall short of the ideal application, such as when the data are preexisting.

(Nosek et al., 2018)

Services are now available for preregistration across all disciplines, facilitating a rapid increase in the practice. Widespread adoption of preregistration will increase distinctiveness between hypothesis generation and hypothesis testing and will improve the credibility of research findings.

(Nosek et al., 2018)

Types

Illustration about registered reports from (Center for Open Science, n.d.)

Commentary about

  • (Ledgerwood, 2018)
    • Preregistering theoretical predictions \(\neq\) preregistering analysis plans
  • (Goldin-Meadow, 2016)
    • Could stifle discovery
    • Not applicable to all kinds of research

Is it working (Claesen et al., 2021)

Figure 1 from (Claesen et al., 2021). Flowchart of assessment of preregistered studies.

Figure 2 from (Claesen et al., 2021). Tile plot of the assessment of each methodological aspect per preregistration plan. Only the 27 studies that were accessible and included the minimal number of methodological details required for our adherence assessment are shown.

Figure 3 from (Claesen et al., 2021). An overview of adherence per methodological aspect.

Author-reported reasons for deviations

  • Suggestions from reviewers
  • Did not preregister or report details because they were thought redundant or irrelevant
  • Lack of consensus about what should be disclosed
  • Small discrepancy with the preregistered sample size was small and not really a deviation.

Conclusions

Our work suggests that it is not obvious for preregistration to live up to its promise of making research results more transparent and more interpretable in the manuscript. This insight calls for better standards in preregistering and reporting preregistered research…

It should, however, not be misinterpreted as a plea against the practice of preregistration. Neither does our work suggest that the results of the preregistered studies we assessed should not be trusted. We do not, and cannot, claim that the observed deviations have been deliberately unreported or constitute evidence of the exploitation of researcher degrees of freedom…

The process of preregistering and reporting in accordance with the preregistration is tricky, and we realize that, given its relatively young status in psychology, the field collectively needs to go through a learning phase.

(Claesen et al., 2021)

Your thoughts?

  • Are you surprised that studies differ from their preregistered plans? Why or why not?
  • Does preregistration make you nervous?
  • How do thesis/dissertation/grant proposals differ from preregistration?
  • What practice(s) would reduce deviations or alleviate skeptics’ concerns?

My experience

Next time

Data sharing

Resources

References

PSYCH 490.009: 2023-11-06 Mon

Adolph, K. (2016). Video as data. APS Observer, 29(3). Retrieved from https://www.psychologicalscience.org/observer/video-as-data
Center for Open Science. (n.d.). Registered reports. https://www.cos.io/initiatives/registered-reports. Retrieved from https://www.cos.io/initiatives/registered-reports
Chambers, C. (n.d.). Registered reports: What are they and why are they important? https://royalsociety.org/blog/2016/11/registered-reports-what-are-they-and-why-are-they-important/. Retrieved from https://royalsociety.org/blog/2016/11/registered-reports-what-are-they-and-why-are-they-important/
Claesen, A., Gomes, S., Tuerlinckx, F., & Vanpaemel, W. (2021). Comparing dream to reality: An assessment of adherence of the first generation of preregistered studies. Royal Society Open Science, 8(211037). https://doi.org/10.1098/rsos.211037
Gilmore, R. O., & Adolph, K. E. (2017). Video can make behavioural research more reproducible. Nature Human Behavior, 1. https://doi.org/10.1038/s41562-017-0128
Goldin-Meadow, S. (2016). Why preregistration makes me nervous. APS Observer, 29(7). Retrieved from https://www.psychologicalscience.org/observer/why-preregistration-makes-me-nervous
Houtkoop, B. L., Chambers, C., Macleod, M., Bishop, D. V. M., Nichols, T. E., & Wagenmakers, E.-J. (2018). Data sharing in psychology: A survey on barriers and preconditions. Advances in Methods and Practices in Psychological Science, 2515245917751886. https://doi.org/10.1177/2515245917751886
Ledgerwood, A. (2018). The preregistration revolution needs to distinguish between predictions and analyses. Proceedings of the National Academy of Sciences of the United States of America, 115(45), E10516–E10517. https://doi.org/10.1073/pnas.1812592115
Meyer, M. N. (2018). Practical tips for ethical data sharing. Advances in Methods and Practices in Psychological Science, 2515245917747656. https://doi.org/10.1177/2515245917747656
Mischel, W. (2011). Becoming a cumulative science. APS Observer, 22(1). Retrieved from https://www.psychologicalscience.org/observer/becoming-a-cumulative-science
National Institutes of Health. (n.d.). NOT-OD-21-013: Final NIH policy for data management and sharing. https://grants.nih.gov/grants/guide/notice-files/NOT-OD-21-013.html. Retrieved from https://grants.nih.gov/grants/guide/notice-files/NOT-OD-21-013.html
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences of the United States of America, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114
Qian, Y., Berenbaum, S. A., & Gilmore, R. O. (2021, June 18). Individual difference in visual perception (a pregistration). https://aspredicted.org/5iv9a.pdf. Retrieved from https://aspredicted.org/5iv9a.pdf
Qian, Y., Berenbaum, S. A., & Gilmore, R. O. (2022a). Vision contributes to sex differences in spatial cognition and activity interests. Scientific Reports, 12(1), 17623. https://doi.org/10.1038/s41598-022-22269-y
Qian, Y., Berenbaum, S. A., & Gilmore, R. O. (2022b). Vision contributes to sex differences in spatial cognition and activity interests (analysis plan). Retrieved from https://gilmore-lab.github.io/sex-differences-project/analysis/qian-berenbaum-gilmore.html
Tenopir, C., Rice, N. M., Allard, S., Baird, L., Borycz, J., Christian, L., … Sandusky, R. J. (2020). Data sharing, management, use, and reuse: Practices and perceptions of scientists worldwide. PloS One, 15(3), e0229003. https://doi.org/10.1371/journal.pone.0229003