Preregistration

2024-11-11 Mon

Rick Gilmore

Prelude

McCutcheon (2019)

Overview

Announcement

Today

Preregistration

  • Read
    • Nosek, Ebersole, DeHaven, & Mellor (2018)
    • Ledgerwood (2018) or Goldin-Meadow (2016)
    • (Optional) Claesen, Gomes, Tuerlinckx, & Vanpaemel (2021)
  • Explore

Preregistration

Nosek, B. A., Ebersole, C. R., DeHaven, A. C. & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences of the United States of America, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114

Remember HARKing?

Figure 1 from Munafò et al. (2017)

Abstract

Progress in science relies in part on generating hypotheses with existing observations and testing hypotheses with new observations. This distinction between postdiction and prediction is appreciated conceptually but is not respected in practice. Mistaking generation of postdictions with testing of predictions reduces the credibility of research findings…

Nosek et al. (2018)

However, ordinary biases in human reasoning, such as hindsight bias, make it hard to avoid this mistake. An effective solution is to define the research questions and analysis plan before observing the research outcomes—a process called preregistration…

Nosek et al. (2018)

Preregistration distinguishes analyses and outcomes that result from predictions from those that result from postdictions. A variety of practical strategies are available to make the best possible use of preregistration in circumstances that fall short of the ideal application, such as when the data are preexisting.

Nosek et al. (2018)

Services are now available for preregistration across all disciplines, facilitating a rapid increase in the practice. Widespread adoption of preregistration will increase distinctiveness between hypothesis generation and hypothesis testing and will improve the credibility of research findings.

Nosek et al. (2018)

Key ideas

  • Pre-diction vs. post-diction
  • Confirmatory vs. Exploratory

Types

  • Preregistration (of analysis plans)
  • Registered reports, Chambers (n.d.); Center for Open Science (n.d.)(https://www.cos.io/initiatives/registered-reports)

Illustration about registered reports from Center for Open Science (n.d.)

Commentary about

  • Ledgerwood (2018)
    • Preregistering theoretical predictions \(\neq\) preregistering analysis plans
  • Goldin-Meadow (2016)
    • Could stifle discovery
    • Not applicable to all kinds of research
  • Rubin & Donkin (2022)

Exists on a continuum

Figure 2 from Hardwicke & Wagenmakers (2023)

Is it working?

Figure 1 from Claesen et al. (2021). Flowchart of assessment of preregistered studies.

Figure 2 from Claesen et al. (2021). Tile plot of the assessment of each methodological aspect per preregistration plan. Only the 27 studies that were accessible and included the minimal number of methodological details required for our adherence assessment are shown.

Figure 3 from Claesen et al. (2021). An overview of adherence per methodological aspect.

Author-reported reasons for deviations

  • Suggestions from reviewers
  • Did not preregister or report details because they were thought redundant or irrelevant
  • Lack of consensus about what should be disclosed
  • Small discrepancy with the preregistered sample size was small and not really a deviation.

Conclusions

Our work suggests that it is not obvious for preregistration to live up to its promise of making research results more transparent and more interpretable in the manuscript. This insight calls for better standards in preregistering and reporting preregistered research…

It should, however, not be misinterpreted as a plea against the practice of preregistration. Neither does our work suggest that the results of the preregistered studies we assessed should not be trusted. We do not, and cannot, claim that the observed deviations have been deliberately unreported or constitute evidence of the exploitation of researcher degrees of freedom…

The process of preregistering and reporting in accordance with the preregistration is tricky, and we realize that, given its relatively young status in psychology, the field collectively needs to go through a learning phase.

Claesen et al. (2021)

Your thoughts?

  • Are you surprised that studies differ from their preregistered plans? Why or why not?
  • Does preregistration make you nervous?
  • How do thesis/dissertation/grant proposals differ from preregistration?
  • What practice(s) would reduce deviations or alleviate skeptics’ concerns?

Changes across time

Figure 7 from Ferguson et al. (2023)

Figure 7 from Ferguson et al. (2023)

My experience

  • Paper: Qian, Berenbaum, & Gilmore (2022a); preregistration Qian, Berenbaum, & Gilmore (2021).
  • Published R Markdown document (in HTML) with entire workflow Qian, Berenbaum, & Gilmore (2022b).

Cautionary tale

Enserink (2024)

“The concerns relate to lack of transparency and misstatement of the hypotheses and predictions the reported metastudy was designed to test; lack of preregistration for measures and analyses supporting the titular claim (against statements asserting preregistration in the published article); selection of outcome measures and analyses with knowledge of the data; and incomplete reporting of data and analyses.”

Protzko et al. (2024)

Next time

Data sharing

Resources

References

PSYCH 490.012 • Fall 2024

Adolph, K. (2016). Video as data. APS Observer, 29(3). Retrieved from https://www.psychologicalscience.org/observer/video-as-data
Center for Open Science. (n.d.). Registered reports. https://www.cos.io/initiatives/registered-reports. Retrieved from https://www.cos.io/initiatives/registered-reports
Chambers, C. (n.d.). Registered reports: What are they and why are they important? https://royalsociety.org/blog/2016/11/registered-reports-what-are-they-and-why-are-they-important/. Retrieved from https://royalsociety.org/blog/2016/11/registered-reports-what-are-they-and-why-are-they-important/
Claesen, A., Gomes, S., Tuerlinckx, F., & Vanpaemel, W. (2021). Comparing dream to reality: An assessment of adherence of the first generation of preregistered studies. Royal Society Open Science, 8(211037). https://doi.org/10.1098/rsos.211037
Enserink, M. (2024, September). “We are embarrassed”: Scientific rigor proponents retract paper on benefits of scientific rigor. https://www.science.org/content/article/we-are-embarrassed-scientific-rigor-proponents-retract-paper-benefits-scientific-rigor.
Ferguson, J., Littman, R., Christensen, G., Paluck, E. L., Swanson, N., Wang, Z., … Pezzuto, J.-H. (2023). Survey of open science practices and attitudes in the social sciences. Nature Communications, 14(1), 5401. https://doi.org/10.1038/s41467-023-41111-1
Gilmore, R. O., & Adolph, K. E. (2017). Video can make behavioural research more reproducible. Nature Human Behavior, 1. https://doi.org/10.1038/s41562-017-0128
Goldin-Meadow, S. (2016). Why preregistration makes me nervous. APS Observer, 29(7). Retrieved from https://www.psychologicalscience.org/observer/why-preregistration-makes-me-nervous
Hardwicke, T. E., & Wagenmakers, E.-J. (2023). Reducing bias, increasing transparency and calibrating confidence with preregistration. Nature Human Behaviour, 7(1), 15–26. https://doi.org/10.1038/s41562-022-01497-2
Houtkoop, B. L., Chambers, C., Macleod, M., Bishop, D. V. M., Nichols, T. E., & Wagenmakers, E.-J. (2018). Data sharing in psychology: A survey on barriers and preconditions. Advances in Methods and Practices in Psychological Science, 2515245917751886. https://doi.org/10.1177/2515245917751886
Ledgerwood, A. (2018). The preregistration revolution needs to distinguish between predictions and analyses. Proceedings of the National Academy of Sciences of the United States of America, 115(45), E10516–E10517. https://doi.org/10.1073/pnas.1812592115
McCutcheon, J. (2019, May). Write it down. Youtube. Retrieved from https://www.youtube.com/watch?v=mYjIkxD5u0w
Meyer, M. N. (2018). Practical tips for ethical data sharing. Advances in Methods and Practices in Psychological Science, 2515245917747656. https://doi.org/10.1177/2515245917747656
Mischel, W. (2011). Becoming a cumulative science. APS Observer, 22(1). Retrieved from https://www.psychologicalscience.org/observer/becoming-a-cumulative-science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Sert, N. P. du, … Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021. https://doi.org/10.1038/s41562-016-0021
National Institutes of Health. (n.d.). NOT-OD-21-013: Final NIH policy for data management and sharing. https://grants.nih.gov/grants/guide/notice-files/NOT-OD-21-013.html. Retrieved from https://grants.nih.gov/grants/guide/notice-files/NOT-OD-21-013.html
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences of the United States of America, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114
Protzko, J., Krosnick, J., Nelson, L., Nosek, B. A., Axt, J., Berent, M., … Schooler, J. W. (2024). Retraction note: High replicability of newly discovered social-behavioural findings is achievable. Nature Human Behaviour, 8(10), 2067. https://doi.org/10.1038/s41562-024-01997-3
Qian, Y., Berenbaum, S. A., & Gilmore, R. O. (2021, June 18). Individual difference in visual perception (a pregistration). https://aspredicted.org/5iv9a.pdf. Retrieved from https://aspredicted.org/5iv9a.pdf
Qian, Y., Berenbaum, S. A., & Gilmore, R. O. (2022a). Vision contributes to sex differences in spatial cognition and activity interests. Scientific Reports, 12(1), 17623. https://doi.org/10.1038/s41598-022-22269-y
Qian, Y., Berenbaum, S. A., & Gilmore, R. O. (2022b). Vision contributes to sex differences in spatial cognition and activity interests (analysis plan). Retrieved from https://gilmore-lab.github.io/sex-differences-project/analysis/qian-berenbaum-gilmore.html
Rubin, M., & Donkin, C. (2022). Exploratory hypothesis tests can be more compelling than confirmatory hypothesis tests. Philosophical Psychology, 1–29. https://doi.org/10.1080/09515089.2022.2113771
Tenopir, C., Rice, N. M., Allard, S., Baird, L., Borycz, J., Christian, L., … Sandusky, R. J. (2020). Data sharing, management, use, and reuse: Practices and perceptions of scientists worldwide. PloS One, 15(3), e0229003. https://doi.org/10.1371/journal.pone.0229003