Open science tools

2024-11-20 Wed

Rick Gilmore

Overview

In the news

Tyson (2024)

Tyson (2024)

Tyson (2024)

Open science perspective

Engber (2024)

Ioannidis

John Ioannidis, MD, DSc

…Let me start with disclosures. My main conflict of interest is that I try to be a scientist. This means I am probably biased and often wrong, but hopefully not totally resistant to the possibility of getting corrected.

Ioannidis (2024)

Let me also make some preemptive comments. First and foremost, science is the best thing that can happen to humans, and research should be supported with heightened commitments. You have probably heard this too many times, but it is worth repeating.

Ioannidis (2024)

However, most research done to date has used nonreproducible, nontransparent, and suboptimal research practices. Science is becoming more massive and more complex. Scientific publications…are mostly advertisements (“trust me, this research was done”); raw data and experimental materials and algorithms are not usually shared. Moreover, our reward systems in academia and science are aligned with nonreproducible, nontransparent, and suboptimal research practices. Can we do better?

Ioannidis (2024)

Announcements

Last time…

Your turn

  • What do students want researchers to share?
  • What are the pros and cons of active vs. post hoc data curation?

Today

Open science tools

  • Read
    • Kathawalla, Silverstein, & Syed (2021)
    • Chopik, Bremner, Defever, & Keller (2018)
    • (Optional) Crüwell et al. (2019)
  • Explore
    • “FORRT - Framework for Open and Reproducible Research Training” (n.d.)
  • Complete (optional)

Kathawalla et al. (2021)

This article provides a roadmap to assist graduate students and their advisors to engage in open science practices. We suggest eight open science practices that novice graduate students could begin adopting today. The topics we cover include journal clubs, project workflow, preprints, reproducible code, data sharing, transparent writing, preregistration, and registered reports.

Kathawalla et al. (2021)

To address concerns about not knowing how to engage in open science practices, we provide a difficulty rating of each behavior (easy, medium, difficult), present them in order of suggested adoption, and follow the format of what, why, how, and worries.

Kathawalla et al. (2021)

We give graduate students ideas on how to approach conversations with their advisors/collaborators, ideas on how to integrate open science practices within the graduate school framework, and specific resources on how to engage with each behavior.

Kathawalla et al. (2021)

We emphasize that engaging in open science behaviors need not be an all or nothing approach, but rather graduate students can engage with any number of the behaviors outlined.

Kathawalla et al. (2021)

Figure 1 from (Kathawalla et al., 2021)

Chopik et al. (2018)

Over the past 10 years, crises surrounding replication, fraud, and best practices in research methods have dominated discussions in the field of psychology. However, no research exists examining how to communicate these issues to undergraduates and what effect this has on their attitudes toward the field.

Chopik et al. (2018)

We developed and validated a 1-hr lecture communicating issues surrounding the replication crisis and current recommendations to increase reproducibility. Pre- and post-lecture surveys suggest that the lecture serves as an excellent pedagogical tool.

Chopik et al. (2018)

Following the lecture, students trusted psychological studies slightly less but saw greater similarities between psychology and natural science fields. We discuss challenges for instructors taking the initiative to communicate these issues to undergraduates in an evenhanded way.

Chopik et al. (2018)

Reproducibility notes

  • Survey questions in article
  • OSF project for materials: Defever, Chopik, Keller, & Bremner (2016)
  • Data shared, but not in an easily reproducible form

What do you think?

Next time

Work session: Data sharing & Final Projects

Resources

References

PSYCH 490.012 • Fall 2024

Chopik, W. J., Bremner, R. H., Defever, A. M., & Keller, V. N. (2018). How (and whether) to teach undergraduates about the replication crisis in psychological science. Teaching of Psychology, 45(2), 158–163. https://doi.org/10.1177/0098628318762900
Crüwell, S., Doorn, J. van, Etz, A., Makel, M. C., Moshontz, H., Niebaum, J. C., … Schulte-Mecklenbeck, M. (2019). Seven easy steps to open science. Zeitschrift für Psychologie, 227(4), 237–248. https://doi.org/10.1027/2151-2604/a000387
Defever, A. M., Chopik, W. J., Keller, V. N., & Bremner, R. (2016, August). Best practices pedagogy. Center For Open Science. Retrieved from https://osf.io/mh9pe/
Engber, D. (2024, November). The business-school scandal that just keeps getting bigger. https://www.theatlantic.com/magazine/archive/2025/01/business-school-fraud-research/680669/.
FORRT - Framework for Open and Reproducible Research Training. (n.d.). https://forrt.org/. Retrieved from https://forrt.org/
Ioannidis, J. P. (2024). Transparency, bias, and reproducibility across science: A meta-research view. The Journal of Clinical Investigation, 134(22). https://doi.org/10.1172/JCI181923
Kathawalla, U.-K., Silverstein, P., & Syed, M. (2021). Easing into open science: A guide for graduate students and their advisors. Collabra. Psychology, 7(1). https://doi.org/10.1525/collabra.18684
Tyson, A. (2024, November). Public trust in scientists and views on their role in policymaking. https://www.pewresearch.org/science/2024/11/14/public-trust-in-scientists-and-views-on-their-role-in-policymaking/.