• Thank you to NICHD, NIMH, NIDA, NIH OD, NSF, the Alfred P. Sloan Foundation, the James S. McDonnell Foundation, the LEGO Foundation, and the John S. Templeton Foundation

• Prelude
• Some questions to ponder
• Discuss (Errington, Denis, Perfito, Iorns, & Nosek, 2021)
• Discuss (Munafò et al., 2017)
• Discuss (Silberzahn et al., 2018)
• Discuss (Gilmore, Kennedy, & Adolph, 2018)

• Tools for reproducible science
• An open science future

(NYU Health Sciences Library, 2013)

• 100%
• 90%
• 70%
• 50%
• 30%

• Yes, a significant crisis
• Yes, a slight crisis
• No crisis
• Don’t know

• Enough details about materials & methods recorded (& reported)
• Same results with same materials & methods

(Goodman, Fanelli, & Ioannidis, 2016)

• Same results from independent study

(Goodman et al., 2016)

• Same inferences from one or more studies or reanalyses

(Goodman et al., 2016)

• universalism: scientific validity is independent of sociopolitical status/personal attributes of its participants
• communalism: common ownership of scientific goods (intellectual property)
• disinterestedness: scientific institutions benefit a common scientific enterprise, not specific individuals
• organized skepticism: claims should be exposed to critical scrutiny before being accepted

“…psychologists tend to treat other peoples’ theories like toothbrushes; no self-respecting individual wants to use anyone else’s.”

(Mischel, 2011)

“The toothbrush culture undermines the building of a genuinely cumulative science, encouraging more parallel play and solo game playing, rather than building on each other’s directly relevant best work.”

(Mischel, 2011)

“We conducted the Reproducibility Project: Cancer Biology to investigate the replicability of preclinical research in cancer biology. The initial aim of the project was to repeat 193 experiments from 53 high-impact papers, using an approach in which the experimental protocols and plans for data analysis had to be peer reviewed and accepted for publication before experimental work could begin. However, the various barriers and challenges we encountered while designing and conducting the experiments meant that we were only able to repeat 50 experiments from 23 papers…”

(Errington et al., 2021)

“…First,…the data needed to compute effect sizes and conduct power analyses was publicly accessible for just 4 of 193 experiments. Moreover, despite contacting the authors of the original papers, we were unable to obtain these data for 68% of the experiments…”

(Errington et al., 2021)

“Second, none of the 193 experiments were described in sufficient detail in the original paper to enable us to design protocols to repeat the experiments, so we had to seek clarifications from the original authors. While authors were extremely or very helpful for 41% of experiments, they were minimally helpful for 9% of experiments, and not at all helpful (or did not respond to us) for 32% of experiments…”

(Errington et al., 2021)

“Third, once experimental work started, 67% of the peer-reviewed protocols required modifications to complete the research and just 41% of those modifications could be implemented.”

(Errington et al., 2021)

“…Cumulatively, these three factors limited the number of experiments that could be repeated. This experience draws attention to a basic and fundamental concern about replication – it is hard to assess whether reported findings are credible.”

(Errington et al., 2021)

“Assuming a realistic range of prior probabilities for null hypotheses, false report probability is likely to exceed 50% for the whole literature.”

(Szucs & Ioannidis, 2017)

“Twenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players.”

(Silberzahn et al., 2018)

“Here, we have demonstrated that as a result of researchers’ choices and assumptions during analysis, variation in estimated effect sizes can emerge even when analyses use the same data. ”

(Silberzahn et al., 2018)

“These findings suggest that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions.”

(Silberzahn et al., 2018)

“The best defense against subjectivity in science is to expose it.”

(Silberzahn et al., 2018)

“Transparency in data, methods, and process gives the rest of the community opportunity to see the decisions, question them, offer alternatives, and test these alternatives in further research.”

(Silberzahn et al., 2018)

• Build upon existing (secondary) data and collect (much) larger samples
• Share protocols and complete methods (consider video)
• Share data (FAIRly)
• Script analyses and share analysis code
• Preregister data analyses
• Use version control

• Data
• Analysis code
• Displays, materials
• Procedure manuals

• With ethics board/IRB approval
• With participant permission

• Lab website vs.
• Supplemental information with journal article vs.
• Data repository

• Open Science Framework (OSF)
• ICPSR
• Databrary
• OpenNeuro

• Paper goes out for review or is published
  • Some journals in some fields require post-acceptance “verification”
• Grant ends
• Never

• thesis_new.docx
• thesis_new.new.docx
• thesis_new.new.final.docx

vs.

• thesis_2019-01-15v01.docx
• thesis_2019-01-15v02.docx
• thesis_2019-01-16v01.docx

• Used in large-scale software engineering
• svn, bitbucket, git
• GitHub

• Every project gets a repository
• Work locally, commit (save & increment version), push to GitHub
• Talks, classes, software, analyses, web sites

Data should be…

• Findable
• Accessible
• Interoperable
• Reusable

(Wilkinson et al., 2016)

• Data in interoperable formats (.txt or .csv)
• Scripted, automated = minimize human-dependent steps.
• Well-documented
• Kind to your future (forgetful) self
• Transparent to me & colleagues == transparent to others

• SPSS, SAS, R, Python
• Jupyter notebooks
• R Markdown via RStudio

# Import/gather data

# Clean data

# Visualize data

# Analyze data

# Report findings

# Import data
my_data <- read.csv("path/2/data_file.csv")

# Clean data
my_data$gender <- tolower(my_data$gender) # make lower case
...

# Import data
source("R/Import_data.R") # source() runs scripts, loads functions

# Clean data
source("R/Clean_data.R")

# Visualize data
source("R/Visualize_data.R")
...

• Great! How are you sharing these files?
• (And how much would SPSS cost you if you had to buy it yourself?)

• Great! Let’s talk about R Markdown

• Add-on package to R, developed by the RStudio team
• Combine text, code, images, video, equations into one document
• Render into PDF, MS Word, HTML (web page or site, slides, a blog, or even a book)

• R Markdown documentation; online tutorial; Mike Frank and Chris Hartgerink’s tutorial
• Similar to Mathematica notebooks, Jupyter notebooks

x <- rnorm(n = 100, mean = 0, sd = 1)
hist(x)

The mean is -0.0480519, the range is [-2.8958845, 1.9151122].

• Web pages | Rmd
• Talks | Rmd
• Project notes
• Documentation: HTML | Rmd

• Scriptable figures: PDF | Rmd
• Scriptable timelines from Datavyu files: Example | Rmd
• Papers: PDF | Rmd

• Jupyter notebooks, Jupyter Lab
• Initially Python-focused, but has broadened to include other languages

https://cos.io/rr/

• Nosek: “Don’t fool yourself” (“…and you are the easiest to fool” – R. Feynmann)
• Separate confirmatory from exploratory analyses
• Confirmatory (hypothesis-driven): p-hacking matters
• Exploratory: p-values hard(er) to interpret

• OSF: http://help.osf.io/m/registrations/l/524205-register-your-project
• AsPredicted.org

• Susan Goldin-Meadow (skeptic), “Why pregistration makes me nervous”

• Mike Frank (former skeptic, now advocate), “Pregister everything”

• Preregistration can constrain the data analysis process, but…

“To investigate whether,in psychology, preregistration lives up to that potential, we focused on all article spublished in Psychological Science with a preregistered badge between February 2015 and November 2017, and assessed the adherence to their corresponding preregistration plans. We observed deviations from the plan in all studies, and, more importantly, in all but one study, at least one of these deviations was not fully disclosed.”

Claesen et al., 2019

“…The mean effect size (r) of the replication effects…was half the magnitude of the mean effect size of the original effects…”

(Collaboration, 2015)

“…39% of effects were subjectively rated to have replicated the original result…”

(Collaboration, 2015)

https://80000hours.org/psychology-replication-quiz/

“The advancement of detailed and diverse knowledge about the development of the world’s children is essential for improving the health and well-being of humanity…”

(SRCD, 2019)

“We regard scientific integrity, transparency, and openness as essential for the conduct of research and its application to practice and policy…”

(SRCD, 2019)

“…the principles of human subject research require an analysis of both risks and benefits…such an analysis suggests that researchers may have a positive duty to share data in order to maximize the contribution that individual participants have made.”

(Brakewood & Poldrack, 2013)

This talk was produced on 2022-04-05 in RStudio using R Markdown and the ioslides framework. The code and materials used to generate the slides may be found at https://psu-psychology.github.io/psy-543-clinical-research-methods-2022/. Information about the R Session that produced the code is as follows:

## R version 4.1.2 (2021-11-01)
## Platform: x86_64-apple-darwin17.0 (64-bit)
## Running under: macOS Monterey 12.3
## 
## Matrix products: default
## LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## loaded via a namespace (and not attached):
##  [1] digest_0.6.29   mime_0.12       R6_2.5.1        jsonlite_1.7.2 
##  [5] magrittr_2.0.1  evaluate_0.14   highr_0.9       rlang_0.4.12   
##  [9] stringi_1.7.6   jquerylib_0.1.4 bslib_0.3.1     rmarkdown_2.11 
## [13] tools_4.1.2     stringr_1.4.0   xfun_0.29       yaml_2.2.1     
## [17] fastmap_1.1.0   compiler_4.1.2  htmltools_0.5.2 knitr_1.37     
## [21] sass_0.4.0