• Thank you to NICHD, NIMH, NIDA, NIH, NSF, the Alfred P. Sloan Foundation, the James S. McDonnell Foundation, and the John S. Templeton Foundation

• Prelude
• Some questions to ponder
• Discuss Munafò et al. 2017
• Discuss Silberzahn et al. 2018
• Discuss Gilmore et al. 2018

• Tools for reproducible science
• An open science future

https://www.youtube.com/embed/66oNv_DJuPc

• 100%
• 90%
• 70%
• 50%
• 30%

• Yes, a significant crisis
• Yes, a slight crisis
• No crisis
• Don’t know

• Enough details about materials & methods recorded (& reported)
• Same results with same materials & methods

Goodman et al., 2016

• Same results from independent study

(Goodman et al., 2016)

• Same inferences from one or more studies or reanalyses

Goodman et al., 2016

• universalism: scientific validity is independent of sociopolitical status/personal attributes of its participants
• communalism: common ownership of scientific goods (intellectual property)
• disinterestedness: scientific institutions benefit a common scientific enterprise, not specific individuals
• organized skepticism: claims should be exposed to critical scrutiny before being accepted

“…psychologists tend to treat other peoples’ theories like toothbrushes; no self-respecting individual wants to use anyone else’s.”

Mischel 2009

“The toothbrush culture undermines the building of a genuinely cumulative science, encouraging more parallel play and solo game playing, rather than building on each other’s directly relevant best work.”

Mischel, 2009

“Twenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players.”

“Here, we have demonstrated that as a result of researchers’ choices and assumptions during analysis, variation in estimated effect sizes can emerge even when analyses use the same data. ”

“These findings suggest that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions.”

“The best defense against subjectivity in science is to expose it.”

“Transparency in data, methods, and process gives the rest of the community opportunity to see the decisions, question them, offer alternatives, and test these alternatives in further research.”

• Data
• Analysis code
• Displays, materials
• Procedure manuals

• With ethics board/IRB approval
• With participant permission

• Lab website vs.
• Supplemental information with journal article vs.
• Data repository

• Open Science Framework (OSF)
• ICPSR
• Databrary
• OpenNeuro

• Paper goes out for review or is published
  • Some journals in some fields require post-acceptance “verification”
• Grant ends
• Never

• thesis_new.docx
• thesis_new.new.docx
• thesis_new.new.final.docx

vs.

• thesis_2019-01-15v01.docx
• thesis_2019-01-15v02.docx
• thesis_2019-01-16v01.docx

• Used in large-scale software engineering
• svn, bitbucket, git
• GitHub

• Every project gets a repository
• Work locally, commit (save & increment version), push to GitHub
• Talks, classes, software, analyses, web sites

Data should be…

• Findable
• Accessible
• Interoperable
• Reusable

Wilkinson et al. 2016

• Data in interoperable formats (.txt or .csv)
• Scripted, automated = minimize human-dependent steps.
• Well-documented
• Kind to your future (forgetful) self
• Transparent to me & colleagues == transparent to others

• SPSS, SAS, R, Python
• Jupyter notebooks
• Rmarkdown via RStudio

# Import/gather data

# Clean data

# Visualize data

# Analyze data

# Report findings

# Import data
my_data <- read.csv("path/2/data_file.csv")

# Clean data
my_data$gender <- tolower(my_data$gender) # make lower case
...

# Import data
source("R/Import_data.R") # source() runs scripts, loads functions

# Clean data
source("R/Clean_data.R")

# Visualize data
source("R/Visualize_data.R")
...

• Great! How are you sharing these files?
• (And how much would SPSS cost you if you had to buy it yourself?)

• Great! Let’s talk about R Markdown

• Add-on package to R, developed by the RStudio team
• Combine text, code, images, video, equations into one document
• Render into PDF, MS Word, HTML (web page or site, slides, a blog, or even a book)

• R Markdown documentation; online tutorial; Mike Frank and Chris Hartgerink’s tutorial
• Similar to Mathematica notebooks, Jupyter notebooks

x <- rnorm(n = 100, mean = 0, sd = 1)
hist(x)

The mean is 0.2717844, the range is [-2.2152884, 2.7838795].

• Web pages | Rmd
• Talks | Rmd
• Project notes
• Documentation: HTML | Rmd

• Scriptable figures: PDF | Rmd
• Scriptable timelines from Datavyu files: Example | Rmd
• Papers: PDF | Rmd

https://cos.io/rr/

• Nosek: “Don’t fool yourself”
• Separate confirmatory from exploratory analyses
• Confirmatory (hypothesis-driven): p-hacking matters
• Exploratory: p-values hard(er) to interpret

• OSF: http://help.osf.io/m/registrations/l/524205-register-your-project
• AsPredicted.org

• Susan Goldin-Meadow (skeptic), “Why pregistration makes me nervous”

• Mike Frank (former skeptic, now advocate), “Pregister everything”

• Preregistration can constrain the data analysis process, but…

“To investigate whether,in psychology, preregistration lives up to that potential, we focused on all article spublished in Psychological Science with a preregisteredbadge between February 2015 and November 2017, and assessed the adherence to their corresponding preregistration plans. We observed deviations from the plan in all studies, and, more importantly, in all but one study, at least one of these deviations was not fully disclosed.”

Claesen et al., 2019

“…The mean effect size (r) of the replication effects…was half the magnitude of the mean effect size of the original effects…”

Open Science Collaboration, 2015

“…39% of effects were subjectively rated to have replicated the original result…”

Open Science Collaboration, 2015

https://80000hours.org/psychology-replication-quiz/

“The advancement of detailed and diverse knowledge about the development of the world’s children is essential for improving the health and well-being of humanity…”

SRCD Task Force on Scientific Integrity and Openness

“We regard scientific integrity, transparency, and openness as essential for the conduct of research and its application to practice and policy…”

SRCD Task Force on Scientific Integrity and Openness

“…the principles of human subject research require an analysis of both risks and benefits…such an analysis suggests that researchers may have a positive duty to share data in order to maximize the contribution that individual participants have made.”

Brakewood & Poldack, 2013

This talk was produced on 2021-04-27 in RStudio using R Markdown and the reveal.JS framework. The code and materials used to generate the slides may be found at https://psu-psychology.github.io/psy-543-clinical-research-methods-2021/. Information about the R Session that produced the code is as follows:

## R version 3.6.2 (2019-12-12)
## Platform: x86_64-apple-darwin15.6.0 (64-bit)
## Running under: macOS Catalina 10.15.7
## 
## Matrix products: default
## BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
## LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## loaded via a namespace (and not attached):
##  [1] compiler_3.6.2  magrittr_1.5    htmltools_0.5.0 tools_3.6.2    
##  [5] yaml_2.2.1      tinytex_0.25    stringi_1.5.3   rmarkdown_2.7  
##  [9] highr_0.8       knitr_1.29      stringr_1.4.0   digest_0.6.25  
## [13] xfun_0.17       mime_0.9        rlang_0.4.7     evaluate_0.14