• Reproducibility in neuroscience

https://www.youtube.com/embed/66oNv_DJuPc

• Yes, a significant crisis.
• Yes, a slight crisis.
• No, there is no crisis.
• Don't know.

• Someone else's
• My own

• Published
• Failed to publish

• Selective reporting
• Pressure to publish
• Low power
• Replication in original labs
• Mentoring/oversight
• Methods, code, data unavailable
• Poor design

• Fraud
• Insufficient peer review
• Variability of reagents
• Bad luck

• Protect participant's identities
• Protect from harm/embarrassment

• Much (lab-based) data collected are from Western, Educated Industrialized, Rich, Democratic (WEIRD) populations (Henrich, Heine, & Norenzayan, 2010)

• Direct replications rare, discouraged, hard to publish
• Critical methodological details go unreported

• Data sharing is not universal practice
• Even after a publication has gone to press
• Despite implicit agreement to do so if publishing in certain journals (e.g., American Psychological Association)

• Stapel was Dean of the School of Social and Behavioral Sciences at Tilburg University, teacher of Scientific Ethics course
• Fraud investigation launched when 3 grad students noticed anomalies – duplicate entries in data tables
• Stapel confessed, lost position, gave up Ph.D., wrote a book

• Marc Hauser
• Evolutionary/Comparative Psychologist, Professor at Harvard
• Resigned 2011 after internal investigation found him responsible for research misconduct
• Details see 2012 report by NIH Office of Research Integrity (ORI) and Hauser's response.

• C. Reinhart and K. Rogoff paper, http://www.nber.org/papers/w15639.pdf?new_window=1
  • Omitted data in Excel spreadsheet used to model effect of public debt on economic growth. https://www.bloomberg.com/news/articles/2013-04-18/faq-reinhart-rogoff-and-the-excel-error-that-changed-history
  • Error only found when data were shared.

"This article reports 9 experiments, involving more than 1,000 participants, that test for retroactive influence by "time-reversing" well-established psychological effects so that the individual's responses are obtained before the putatively causal stimulus events occur."

(Bem, 2011)

"We argue that in order to convince a skeptical audience of a controversial claim, one needs to conduct strictly confirmatory studies and analyze the results with statistical tests that are conservative rather than liberal. We conclude that Bem's p values do not indicate evidence in favor of precognition; instead, they indicate that experimental psychologists need to change the way they conduct their experiments and analyze their data."

(E.-J. Wagenmakers, Wetzels, Borsboom, & Maas, 2011)

• Reproducibility Project: Psychology
• Attempt to replicate 100 experimental and correlational studies published in three psychology journals in 2008 using high-powered designs and original materials when available.
• Materials, data, protocols, analysis code shared via the Open Science Framework (OSF) from the Center for Open Science (COS)

• 39/98 (39.7%) replication attempts were successful
• 97% of original studies reported statistically significant results vs. 36% of replications

• Samples not equal
  • Sampling error differences predicts < 100% reproducibility
• Protocols not identical
  • Only 69% of original PIs "endorsed" replication protocol. Replication rate 4x higher (59.7% vs. 15.4%) in studies with endorsed protocol.
• (Collaboration, 2015) "…seriously underestimated reproducibility of psychological science."

• High-powered (90% power to detect effects smaller than original) replications of $n=$21 studies published in Science and Nature between 2010-2015
• Phase I 90% power to detect \(0.75d_{orig}\); Phase II 90% power to detect \(0.5d_{orig}\)

• "…replication sample sizes in stage 1 were about three times as large as the original sample sizes and replication sample sizes in stage 2 were about six times as large as the original sample sizes."
• "…replication and analysis plans were made publicly known on the project website, pre-registered at the Open Science Framework (OSF) and sent to the original authors for feedback and verification prior to data collecion…"

• significant effect in same direction for 13 (62%) studies
• effect size ~50% of the original
• replicability between 12 (57%) and 14 (67%) from complementary replicability indicators
• estimated true-positive rate is 67% in a Bayesian analysis
• prediction market valuations corresponded to 'success' of replication efforts

Goodman et al., 2016

• Same inferences from one or more studies or reanalyses

Goodman et al., 2016

• Same results from independent study

Goodman et al., 2016

• Enough details about materials & methods recorded (& reported)
• Same results with same materials & methods

Goodman et al., 2016

"…psychologists tend to treat other peoples’ theories like toothbrushes; no self-respecting individual wants to use anyone else’s."

Mischel, 2009

"The toothbrush culture undermines the building of a genuinely cumulative science, encouraging more parallel play and solo game playing, rather than building on each other’s directly relevant best work."

Mischel, 2009

• Behaviors
• Brain/physiological states
• Thought processes, feelings, opinions, …

• Open Science Framework (OSF)
  • Reproducibility Project
  • Pre-registration Challenges
  • Pre-print services

• Transparency and opennness promotion (TOP) guidelines, (B. A. Nosek et al., 2015)

• Data, materials, code citation
• Data transparency
• Analytic methods (code) transparency
• Design and analysis transparency
• Preregistration of studies
• Preregistration of analysis plans
• Replication

• Literate Programming
  • RStudio and R Markdown
  • Jupyter notebooks
  • Electronic lab notebooks, talks, reports, articles…
• Software Carpentry
• Version control (e.g., GitHub)
• PSY 511 Spring 2017
• R Bootcamp

• Inter-university Consortium for Political and Social Research (ICPSR)
• Dataverse
• Dryad Data Repository

• Specializes in storing, sharing video
• Video captures behavior unlike other methods, but is identifiable
• Policy framework for sharing identifiable data
  • Permission to share -> builds on informed consent
  • Restricted access for (institutionally) authorized researchers
• Datavyu, free, open-source video coding tool

• All data, materials, code shared
  • when paper goes to press or at end of grant period
• Shared in repositories that encourage data linkage (w/ permission)
  • People, places, times, tasks, behaviors, …

• Commonplace citations of data, materials, code, findings
• Ecosystems for new discovery

This talk was produced on 2018-11-16 in RStudio version 1.1.453 using R Markdown. Information about the R Session that produced the code is as follows:

## R version 3.5.1 (2018-07-02)
## Platform: x86_64-apple-darwin15.6.0 (64-bit)
## Running under: macOS Sierra 10.12.6
## 
## Matrix products: default
## BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
## LAPACK: /Library/Frameworks/R.framework/Versions/3.5/Resources/lib/libRlapack.dylib
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] dplyr_0.7.6   ggplot2_3.0.0
## 
## loaded via a namespace (and not attached):
##  [1] Rcpp_0.12.18     pillar_1.3.0     compiler_3.5.1   plyr_1.8.4      
##  [5] highr_0.7        bindr_0.1.1      tools_3.5.1      digest_0.6.16   
##  [9] evaluate_0.11    memoise_1.1.0    tibble_1.4.2     gtable_0.2.0    
## [13] pkgconfig_2.0.2  rlang_0.2.2      rstudioapi_0.7   yaml_2.2.0      
## [17] xfun_0.3         bindrcpp_0.2.2   withr_2.1.2      stringr_1.3.1   
## [21] knitr_1.20       devtools_1.13.6  rprojroot_1.3-2  grid_3.5.1      
## [25] tidyselect_0.2.4 glue_1.3.0       R6_2.2.2         jpeg_0.1-8      
## [29] rmarkdown_1.10   purrr_0.2.5      magrittr_1.5     backports_1.1.2 
## [33] scales_1.0.0     htmltools_0.3.6  assertthat_0.2.0 colorspace_1.3-2
## [37] labeling_0.3     tinytex_0.8      stringi_1.2.4    lazyeval_0.2.1  
## [41] munsell_0.5.0    crayon_1.3.4