Evaluating Rigor, Reproducibility, & Transparency
WarningUnder construction
This page is in a very early and rough state.
Do not rely upon it.
1 About
This page describes a rubric for evaluating the rigor, reproducibility, and transparency of empirical articles in cognitive development.
Gilmore is considering whether students would like to add an evaluation of these factors to the classwork we do together on the days when students present.
2 Components
2.1 Reproducibility
2.1.1 Can the paper’s main reported findings be reproduced exactly?
- Are the data shared?
- Is any analysis or visualization code shared?
- If 1. and 2. are true, does the analysis and visualization code run?
- If 3., are the results the same as those the paper reports?
2.2 Transparency
2.2.1 Was the study pre-registered?
2.3 Recombinability
2.3.1 Does the paper report effect sizes or information that can be used to compute effect sizes so that these results can be combined with other reports in a meta-analysis?
2.4 Replicability
2.4.1 Have exact replications of the paper’s main findings been published?
2.4.2 Has a systematic review or meta-analysis been published?
2.4.3 How thorough is the methods section?
2.4.3.1 Displays
- If computer-generated displays were used, is the computer code to generate them shared?
- Are videos of the displays shared?