Exercise 04: Replication

Dates

Due: Thursday, March 30, 2023.

Goals

Replication is one standard among many that can be used to evaluate scientific findings. In this exercise, you will evaluate a published scientific finding on the basis of whether it 1) replicates prior work and 2) could be replicated by someone else.

Assignment

Choose a published scientific paper on a topic you find interesting and important.

Does the paper:

  • Report on whether data reported replicate a prior finding?
  • Does the paper embrace any of the recommendations by Begley (2013) (e.g., blinding, replication or repetition of key effects, etc.).
  • Does the paper implement any of the recommendations from the Munafò et al. (2017) paper?
  • Does the paper report effect sizes, sample sizes, and statistical power?
  • Was the paper pre-registered?
  • What is the TOP Factor for the journal? If none, report 0.
  • Does the paper share data? If so, where? Could you get access to the data?
  • Does the paper share any statistical or computer code used to analyze the data or make figures? If so, where? Could you access the code?
  • Does the paper share sufficient information about the measures used (e.g., a protocol, computer code to generate computer tasks, specific survey questions, or other information needed to replicate the tasks or measures) so that an independent third party like yourself could replicate the study?

If you had to give the author(s) of your paper a grade for replicability (A, B, C, D, F), what grade would you give it and why?

Submit

In a short (2-3 pp double-spaced report) discuss the paper’s finding and why you think it is interesting and important. Then, evaluate the paper from the perspective of “replicability” by answering the questions in the previous section.

References

Begley, C Glenn. 2013. “Six Red Flags for Suspect Work.” Nature 497 (7450): 433–34. https://doi.org/10.1038/497433a.
Munafò, Marcus R., Brian A. Nosek, Dorothy V. M. Bishop, Katherine S. Button, Christopher D. Chambers, Nathalie Percie du Sert, Uri Simonsohn, Eric-Jan Wagenmakers, Jennifer J. Ware, and John P. A. Ioannidis. 2017. “A Manifesto for Reproducible Science.” Nature Human Behaviour 1 (January): 0021. https://doi.org/10.1038/s41562-016-0021.