August 21-25

Monday, August 21

Introduction to the course: Why trust science?

Wednesday, August 23

Don’t Fool Yourself

Friday, August 25

How to read a scientific paper

August 28-31

Monday, August 28

How science works (or should)

Wednesday, August 30

Scientific norms and counter-norms

Friday, September 1

Work session: Norms and counter-norms

September 4-8

Monday, September 4

Labor Day: NO CLASS

Wednesday, September 6

Adherence to norms and counter-norms

Friday, September 8

A replication crisis (or not)

September 11-15

Monday, September 11

Replication attempt: The “Lady Macbeth Effect”

Wednesday, September 13

Replication attempt: Priming effect

Friday, September 15

Work session: Replication with R, R Markdown, & Quarto

September 18-22

Monday, September 18

Replication in cancer biology

Wednesday, September 20

What R we talking about?

Friday, September 22

  • Catch-up

Work session: Scientific integrity

September 25-29

Monday, September 25


Wednesday, September 27

Fraud & misconduct

Friday, September 29


On Zoom: Check-in for attendance. Join from anywhere convenient to you.

October 2-6

Monday, October 2

Retraction and scientific integrity

Wednesday, October 4

Questionable research practices

Friday, October 6

More on questionable research practices

October 9-13

Monday, October 9

Prevalence of QRPs

Wednesday, October 11

File drawer effect

Friday, October 13


October 16-20

Monday, October 16


Wednesday, October 18


Friday, October 20

Work session: Exercise 05, Replication, Final Projects

October 23-27

Monday, October 23


Wednesday, October 25


Changing journal policies

Friday, October 27

Final project work day

October 30 - November 3

Monday, October 30

Large-scale replication studies

Wednesday, November 1

Meta-analysis & many analysts

Friday, November 3

Final project work day

November 6-10

Monday, November 6


Wednesday, November 8

Data sharing

Friday, November 10

Work session: Data sharing

November 13-17

Monday, November 13

Materials, code, & protocol sharing

Wednesday, November 15

Open science tools

Friday, November 17

Final project work day

November 20-24 Thanksgiving Break

November 27 - December 1

Monday, November 27

In-class final project work day

Meeting on Zoom:

Wednesday, November 29


You may schedule individual or group meetings to discuss your final project using this link:

Friday, December 1

Project presentations

December 4-8

Monday, December 4

Final project workday

Wednesday, December 6

Project presentations

Friday, December 8

Project presentations

December 11-15

Wednesday, December 13


Bargh, J. A., Chen, M., & Burrows, L. (1996). Automaticity of social behavior: Direct effects of trait construct and stereotype-activation on action. Journal of Personality and Social Psychology, 71(2), 230–244.
Begley, C. G. (2013). Six red flags for suspect work. Nature, 497(7450), 433–434.
Begley, C. G., & Ellis, L. M. (2012). Drug development: Raise standards for preclinical cancer research. Nature, 483(7391), 531–533.
Bhattacharjee, Y. (2013). The mind of a con man. The New York Times. Retrieved from
Brainerd, J., & You, J. (2018). What a massive database of retracted papers reveals about science publishing’s “death penalty.” Science.
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., … Wu, H. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 1.
Carey, M. A., Steiner, K. L., & Petri, W. A., Jr. (2020). Ten simple rules for reading a scientific paper. PLoS Computational Biology, 16(7), e1008032.
Carney, D. R., Cuddy, A. J. C., & Yap, A. J. (2010). Power posing: Brief nonverbal displays affect neuroendocrine levels and risk tolerance. Psychological Science, 21(10), 1363–1368.
Carpenter, S. (2012). Harvard psychology researcher committed fraud, US investigation concludes. Science, 6. Retrieved from
Claesen, A., Gomes, S., Tuerlinckx, F., & Vanpaemel, W. (2021). Comparing dream to reality: An assessment of adherence of the first generation of preregistered studies. Royal Society Open Science, 8(211037).
Collaboration, O. S. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.
Crüwell, S., Doorn, J. van, Etz, A., Makel, M. C., Moshontz, H., Niebaum, J. C., … Schulte-Mecklenbeck, M. (2019). Seven easy steps to open science. Zeitschrift für Psychologie, 227(4), 237–248.
Cuddy, A. (2012). Your body language may shape who you are. Retrieved from
Doyen, S., Klein, O., Pichon, C.-L., & Cleeremans, A. (2012). Behavioral priming: It’s all in the mind, but whose mind? PloS One, 7(1), e29081.
Earp, B. D., Everett, J. A. C., Madva, E. N., & Hamlin, J. K. (2014). Out, damned spot: Can the Macbeth effect” be replicated? Basic and Applied Social Psychology, 36(1), 91–98.
Errington, T. M., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021). Challenges for assessing replicability in preclinical cancer biology. eLife, 10, e67995.
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021). Investigating the replicability of preclinical cancer biology. eLife, 10, e71601.
Feynman, R. P. (1974). Cargo cult science. Retrieved from
Fidler, F., & Wilcox, J. (2021). Reproducibility of scientific results. In E. N. Zalta (Ed.), The Stanford Encyclopedia of Philosophy (Summer 2021). Metaphysics Research Lab, Stanford University. Retrieved from
FORRT - framework for open and reproducible research training. (n.d.). Retrieved from
Franco, A., Malhotra, N., & Simonovits, G. (2014). Social science. Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502–1505.
Gilmore, R. O., & Adolph, K. E. (2017). Video can make behavioural research more reproducible. Nature Human Behavior, 1.
Gilmore, R. O., Cole, P. M., Verma, S., Aken, M. A. G., & Worthman, C. M. (2020). Advancing scientific integrity, transparency, and openness in child development research: Challenges and possible solutions. Child Development Perspectives, 14(1), 9–14.
Gilroy, S. P., & Kaplan, B. A. (2019). Furthering open science in behavior analysis: An introduction and tutorial for using GitHub in research. Perspectives on Behavior Science, 42(3), 565–581.
Goldin-Meadow, S. (2016). Why preregistration makes me nervous. APS Observer, 29(7). Retrieved from
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016). What does research reproducibility mean? Science Translational Medicine, 8(341), 341ps12–341ps12.
Houtkoop, B. L., Chambers, C., Macleod, M., Bishop, D. V. M., Nichols, T. E., & Wagenmakers, E.-J. (2018). Data sharing in psychology: A survey on barriers and preconditions. Advances in Methods and Practices in Psychological Science, 2515245917751886.
John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532.
Kardash, C. M., & Edwards, O. V. (2012). Thinking and behaving like scientists: Perceptions of undergraduate science interns and their faculty mentors. Instructional Science, 40(6), 875–899.
Kathawalla, U.-K., Silverstein, P., & Syed, M. (2021). Easing into open science: A guide for graduate students and their advisors. Collabra. Psychology, 7(1).
Ledgerwood, A. (2018). The preregistration revolution needs to distinguish between predictions and analyses. Proceedings of the National Academy of Sciences of the United States of America, 115(45), E10516–E10517.
Levelt, W. J. M., Drenth, P. J. D., & Noort, E. (2012). Flawed science: The fraudulent research practices of social psychologist diederik stapel.; Retrieved from
Macfarlane, B., & Cheng, M. (2008). Communism, universalism and disinterestedness: Re-examining contemporary support among academics for merton’s scientific norms. Journal of Academic Ethics, 6(1), 67–78.
Merton, R. W. (1973). The normative structure of science. In R. K. Merton & N. W. Storer (Eds.), The Sociology of Science: Theoretical and Empirical Investigations (pp. 267–278). The University of Chicago Press.
Meyer, M. N. (2018). Practical tips for ethical data sharing. Advances in Methods and Practices in Psychological Science, 2515245917747656.
Mitroff, I. I. (1974). Norms and counter-norms in a select group of the Apollo moon scientists: A case study of the ambivalence of scientists. American Sociological Review, 39(4), 579–595.
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Sert, N. P. du, … Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021.
National Institutes of Health. (n.d.). NOT-OD-21-013: Final NIH policy for data management and sharing. Retrieved from
Ngiam, W. (2020, April). ReproducibiliTea | simmons, nelson and simonsohn (2011). False-Positive psychology. Youtube. Retrieved from
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., … Yarkoni, T. (2015). Promoting an open research culture. Science, 348(6242), 1422–1425.
Nosek, Brian A., & Bar-Anan, Y. (2012). Scientific utopia i: Opening scientific communication. Psychological Inquiry, 23(3), 217–243.
Nosek, Brian A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences of the United States of America, 115(11), 2600–2606.
Nosek, Brian A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., … Vazire, S. (2022). Replicability, robustness, and reproducibility in psychological science. Annual Review of Psychology, 73(2022), 719–748.
Nuijten, M. B., Hartgerink, C. H. J., Assen, M. A. L. M. van, Epskamp, S., & Wicherts, J. M. (2015). The prevalence of statistical reporting errors in psychology (1985–2013). Behavior Research Methods, 1–22.
Oreskes, N. (2019). Why Trust Science. Princeton University Press.
Ranehill, E., Dreber, A., Johannesson, M., Leiberg, S., Sul, S., & Weber, R. A. (2015). Assessing the robustness of power posing: No effect on hormones and risk tolerance in a large sample of men and women. Psychological Science, 26(5), 653–656.
Ritchie, S. (2020). Science fictions: Exposing fraud, bias, negligence and hype in science (1st ed.). Penguin Random House. Retrieved from
Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641.
Ruben, A. (2016). How to read a scientific paper. Science| AAAS [Internet], 20. Retrieved from
Sagan, C. (1996). The Demon-haunted World: Science as a Candle in the Dark (pp. 200–218). Ballantine Books.
Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., … Nosek, B. A. (2018). Many analysts, one data set: Making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 1(3), 337–356.
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366.
Soska, K. C., Xu, M., Gonzalez, S. L., Herzberg, O., Tamis-LeMonda, C. S., Gilmore, R. O., & Adolph, K. E. (2021). (Hyper)active data curation: A video case study from behavioral science. Journal of Escience Librarianship, 10(3).
SRCD. (2019). Policy on scientific integrity, transparency, and openness | society for research in child development SRCD. Retrieved from
Szucs, D., & Ioannidis, J. P. A. (2017). Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature. PLoS Biology, 15(3), e2000797.
Tenopir, C., Rice, N. M., Allard, S., Baird, L., Borycz, J., Christian, L., … Sandusky, R. J. (2020). Data sharing, management, use, and reuse: Practices and perceptions of scientists worldwide. PloS One, 15(3), e0229003.
Wilson, L. C. (2014, September). Introduction to Meta-Analysis: A guide for the novice. Retrieved from
Zhong, C.-B., & Liljenquist, K. (2006). Washing away your sins: Threatened morality and physical cleansing. Science, 313(5792), 1451–1452.