schedule

August 21-25

Monday, August 21

Introduction to the course: Why trust science?

Wednesday, August 23

Don’t Fool Yourself

Friday, August 25

How to read a scientific paper

August 28-31

Monday, August 28

How science works (or should)

Wednesday, August 30

Scientific norms and counter-norms

Friday, September 1

Work session: Norms and counter-norms

September 4-8

Monday, September 4

Labor Day: NO CLASS

Wednesday, September 6

Adherence to norms and counter-norms

Friday, September 8

A replication crisis (or not)

September 11-15

Monday, September 11

Replication attempt: The “Lady Macbeth Effect”

Wednesday, September 13

Replication attempt: Priming effect

Friday, September 15

Work session: Replication with R, R Markdown, & Quarto

September 18-22

Monday, September 18

Replication in cancer biology

Wednesday, September 20

What R we talking about?

Friday, September 22

  • Catch-up

Work session: Scientific integrity

September 25-29

Monday, September 25

NO CLASS

Wednesday, September 27

Fraud & misconduct

Friday, September 29

P-hacking

On Zoom: https://psu.zoom.us/my/rogilmore. Check-in for attendance. Join from anywhere convenient to you.

October 2-6

Monday, October 2

Retraction and scientific integrity

Wednesday, October 4

Questionable research practices

Friday, October 6

More on questionable research practices

October 9-13

Monday, October 9

Prevalence of QRPs

Wednesday, October 11

File drawer effect

Friday, October 13

NO CLASS MEETING

October 16-20

Monday, October 16

Negligence

Wednesday, October 18

Hype

Friday, October 20

Work session: Exercise 05, Replication, Final Projects

October 23-27

Monday, October 23

Solutions

Wednesday, October 25

Solutions

Changing journal policies

Friday, October 27

Final project work day

October 30 - November 3

Monday, October 30

Large-scale replication studies

Wednesday, November 1

Meta-analysis & many analysts

Friday, November 3

Final project work day

November 6-10

Monday, November 6

Preregistration

Wednesday, November 8

Data sharing

Friday, November 10

Work session: Data sharing

November 13-17

Monday, November 13

Materials, code, & protocol sharing

Wednesday, November 15

Open science tools

Friday, November 17

Final project work day

November 20-24 Thanksgiving Break

November 27 - December 1

Monday, November 27

In-class final project work day

Meeting on Zoom: https://psu.zoom.us/my/rogilmore

Wednesday, November 29

NO CLASS

You may schedule individual or group meetings to discuss your final project using this link:

https://doodle.com/mm/rickgilmore/book-a-time

Friday, December 1

Project presentations

December 4-8

Monday, December 4

Final project workday

Wednesday, December 6

Project presentations

Friday, December 8

Project presentations

December 11-15

Wednesday, December 13

References

Bargh, J. A., Chen, M., & Burrows, L. (1996). Automaticity of social behavior: Direct effects of trait construct and stereotype-activation on action. Journal of Personality and Social Psychology, 71(2), 230–244. https://doi.org/10.1037//0022-3514.71.2.230
Begley, C. G. (2013). Six red flags for suspect work. Nature, 497(7450), 433–434. https://doi.org/10.1038/497433a
Begley, C. G., & Ellis, L. M. (2012). Drug development: Raise standards for preclinical cancer research. Nature, 483(7391), 531–533. https://doi.org/10.1038/483531a
Bhattacharjee, Y. (2013). The mind of a con man. The New York Times. Retrieved from https://www.nytimes.com/2013/04/28/magazine/diederik-stapels-audacious-academic-fraud.html
Brainerd, J., & You, J. (2018). What a massive database of retracted papers reveals about science publishing’s “death penalty.” Science. https://doi.org/10.1126/science.aav8384
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., … Wu, H. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 1. https://doi.org/10.1038/s41562-018-0399-z
Carey, M. A., Steiner, K. L., & Petri, W. A., Jr. (2020). Ten simple rules for reading a scientific paper. PLoS Computational Biology, 16(7), e1008032. https://doi.org/10.1371/journal.pcbi.1008032
Carney, D. R., Cuddy, A. J. C., & Yap, A. J. (2010). Power posing: Brief nonverbal displays affect neuroendocrine levels and risk tolerance. Psychological Science, 21(10), 1363–1368. https://doi.org/10.1177/0956797610383437
Carpenter, S. (2012). Harvard psychology researcher committed fraud, US investigation concludes. Science, 6. Retrieved from https://www.science.org/content/article/harvard-psychology-researcher-committed-fraud-us-investigation-concludes
Claesen, A., Gomes, S., Tuerlinckx, F., & Vanpaemel, W. (2021). Comparing dream to reality: An assessment of adherence of the first generation of preregistered studies. Royal Society Open Science, 8(211037). https://doi.org/10.1098/rsos.211037
Collaboration, O. S. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716
Crüwell, S., Doorn, J. van, Etz, A., Makel, M. C., Moshontz, H., Niebaum, J. C., … Schulte-Mecklenbeck, M. (2019). Seven easy steps to open science. Zeitschrift für Psychologie, 227(4), 237–248. https://doi.org/10.1027/2151-2604/a000387
Cuddy, A. (2012). Your body language may shape who you are. Retrieved from https://www.ted.com/talks/amy_cuddy_your_body_language_may_shape_who_you_are
Doyen, S., Klein, O., Pichon, C.-L., & Cleeremans, A. (2012). Behavioral priming: It’s all in the mind, but whose mind? PloS One, 7(1), e29081. https://doi.org/10.1371/journal.pone.0029081
Earp, B. D., Everett, J. A. C., Madva, E. N., & Hamlin, J. K. (2014). Out, damned spot: Can the “Macbeth effect” be replicated? Basic and Applied Social Psychology, 36(1), 91–98. https://doi.org/10.1080/01973533.2013.856792
Errington, T. M., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021). Challenges for assessing replicability in preclinical cancer biology. eLife, 10, e67995. https://doi.org/10.7554/eLife.67995
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021). Investigating the replicability of preclinical cancer biology. eLife, 10, e71601. https://doi.org/10.7554/eLife.71601
Feynman, R. P. (1974). Cargo cult science. https://calteches.library.caltech.edu/51/2/CargoCult.htm. Retrieved from https://calteches.library.caltech.edu/51/2/CargoCult.htm
Fidler, F., & Wilcox, J. (2021). Reproducibility of scientific results. In E. N. Zalta (Ed.), The Stanford Encyclopedia of Philosophy (Summer 2021). Metaphysics Research Lab, Stanford University. Retrieved from https://plato.stanford.edu/archives/sum2021/entries/scientific-reproducibility/
FORRT - framework for open and reproducible research training. (n.d.). https://forrt.org/. Retrieved from https://forrt.org/
Franco, A., Malhotra, N., & Simonovits, G. (2014). Social science. Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502–1505. https://doi.org/10.1126/science.1255484
Gilmore, R. O., & Adolph, K. E. (2017). Video can make behavioural research more reproducible. Nature Human Behavior, 1. https://doi.org/10.1038/s41562-017-0128
Gilmore, R. O., Cole, P. M., Verma, S., Aken, M. A. G., & Worthman, C. M. (2020). Advancing scientific integrity, transparency, and openness in child development research: Challenges and possible solutions. Child Development Perspectives, 14(1), 9–14. https://doi.org/10.1111/cdep.12360
Gilroy, S. P., & Kaplan, B. A. (2019). Furthering open science in behavior analysis: An introduction and tutorial for using GitHub in research. Perspectives on Behavior Science, 42(3), 565–581. https://doi.org/10.1007/s40614-019-00202-5
Goldin-Meadow, S. (2016). Why preregistration makes me nervous. APS Observer, 29(7). Retrieved from https://www.psychologicalscience.org/observer/why-preregistration-makes-me-nervous
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016). What does research reproducibility mean? Science Translational Medicine, 8(341), 341ps12–341ps12. https://doi.org/10.1126/scitranslmed.aaf5027
Houtkoop, B. L., Chambers, C., Macleod, M., Bishop, D. V. M., Nichols, T. E., & Wagenmakers, E.-J. (2018). Data sharing in psychology: A survey on barriers and preconditions. Advances in Methods and Practices in Psychological Science, 2515245917751886. https://doi.org/10.1177/2515245917751886
John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953
Kardash, C. M., & Edwards, O. V. (2012). Thinking and behaving like scientists: Perceptions of undergraduate science interns and their faculty mentors. Instructional Science, 40(6), 875–899. https://doi.org/10.1007/s11251-011-9195-0
Kathawalla, U.-K., Silverstein, P., & Syed, M. (2021). Easing into open science: A guide for graduate students and their advisors. Collabra. Psychology, 7(1). https://doi.org/10.1525/collabra.18684
Ledgerwood, A. (2018). The preregistration revolution needs to distinguish between predictions and analyses. Proceedings of the National Academy of Sciences of the United States of America, 115(45), E10516–E10517. https://doi.org/10.1073/pnas.1812592115
Levelt, W. J. M., Drenth, P. J. D., & Noort, E. (2012). Flawed science: The fraudulent research practices of social psychologist diederik stapel. https://pure.mpg.de/rest/items/item_1569964/component/file_1569966/content; pure.mpg.de. Retrieved from https://pure.mpg.de/rest/items/item_1569964/component/file_1569966/content
Macfarlane, B., & Cheng, M. (2008). Communism, universalism and disinterestedness: Re-examining contemporary support among academics for merton’s scientific norms. Journal of Academic Ethics, 6(1), 67–78. https://doi.org/10.1007/s10805-008-9055-y
Merton, R. W. (1973). The normative structure of science. In R. K. Merton & N. W. Storer (Eds.), The Sociology of Science: Theoretical and Empirical Investigations (pp. 267–278). The University of Chicago Press.
Meyer, M. N. (2018). Practical tips for ethical data sharing. Advances in Methods and Practices in Psychological Science, 2515245917747656. https://doi.org/10.1177/2515245917747656
Mitroff, I. I. (1974). Norms and counter-norms in a select group of the Apollo moon scientists: A case study of the ambivalence of scientists. American Sociological Review, 39(4), 579–595. https://doi.org/10.2307/2094423
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Sert, N. P. du, … Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021. https://doi.org/10.1038/s41562-016-0021
National Institutes of Health. (n.d.). NOT-OD-21-013: Final NIH policy for data management and sharing. https://grants.nih.gov/grants/guide/notice-files/NOT-OD-21-013.html. Retrieved from https://grants.nih.gov/grants/guide/notice-files/NOT-OD-21-013.html
Ngiam, W. (2020, April). ReproducibiliTea | simmons, nelson and simonsohn (2011). False-Positive psychology. Youtube. Retrieved from https://www.youtube.com/watch?v=bf3GqyBRgzY
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., … Yarkoni, T. (2015). Promoting an open research culture. Science, 348(6242), 1422–1425. https://doi.org/10.1126/science.aab2374
Nosek, Brian A., & Bar-Anan, Y. (2012). Scientific utopia i: Opening scientific communication. Psychological Inquiry, 23(3), 217–243. https://doi.org/10.1080/1047840X.2012.692215
Nosek, Brian A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences of the United States of America, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114
Nosek, Brian A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., … Vazire, S. (2022). Replicability, robustness, and reproducibility in psychological science. Annual Review of Psychology, 73(2022), 719–748. https://doi.org/10.1146/annurev-psych-020821-114157
Nuijten, M. B., Hartgerink, C. H. J., Assen, M. A. L. M. van, Epskamp, S., & Wicherts, J. M. (2015). The prevalence of statistical reporting errors in psychology (1985–2013). Behavior Research Methods, 1–22. https://doi.org/10.3758/s13428-015-0664-2
Oreskes, N. (2019). Why Trust Science. Princeton University Press.
Ranehill, E., Dreber, A., Johannesson, M., Leiberg, S., Sul, S., & Weber, R. A. (2015). Assessing the robustness of power posing: No effect on hormones and risk tolerance in a large sample of men and women. Psychological Science, 26(5), 653–656. https://doi.org/10.1177/0956797614553946
Ritchie, S. (2020). Science fictions: Exposing fraud, bias, negligence and hype in science (1st ed.). Penguin Random House. Retrieved from https://www.amazon.com/Science-Fictions/dp/1847925669
Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638
Ruben, A. (2016). How to read a scientific paper. Science| AAAS [Internet], 20. Retrieved from https://www.science.org/content/article/how-read-scientific-paper-rev2
Sagan, C. (1996). The Demon-haunted World: Science as a Candle in the Dark (pp. 200–218). Ballantine Books.
Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., … Nosek, B. A. (2018). Many analysts, one data set: Making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 1(3), 337–356. https://doi.org/10.1177/2515245917747646
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632
Soska, K. C., Xu, M., Gonzalez, S. L., Herzberg, O., Tamis-LeMonda, C. S., Gilmore, R. O., & Adolph, K. E. (2021). (Hyper)active data curation: A video case study from behavioral science. Journal of Escience Librarianship, 10(3). https://doi.org/10.7191/jeslib.2021.1208
SRCD. (2019). Policy on scientific integrity, transparency, and openness | society for research in child development SRCD. https://www.srcd.org/policy-scientific-integrity-transparency-and-openness. Retrieved from https://www.srcd.org/policy-scientific-integrity-transparency-and-openness
Szucs, D., & Ioannidis, J. P. A. (2017). Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature. PLoS Biology, 15(3), e2000797. https://doi.org/10.1371/journal.pbio.2000797
Tenopir, C., Rice, N. M., Allard, S., Baird, L., Borycz, J., Christian, L., … Sandusky, R. J. (2020). Data sharing, management, use, and reuse: Practices and perceptions of scientists worldwide. PloS One, 15(3), e0229003. https://doi.org/10.1371/journal.pone.0229003
Wilson, L. C. (2014, September). Introduction to Meta-Analysis: A guide for the novice. https://www.psychologicalscience.org/observer/introduction-to-meta-analysis-a-guide-for-the-novice. Retrieved from https://www.psychologicalscience.org/observer/introduction-to-meta-analysis-a-guide-for-the-novice
Zhong, C.-B., & Liljenquist, K. (2006). Washing away your sins: Threatened morality and physical cleansing. Science, 313(5792), 1451–1452. https://doi.org/10.1126/science.1130726