Modified

September 6, 2024

Schedule

Week 1: August 26-30

Monday, August 26

Introduction to the course: Why trust science?

Tip

To earn 3 extra credit points for completing the survey, send the TA one of the following:

  1. The date and time you completed the survey in the following format: “7/30/2024 12:20:23”
  2. A special code or phrase that is likely to be unique to you but doesn’t contain identifiable information.

Wednesday, August 28

Don’t Fool Yourself

Friday, August 30

Work Session: How to read a scientific paper

Week 2: September 2-6

Monday, September 02

NO CLASS, LABOR DAY

Wednesday, September 04

How science works (or should)

Friday, September 06

Work Session: Reading a paper; Evaluating its claims

Week 3: September 9-13

Monday, September 09

Scientific norms and counter-norms

Wednesday, September 11

Adherence to norms and counter-norms

Friday, September 13

Work session: Norms and counter-norms

Week 4: September 16-20

Monday, September 16

A replication crisis (or not)

Wednesday, September 18

Replication attempt: The “Lady Macbeth Effect”

Friday, September 20

Wrap-up on ‘Macbeth effect’ replication

Week 5: September 23-27

Monday, September 23

Priming effect: Original study

Wednesday, September 25

Priming effect: Replication study

Friday, September 27

Work session: Scientific integrity & Final project proposals

Week 6: September 30 - October 4

Monday, September 30

Mind your p’s

Wednesday, October 02

Fraud & misconduct

Friday, October 04

Work session: P-hacking & Final project proposals

Week 7: October 7-11

Monday, October 07

Retraction and scientific integrity

On Zoom: https://psu.zoom.us/my/rogilmore. Check-in for attendance. Join from anywhere convenient to you.

Wednesday, October 09

NO CLASS

Friday, October 11

Questionable research practices

Week 8: October 14-18

Monday, October 14

Work Session: P-hacking and Final Project Proposals

Wednesday, October 16

Prevalence of QRPs

Friday, October 18

File drawer effect & Work Session: Alpha, Power, Effect Sizes, & Sample Size

Week 9: October 21-25

Monday, October 21

Negligence

  • Read
    • Nuijten, Hartgerink, Assen, Epskamp, & Wicherts (2015)
    • (skim) Szucs & Ioannidis (2017)
  • Class notes

Wednesday, October 23

Hype

Friday, October 25

Hype, continued

Work session: Alpha, Power, Effect Sizes, & Sample Size & Replication

Week 10: October 28 - November 1

Monday, October 28

Solutions

  • Read

Wednesday, October 30

Changing journal policies

Friday, November 01

Catch-up day

Week 11: November 4-8

Monday, November 04

Large-scale replication studies

  • Read
    • Collaboration (2015)

Wednesday, November 06

Meta-analysis & many analysts

  • Read
    • Wilson (2014)
    • Silberzahn et al. (2018)

Friday, November 08

Work Session: Final Projects

Week 12: November 11-15

Monday, November 11

Preregistration

  • Read
    • Brian A. Nosek, Ebersole, DeHaven, & Mellor (2018)
    • Ledgerwood (2018) or Goldin-Meadow (2016)
    • (Optional) Claesen, Gomes, Tuerlinckx, & Vanpaemel (2021)
  • Explore

Wednesday, November 13

Data sharing

Friday, November 15

Work Session: Final Projects

Week 13: November 18-22

Monday, November 18

Materials, code, & protocol sharing

Wednesday, November 20

Open science tools

  • Read
    • Kathawalla, Silverstein, & Syed (2021)
    • Chopik, Bremner, Defever, & Keller (2018)
    • (Optional) Crüwell et al. (2019)
  • Explore
    • “FORRT - framework for open and reproducible research training” (n.d.)
  • Complete (optional)

Friday, November 22

Work session: Data sharing

November 25-29 Thanksgiving Break

Week 14: December 2-6

Monday, December 2

In-class final project work day

Wednesday, December 4

In-class final project work day

Friday, December 6

Project presentations

Week 15: December 9-13

Monday, December 9

Project presentations

Wednesday, December 11

Project presentations

Friday, December 13

Our open science future

December 16-20

Wednesday, December 18

References

Bargh, J. A., Chen, M., & Burrows, L. (1996). Automaticity of social behavior: Direct effects of trait construct and stereotype-activation on action. Journal of Personality and Social Psychology, 71(2), 230–244. https://doi.org/10.1037//0022-3514.71.2.230
Begley, C. G. (2013). Six red flags for suspect work. Nature, 497(7450), 433–434. https://doi.org/10.1038/497433a
Bhattacharjee, Y. (2013). The mind of a con man. The New York Times. Retrieved from https://www.nytimes.com/2013/04/28/magazine/diederik-stapels-audacious-academic-fraud.html
Brainerd, J., & You, J. (2018). What a massive database of retracted papers reveals about science publishing’s “death penalty.” Science. https://doi.org/10.1126/science.aav8384
Carey, M. A., Steiner, K. L., & Petri, W. A., Jr. (2020). Ten simple rules for reading a scientific paper. PLoS Computational Biology, 16(7), e1008032. https://doi.org/10.1371/journal.pcbi.1008032
Carney, D. R., Cuddy, A. J. C., & Yap, A. J. (2010). Power posing: Brief nonverbal displays affect neuroendocrine levels and risk tolerance. Psychological Science, 21(10), 1363–1368. https://doi.org/10.1177/0956797610383437
Carpenter, S. (2012). Harvard psychology researcher committed fraud, US investigation concludes. Science, 6. Retrieved from https://www.science.org/content/article/harvard-psychology-researcher-committed-fraud-us-investigation-concludes
Chopik, W. J., Bremner, R. H., Defever, A. M., & Keller, V. N. (2018). How (and whether) to teach undergraduates about the replication crisis in psychological science. Teaching of Psychology, 45(2), 158–163. https://doi.org/10.1177/0098628318762900
Claesen, A., Gomes, S., Tuerlinckx, F., & Vanpaemel, W. (2021). Comparing dream to reality: An assessment of adherence of the first generation of preregistered studies. Royal Society Open Science, 8(211037). https://doi.org/10.1098/rsos.211037
Collaboration, O. S. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716
Crüwell, S., Doorn, J. van, Etz, A., Makel, M. C., Moshontz, H., Niebaum, J. C., … Schulte-Mecklenbeck, M. (2019). Seven easy steps to open science. Zeitschrift für Psychologie, 227(4), 237–248. https://doi.org/10.1027/2151-2604/a000387
Cuddy, A. (2012). Your body language may shape who you are. Retrieved from https://www.ted.com/talks/amy_cuddy_your_body_language_may_shape_who_you_are
Denworth, L. (2019, October). The significant problem of P values. https://www.scientificamerican.com/article/the-significant-problem-of-p-values/.
Doyen, S., Klein, O., Pichon, C.-L., & Cleeremans, A. (2012). Behavioral priming: It’s all in the mind, but whose mind? PloS One, 7(1), e29081. https://doi.org/10.1371/journal.pone.0029081
Earp, B. D., Everett, J. A. C., Madva, E. N., & Hamlin, J. K. (2014). Out, damned spot: Can the “Macbeth effect” be replicated? Basic and Applied Social Psychology, 36(1), 91–98. https://doi.org/10.1080/01973533.2013.856792
FORRT - framework for open and reproducible research training. (n.d.). https://forrt.org/. Retrieved from https://forrt.org/
Franco, A., Malhotra, N., & Simonovits, G. (2014). Social science. Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502–1505. https://doi.org/10.1126/science.1255484
Gilmore, R. O., & Adolph, K. E. (2017). Video can make behavioural research more reproducible. Nature Human Behavior, 1. https://doi.org/10.1038/s41562-017-0128
Gilmore, R. O., Cole, P. M., Verma, S., Aken, M. A. G., & Worthman, C. M. (2020). Advancing scientific integrity, transparency, and openness in child development research: Challenges and possible solutions. Child Development Perspectives, 14(1), 9–14. https://doi.org/10.1111/cdep.12360
Gilroy, S. P., & Kaplan, B. A. (2019). Furthering open science in behavior analysis: An introduction and tutorial for using GitHub in research. Perspectives on Behavior Science, 42(3), 565–581. https://doi.org/10.1007/s40614-019-00202-5
Goldin-Meadow, S. (2016). Why preregistration makes me nervous. APS Observer, 29(7). Retrieved from https://www.psychologicalscience.org/observer/why-preregistration-makes-me-nervous
Houtkoop, B. L., Chambers, C., Macleod, M., Bishop, D. V. M., Nichols, T. E., & Wagenmakers, E.-J. (2018). Data sharing in psychology: A survey on barriers and preconditions. Advances in Methods and Practices in Psychological Science, 2515245917751886. https://doi.org/10.1177/2515245917751886
John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953
Kardash, C. M., & Edwards, O. V. (2012). Thinking and behaving like scientists: Perceptions of undergraduate science interns and their faculty mentors. Instructional Science, 40(6), 875–899. https://doi.org/10.1007/s11251-011-9195-0
Kathawalla, U.-K., Silverstein, P., & Syed, M. (2021). Easing into open science: A guide for graduate students and their advisors. Collabra. Psychology, 7(1). https://doi.org/10.1525/collabra.18684
Ledgerwood, A. (2018). The preregistration revolution needs to distinguish between predictions and analyses. Proceedings of the National Academy of Sciences of the United States of America, 115(45), E10516–E10517. https://doi.org/10.1073/pnas.1812592115
Levelt, W. J. M., Drenth, P. J. D., & Noort, E. (2012). Flawed science: The fraudulent research practices of social psychologist diederik stapel. https://pure.mpg.de/rest/items/item_1569964/component/file_1569966/content; pure.mpg.de. Retrieved from https://pure.mpg.de/rest/items/item_1569964/component/file_1569966/content
Macfarlane, B., & Cheng, M. (2008). Communism, universalism and disinterestedness: Re-examining contemporary support among academics for merton’s scientific norms. Journal of Academic Ethics, 6(1), 67–78. https://doi.org/10.1007/s10805-008-9055-y
Meyer, M. N. (2018). Practical tips for ethical data sharing. Advances in Methods and Practices in Psychological Science, 2515245917747656. https://doi.org/10.1177/2515245917747656
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Sert, N. P. du, … Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021. https://doi.org/10.1038/s41562-016-0021
National Institutes of Health. (n.d.). NOT-OD-21-013: Final NIH policy for data management and sharing. https://grants.nih.gov/grants/guide/notice-files/NOT-OD-21-013.html. Retrieved from https://grants.nih.gov/grants/guide/notice-files/NOT-OD-21-013.html
Ngiam, W. (2020, April). ReproducibiliTea | simmons, nelson and simonsohn (2011). False-Positive psychology. Youtube. Retrieved from https://www.youtube.com/watch?v=bf3GqyBRgzY
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., … Yarkoni, T. (2015). Promoting an open research culture. Science, 348(6242), 1422–1425. https://doi.org/10.1126/science.aab2374
Nosek, Brian A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences of the United States of America, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114
Nuijten, M. B., Hartgerink, C. H. J., Assen, M. A. L. M. van, Epskamp, S., & Wicherts, J. M. (2015). The prevalence of statistical reporting errors in psychology (1985–2013). Behavior Research Methods, 1–22. https://doi.org/10.3758/s13428-015-0664-2
Oreskes, N. (2019). Why Trust Science. Princeton University Press.
Ranehill, E., Dreber, A., Johannesson, M., Leiberg, S., Sul, S., & Weber, R. A. (2015). Assessing the robustness of power posing: No effect on hormones and risk tolerance in a large sample of men and women. Psychological Science, 26(5), 653–656. https://doi.org/10.1177/0956797614553946
Ritchie, S. (2020). Science fictions: Exposing fraud, bias, negligence and hype in science (1st ed.). Penguin Random House. Retrieved from https://www.amazon.com/Science-Fictions/dp/1847925669
Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638
Ruben, A. (2016). How to read a scientific paper. Science| AAAS [Internet], 20. Retrieved from https://www.science.org/content/article/how-read-scientific-paper-rev2
Sagan, C. (1996). The Demon-haunted World: Science as a Candle in the Dark (pp. 200–218). Ballantine Books.
Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., … Nosek, B. A. (2018). Many analysts, one data set: Making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 1(3), 337–356. https://doi.org/10.1177/2515245917747646
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632
Soska, K. C., Xu, M., Gonzalez, S. L., Herzberg, O., Tamis-LeMonda, C. S., Gilmore, R. O., & Adolph, K. E. (2021). (Hyper)active data curation: A video case study from behavioral science. Journal of Escience Librarianship, 10(3). https://doi.org/10.7191/jeslib.2021.1208
SRCD. (2019). Policy on scientific integrity, transparency, and openness | society for research in child development SRCD. https://www.srcd.org/policy-scientific-integrity-transparency-and-openness. Retrieved from https://www.srcd.org/policy-scientific-integrity-transparency-and-openness
Szucs, D., & Ioannidis, J. P. A. (2017). Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature. PLoS Biology, 15(3), e2000797. https://doi.org/10.1371/journal.pbio.2000797
Tenopir, C., Rice, N. M., Allard, S., Baird, L., Borycz, J., Christian, L., … Sandusky, R. J. (2020). Data sharing, management, use, and reuse: Practices and perceptions of scientists worldwide. PloS One, 15(3), e0229003. https://doi.org/10.1371/journal.pone.0229003
Wilson, L. C. (2014, September). Introduction to Meta-Analysis: A guide for the novice. https://www.psychologicalscience.org/observer/introduction-to-meta-analysis-a-guide-for-the-novice. Retrieved from https://www.psychologicalscience.org/observer/introduction-to-meta-analysis-a-guide-for-the-novice
Zhong, C.-B., & Liljenquist, K. (2006). Washing away your sins: Threatened morality and physical cleansing. Science, 313(5792), 1451–1452. https://doi.org/10.1126/science.1130726