2023-09-15 Fri

Work session: Replication with R, R Markdown, & Quarto

Rick Gilmore



Last time…


Work session: Reproducible workflows with R, R Markdown, & Quarto

Reproducible workflows with R, R Markdown, & Quarto


  • Capture details of the research workflow
  • Make research notebooks that actually do work
    • Not just capture info passively
  • Do so efficiently, flexibly

More goals

  • Make computer work harder
  • Reduce manual work
    • More error-prone
  • Save history of changes (e.g., version control)
  • Make use of HTML and the web
  • Keep track of (& cite) resources

Research workflow

flowchart LR
  A{Idea/Question} -->|Develop|B{Plan}
  B -->|Create measures|C{Collect Info}
  C -->|Gather collected|D{Analyze Info}
  D -->|Visualize|E{Share findings}
  E -->|Evaluate|A
flowchart LR
  A{Idea/Question} -->|Develop|B{Plan}
  B -->|Create measures|C{Collect Info}
  C -->|Gather collected|D{Analyze Info}
  D -->|Visualize|E{Share findings}
  E -->|Evaluate|A

Graph made using Mermaid.js


Survey questions


Q: Why Google Form?

A: Easier than Qualtrics to generate, extract information from.

Also, more broadly available (Google Forms is free to people with a Google account).

Gather & Clean Data

Quarto project

  • Set of text files that combine
    • Notes
    • Code (in the R programming language)
    • Web links
    • Figures


Q: Why not MS Word or Google Doc?

Q: Why not MS Excel or MS PowerPoint?

flowchart TD
  A{Quarto File 1} --> B[HTML]
  C{Quarto File 2} --> B
  D{Quarto File 3} --> B
  A --> E(.docx)
  C --> E
  D --> E
  A --> F((.ppt))
  C --> F
  D --> F
flowchart TD
  A{Quarto File 1} --> B[HTML]
  C{Quarto File 2} --> B
  D{Quarto File 3} --> B
  A --> E(.docx)
  C --> E
  D --> E
  A --> F((.ppt))
  C --> F
  D --> F

Anatomy of a Quarto document

title: "Survey 01"
  update_data: FALSE
  • Document title
  • A parameter that determines whether the document should update the data or not.
## Purpose

This page documents the data processing steps involved with Survey-01 in PSYCH 490.009 Fall 2023.

The survey questions were adapted from those discussed in [@Krumrei-Mancuso2016-qm; @Nadelson2014-ym; @Plohl2023-yf].
  • First section with header title (“Purpose”)
    • Note: the double hash/pound symbols “##” mean this is a level 2 header
    • You do use MS Word’s styles, don’t you?
  • Short-hand names for the references with the “@” symbol
    • Quarto generates in-text citations and a bibliography for us!
## Survey

Direct link: <https://forms.gle/Szk1pLEu4ZLWtFjX7>

<iframe src="https://docs.google.com/forms/d/e/1FAIpQLScp2GG_2TZ3_T6Yl1okn-sHJF93n9RZu_EmLWQtHP0rJcGM3w/viewform?embedded=true" width="640" height="6003" frameborder="0" marginheight="0" marginwidth="0">Loading…</iframe>
  • HTML links surrounded by “<>” become clickable in our output.
  • the “<iframe…” comes from the Google Forms site. It embeds the actual survey in our document.
    • No cutting and pasting!
## Preparation

First, we load the external packages (groups of R commands) that we will be using.

  • Another level 2 header
  • And some R code in a “code chunk”.
    • The brackets “{r}” tell the computer that we’re speaking R to it.
    • We could also speak other languages like “{python}”, etc.

Enough anatomy

Comparing this to other approaches…

Let’s look at Survey-02

Summary figure from Survey-02

What data picture(s) would you like to make?

What analyses would you like to try?

Next time

Replication in cancer biology



Artner, R., Verliefde, T., Steegen, S., Gomes, S., Traets, F., Tuerlinckx, F., & Vanpaemel, W. (2021). The reproducibility of statistical results in psychological research: An investigation using unpublished raw data. Psychological Methods, 26(5), 527–546. https://doi.org/10.1037/met0000365
Bargh, J. A., Chen, M., & Burrows, L. (1996). Automaticity of social behavior: Direct effects of trait construct and stereotype-activation on action. Journal of Personality and Social Psychology, 71(2), 230–244. https://doi.org/10.1037//0022-3514.71.2.230
Doyen, S., Klein, O., Pichon, C.-L., & Cleeremans, A. (2012). Behavioral priming: It’s all in the mind, but whose mind? PloS One, 7(1), e29081. https://doi.org/10.1371/journal.pone.0029081
Earp, B. D., Everett, J. A. C., Madva, E. N., & Hamlin, J. K. (2014). Out, damned spot: Can the Macbeth effect” be replicated? Basic and Applied Social Psychology, 36(1), 91–98. https://doi.org/10.1080/01973533.2013.856792
Errington, T. M., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021). Challenges for assessing replicability in preclinical cancer biology. eLife, 10, e67995. https://doi.org/10.7554/eLife.67995
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021). Investigating the replicability of preclinical cancer biology. eLife, 10, e71601. https://doi.org/10.7554/eLife.71601
Krumrei-Mancuso, E. J., & Rouse, S. V. (2016). The development and validation of the comprehensive intellectual humility scale. Journal of Personality Assessment, 98(2), 209–221. https://doi.org/10.1080/00223891.2015.1068174
Nadelson, L., Jorcyk, C., Yang, D., Jarratt Smith, M., Matson, S., Cornell, K., & Husting, V. (2014). I just don’t trust them: The development and validation of an assessment instrument to measure trust in science and scientists. School Science and Mathematics, 114(2), 76–86. https://doi.org/10.1111/ssm.12051
Plohl, N., & Musil, B. (2023). Assessing the incremental value of intellectual humility and cognitive reflection in predicting trust in science. Personality and Individual Differences, 214, 112340. https://doi.org/10.1016/j.paid.2023.112340
Quarto. (n.d.). https://quarto.org/. Retrieved from https://quarto.org/
Zhong, C.-B., & Liljenquist, K. (2006). Washing away your sins: Threatened morality and physical cleansing. Science, 313(5792), 1451–1452. https://doi.org/10.1126/science.1130726