# Overview

## Announcements

- [Due Friday]{.orange_due}
    - [Final project](exercises/final-project.qmd) proposal

## Today

*Prevalence of QRPs*

- Read & Discuss
    - [@John2012-tk]
- Discuss results from
    - [Exercise 02: P-hack your way to scientific glory](../exercises/ex04-p-hacking.qmd).
    
---

John, L. K., Loewenstein, G. & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. *Psychological Science*, *23*(5), 524–532. <https://doi.org/10.1177/0956797611430953>

---

>Cases of clear scientific misconduct have received significant media attention recently, but less flagrantly questionable research practices may be more prevalent and, ultimately, more damaging to the academic enterprise. Using an anonymous elicitation format supplemented by incentives for honest reporting, we surveyed over 2,000 psychologists about their involvement in questionable research practices. 

<p style="text-align:center;">
[@John2012-tk]
</p>

---

>The impact of truth-telling incentives on self-admissions of questionable research practices was positive, and this impact was greater for practices that respondents judged to be less defensible. Combining three different estimation methods, we found that the percentage of respondents who have engaged in questionable practices was surprisingly high. This finding suggests that some questionable practices may constitute the prevailing research norm.

<p style="text-align:center;">
[@John2012-tk]
</p>

## Accessibility/openness notes

- Paper *not* openly accessible.
- Paper was accessible via authenticated access to PSU library.

---

![Table 1 from [@John2012-tk]](../include/img/john-2012-table-01.png)

---

>The two versions of the survey differed in the incentives they offered to respondents. In the Bayesian-truth-serum (BTS) condition, a scoring algorithm developed by one of the authors (Prelec, 2004) was used to provide incentives for truth telling. This algorithm uses respondents’ answers about their own...

---

>...behavior and their estimates of the sample distribution of answers as inputs in a truth-rewarding scoring formula. Because the survey was anonymous, compensation could not be directly linked to individual scores. 

---

>Instead, respondents were told that we would make a donation to a charity of their choice, selected from five options, and that the size of this donation would depend on the truthfulness of their responses, as determined by the BTS scoring system. By inducing a (correct) belief that dishonesty would reduce donations, we hoped to amplify the moral stakes riding on each answer (for details on the donations, see Supplementary Results in the Supplemental Material). 

---

>Respondents were not given the details of the scoring system but were told that it was based on an algorithm published in Science and were given a link to the article. There was no deception: Respondents’ BTS scores determined our contributions to the five charities. 

---

>Respondents in the control condition were simply told that a charitable donation would be made on behalf of each respondent. (For details on the effect of the size of the incentive on response rates, see Participation Incentive Survey in the Supplemental Material.)

<p style="text-align:center;">
[@John2012-tk]
</p>

---

```{r, echo=FALSE, fig.cap="Figure 1 from [@John2012-tk](http://dx.doi.org/10.1177/0956797611430953)", out.width="80%"}
knitr::include_graphics("../include/img/john-2012-fig-01.png")
```

---

```{r, echo=FALSE, fig.cap="Figure 2 from [@John2012-tk](http://dx.doi.org/10.1177/0956797611430953)", out.width="80%"}
knitr::include_graphics("../include/img/john-2012-fig-02.png")
```

---

::: {.callout-note}

What do you think about the "truth serum" manipulation?

Do the data persuade you that it made respondents more honest?

:::

## Reproducibility notes for [@John2012-tk]

>Supplemental Material
>Additional supporting information may be found at http://pss.sagepub.com/content/by/supplemental-data

<p style="text-align:center;">
[@John2012-tk]
</p>

---

- Visiting this URL (<http://pss.sagepub.com/content/by/supplemental-data>) takes one to the following:

```{r}
knitr::include_url("http://pss.sagepub.com/content/by/supplemental-data", height = 600)
```

---

- I went to the journal page and searched for the article title.
- Since the article is behind a paywall, I wasn't able to access the supplemental materials that way.
- After authenticating to the PSU library, I was able to find a PDF of the [supplementary material](https://psu.instructure.com/courses/2245007/files/folder/readings?preview=146306047). It and the [original paper](https://psu.instructure.com/courses/2245007/files/folder/readings?preview=146306050) are on Canvas.
- I was unable to find the raw data, but I found the questions on p. 5 of the supplementary material.

---

![Supplementary material for [@John2012-tk]](../include/img/john-etal-2012-items.png){fig-align="center"}

# More evidence about prevalance of QRPs

## Survey of $n=3,247$ NIH-funded scientists

![Table 1 from @Martinson2005-fk](../include/img/martinson-etal-2005-table-1.png){fig-align="center" height="75%"}

## Meta-analysis [@Xie2021-us]

![Self-reported prevalence; Figure 2 from @Xie2021-us](../include/img/xie-etal-2021-fig-2.png){fig-align="center"}

---

![Observed prevalence; Figure 3 from @Xie2021-us](../include/img/xie-etal-2021-fig-3.png){fig-align="center"}

# Evaluating *p*-hacking

## Our results

- Who got a "significant" result?
- How many different analyses did you try?
- Who changed their analysis after finding a significant result?
- Did anyone try another analysis--after you got a significant result--and *keep* the non-significant result?
- [Visualizations](../surveys/p-hacking-suppl.qmd) of the data

---

- Do we have good evidence about which party harms or helps the economy?
- Why or why not?
- What is *p*-hacking?
- Did you *p*-hack? Did another classmate? Why do you think so?

---

::: {.columns}

::: {.column width="40%"}

![Daniel Patrick Moynihan from Wikipedia](https://upload.wikimedia.org/wikipedia/commons/thumb/6/6b/Patrick_Moynihan_1998.jpg/220px-Patrick_Moynihan_1998.jpg)

:::

::: {.column width="60%"}

>You are entitled to your own opinion, but you are not entitled to your own facts.

:::

:::

# Next time

*File drawer effect*

- Read
    - [@Rosenthal1979-zi]
    - [@Franco2014-yu]
- [Class notes](wk08-2023-10-09.qmd)
- [Assignment]{.green_assigned}
    - [Exercise 05: Alpha, Power, Effect Sizes, & Sample Size](exercises/ex05-apes.qmd)

# Resources

## References