Work session: Scientific integrity & Final Projects

2024-09-27 Fri

Rick Gilmore

Overview

Announcements

In the news…

Piller (2024)

Last time…

  • Doyen, Klein, Pichon, & Cleeremans (2012) replication of Study 2 from Bargh, Chen, & Burrows (1996)

First, in Experiment 1, despite the use of a larger sample and an experimental procedure devoid of the limitations present in the original experiment, we were not able to replicate Bargh et al’s automatic effect of priming on walking speed.

Figure 1: Plot of data from Experiment 1 in Doyen et al. (2012). Not in original paper.

Walking speed: In this analysis, we used participants’ walking speed as they entered the experiment room, (i.e., before priming) as a covariate. The results show no significant difference between the Prime (M=6.27 SD=2.15) and the No-Prime group (M=6.39 SD=1.11) in the time necessary to walk along the hallway after the priming manipulation (F(1, 119) < 1, \(\eta^2\)=.01).

…in Experiment 2 we were indeed able to obtain the priming effect on walking speed for both subjective and objective timings. Crucially however, this was only possible by manipulating experimenters’ expectations in such a way that they would expect primed participants to walk slower.

Figure 1 from Doyen et al. (2012)

…participants in the Prime (M=7.25s SD=.33) condition walked significantly slower than those in the No-Prime (M=6.73s SD=.32) condition when tested by an experimenter expecting the primed participants to be slower (F(1, 24)=12,32, p=.002, \(\eta^2\)=.339). This result replicates Bargh et al. effect on walking speed.

Strikingly however, as indicated by the significant interaction between participants’ condition X experimenters’ condition (F(3, 46)=18.82, p <.001, \(\eta^2\)=.295), the effect was reversed in the Fast experimenters condition, so that Primed participants (M=5.8s SD=.73) walked faster than No-Prime participants (M=6.43s SD=.4) (F(1, 24)=7,.5, p=.012, \(\eta^2\)=.274).

In terms of objective timing (see Figure 2), we observed a difference between the Prime (M=6.95s SD=.36) and the No-Prime group (M=6.52s SD=.31) remains for the slow experimenter condition (F(1, 24)=7.07, p=.014, \(\eta^2\)=.228). Bargh et al’s effect is thus replicated in this condition.

Figure 2 from Doyen et al. (2012)

…There was no difference between the Prime (M=6.07s SD=.57) and the No-Prime (M=6.01s SD=.39) group (F(1, 24)=.231, p=.636, \(\eta^2\)=.011) in the fast experimenter condition.

Finally, we also found a main effect of the experimenters’ condition was found: participants tested by experimenters in the Slow condition (M=6.75s SD=.43) were slower than those tested by experimenters in the Fast condition (M=6.04s SD=.47) (F(1, 49)=30.44, p<.001, \(\eta^2\)=.404).

Open Science Notes

  • Neither Bargh et al. (1996) or Doyen et al. (2012) shared data
  • Plots could have shown individual participants’ data
  • Videos of procedures would have been helpful

Comments Gilmore doesn’t understand

Additionally, those participants who actually exhibited a slower walking speed reported in good proportion being aware of that particular behavior.

…Thus, if our experiment had used subjective timings only, as in Bargh et al’s original study, we would have erroneously concluded that experimenters’ expectations was a critical factor for the priming effect to occur. But the objective timings show otherwise…

…The reverse priming effect on walking speed observed on subjective timings stems from errors committed while operating the stopwatch: Experimenters in the fast condition were prone to commit more errors because they expected the need to capture a fast event and hence tended to be inaccurate in stopping the watch.

Figure 3 from Doyen et al. (2012)

“Figure 1 from Doyen et al. (2012)”

“Figure 2 from Doyen et al. (2012)”

Today

Next time

Students’ Choice

Options

  1. Read a successful replication study: (Soto, 2019)
  2. Deep dive into how the Survey-01 and Survey-02 code actually works.
  3. Explore some large-scale databases of replication studies in Psychology: (Röseler et al., 2024) and https://forrt.org/reversals/
  1. Discuss norms and counternorms in science vs. other fields based on your own responses to Exercise 03
  2. Explore Protzko et al. (2023) retraction.
  3. Understanding p-values.

Resources

References

PSYCH 490.012 • Fall 2024

Bargh, J. A., Chen, M., & Burrows, L. (1996). Automaticity of social behavior: Direct effects of trait construct and stereotype-activation on action. Journal of Personality and Social Psychology, 71(2), 230–244. https://doi.org/10.1037//0022-3514.71.2.230
Doyen, S., Klein, O., Pichon, C.-L., & Cleeremans, A. (2012). Behavioral priming: It’s all in the mind, but whose mind? PloS One, 7(1), e29081. https://doi.org/10.1371/journal.pone.0029081
Piller, C. (2024, September). Did a top NIH official manipulate alzheimer’s and parkinson’s studies for decades? https://www.science.org/content/article/research-misconduct-finding-neuroscientist-eliezer-masliah-papers-under-suspicion.
Protzko, J., Krosnick, J., Nelson, L., Nosek, B. A., Axt, J., Berent, M., … Schooler, J. W. (2023). High replicability of newly discovered social-behavioural findings is achievable. Nature Human Behaviour. https://doi.org/10.1038/s41562-023-01749-9
Röseler, L., Kaiser, L., Doetsch, C., Klett, N., Seida, C., Schütz, A., … Zhang, Y. (2024). The replication database: Documenting the replicability of psychological science. Journal of Open Psychology Data, 12(1), 8. https://doi.org/10.5334/jopd.101
Soto, C. J. (2019). How replicable are links between personality traits and consequential life outcomes? The life outcomes of personality replication project. Psychological Science, 30(5), 711–727. https://doi.org/10.1177/0956797619831612