Hype
In the news
Youyou, W., Yang, Y. & Uzzi, B. (2023). A discipline-wide investigation of the replicability of Psychology papers over the past two decades. Proceedings of the National Academy of Sciences of the United States of America, 120(6), e2208863120. https://doi.org/10.1073/pnas.2208863120
Conjecture about the weak replicability in social sciences has made scholars eager to quantify the scale and scope of replication failure for a discipline. Yet small-scale manual replication methods alone are ill-suited to deal with this big data problem. Here, we conduct a discipline-wide replication census in science. Our sample (N = 14,126 papers) covers nearly all papers published in the six top-tier Psychology journals over the past 20 y. Using a validated machine learning model that estimates a paper’s likelihood of replication, we found evidence that both supports and refutes speculations drawn from a relatively small sample of manual replications. First, we find that a single overall replication rate of Psychology poorly captures the varying degree of replicability among subfields. Second, we find that replication rates are strongly correlated with research methods in all subfields. Experiments replicate at a significantly lower rate than do non-experimental studies. Third, we find that authors’ cumulative publication number and citation impact are positively related to the likelihood of replication, while other proxies of research quality and rigor, such as an author’s university prestige and a paper’s citations, are unrelated to replicability. Finally, contrary to the ideal that media attention should cover replicable research, we find that media attention is positively related to the likelihood of replication failure. Our assessments of the scale and scope of replicability are important next steps toward broadly resolving issues of replicability.
Roadmap
- Discuss
- (Ritchie, 2020), Chapter 6
- Carney et al. (2010)
- (Optional) Ranehill et al. (2015)
- Watch/discuss
- Due
- Final project proposals.
- Reminder
- Assignment Exercise 03: Alpha, Power, Effect Sizes, & Sample Size, due Tuesday, March 16
The New York Times published an article (Dominus (2017)) on the controversy surrounding Dr. Cuddy’s work called “When the Revolution Came for Amy Cuddy.”
I considered reading this and discussing it, but I was more interested in the substantive claims made in the papers and in the talk.
Reading the Times’ story would make a great final project topic, however.
(Ritchie, 2020), Chapter 6
Problems with scientific press releases
- Not written by scientists
- give unwarranted advice
- cross-species leaps or generalizations
- equating correlation with causation
“Churnalism”
- Journalists do not do their own investigations but repeat press releases
Popular books (by scientists, too) can gloss over complexities, nuances
Positive rhetoric/spin
Counter-example from OPERA study of faster-than-light particle
Carney et al. (2010)
Carney, D. R., Cuddy, A. J. C. & Yap, A. J. (2010). Power posing: brief nonverbal displays affect neuroendocrine levels and risk tolerance. Psychological Science, 21(10), 1363–1368. https://doi.org/10.1177/0956797610383437
Humans and other animals express power through open, expansive postures, and they express powerlessness through contractive postures. But can these postures actually cause power? The results of this study confirmed our prediction posing in high-power nonverbal displays (as opposed to low-power nonverbal displays) would cause neuroendocrine behavioral changes for both male and female participants: High-power posers experienced elevations in testosterone, decreases in cortisol, and increased feelings of power and tolerance for risk; low-power posers exhibited the opposite pattern. In posing in displays of power caused advantaged and adaptive psychological, physiological, and behavioral changes, and findings suggest that embodiment extends beyond mere thinking and feeling, to physiology and subsequent behavioral That a person can, by assuming two simple 1-min poses, embody power and instantly become more powerful has real-actionable implications.
Methods
- \(n=42\) participants (26 female; age or other demographic characteristics?)
- Assigned into two groups (high power vs. low power)
- Experimenter posed bodies
- Two different poses held for 1 min
](include/img/carney-etal-fig01.png)
Figure 73: High power poses. Figure 1 from Carney et al. (2010)
](include/img/carney-etal-fig02.png)
Figure 74: Low power poses. Figure 2 from Carney et al. (2010)
- Other tasks/measures
- Gambling task (risk-taking, powerful feelings)
- Self-reported feelings of power
- Saliva samples (pretest and ~ 17 min after pose)
- Tested for cortisol, testosterone
](include/img/carney-etal-2010-results.png)
](include/img/carney-etal-fig03.png)
Citation data
Paper cited ~1,500 times (https://scholar.google.com/scholar?hl=en&as_sdt=0%2C39&q=Power+posing%3A+brief+nonverbal+displays+affect+neuroendocrine+levels+and+risk+tolerance&btnG=) according to Google Scholar on 2023-03-01.
Cuddy (2012)
Figure 77: Amy Cuddy’s TED talk: https://www.ted.com/talks/amy_cuddy_your_body_language_may_shape_who_you_are
Ranehill et al. (2015)
Ranehill, E., Dreber, A., Johannesson, M., Leiberg, S., Sul, S. & Weber, R. A. (2015). Assessing the robustness of power posing: no effect on hormones and risk tolerance in a large sample of men and women [Review of Assessing the robustness of power posing: no effect on hormones and risk tolerance in a large sample of men and women]. Psychological Science, 26(5), 653–656. journals.sagepub.com. https://doi.org/10.1177/0956797614553946
We conducted a conceptual replication study task in both the gain dom with a similar methodology as that employed by Carney loss domain. In t et al. but using a substantially larger sample (N= 200) and a design in which the experimenter was blind to condition. Our statistical power to detect an effect of the magnitude reported by Carney et al. was more than 95% (see the Supplemental Material available online). In addition to the three outcome measures that Carney et al. used, we also studied two more behavioral tasks (risk taking in the loss domain and willingness to compete).
Consistent with the findings of Carney et al., our results showed a significant effect of power posing on self-reported feelings of power. However, we found no significant effect of power posing on hormonal levels in any of the three behavioral tasks.
Methods
- Sampled \(n=100\) after power analysis; then sampled another 100.
- 3 min pose durations
- Participants modeled poses following computer instructions.
- No deception.
Results
](include/img/ranehill-etal-2015-results.png)
Figure 78: Results excerpt from Ranehill et al. (2015)
](include/img/ranehill-etal-2015-fig01.png)
Figure 79: Figure 1 from Ranehill et al. (2015)
Citation data
Paper cited ~314 times (https://scholar.google.com/scholar?hl=en&as_sdt=0%2C39&q=Assessing+the+robustness+of+power+posing%3A+no+effect+on+hormones+and+risk+tolerance+in+a+large+sample+of+men+and+women+%5BReview+of+Assessing+the+robustness+of+power+posing%3A+no+effect+on+hormones+and+risk+tolerance+in+a+large+sample+of+men+and+women%5D&btnG=) according to Google Scholar on 2023-03-01.
Reproducibility notes
- Article was behind a paywall.
- The PDF was available via authenticated access to PSU Libraries.
- The PDF did not easily permit cutting and pasting of text, so it was hard to excerpt those for this document.
- The HTML version of the article was unavailable, so I had to make screenshots of figures.
- There is an OSF site with data and materials.
- The OSF site includes very helpful details about the experiments.
- The data and code were shared, but I don’t have access to the statistical program used (Stata) to rerun the analysis. The data had a
.dtafile extension. This appears to be a plain text, ‘tidy’ data file format that could be used by another program.
Exploring logical/causal diagrams
Causal diagrams are one tool to reveal an authors’ claims and assumptions.
Figure 81: Implicit causal model tested
More typically, we think that feelings cause behaviors.
Figure 82: Conventional causal model
Or that feelings and physiological responses (e.g., hormones like cortisol and testosterone) together influence behaviors.
Figure 83: Conventional causal model II
Neuroscientists might argue that feelings are the outcome of physiological processes like hormone levels and nervous system activity. Also, physiological processes influence one another.
Figure 84: Neuroscience-oriented causal model
To understand the full system of influences, we might need to look at a more complicated causal model. Here, behaviors influence and are influenced by physiology. Behaviors influence, and are influenced by feelings through influences on physiology. However, the evidence presented in Carney et al. (2010) and Ranehill et al. (2015) show that ‘posing’ behavior influences feelings, but not (hormonal) physiology. If the neuroscientific view is correct, there must be another route for behaviors to influence feeling through physiology. That route is probably nervous system activity.
Figure 85: Plausible fully interactive causal model