How science works (or should)

2024-09-04 Wed

Rick Gilmore

Prelude

Overview

Last time…

  • Are the tips for reading a scientific paper helpful?
  • What about acronyms and jargon?
  • What to do if you’re really stuck?
    • Try making a diagram

Example

The effect of television violence on boys’ aggression was investigated with consideration of teacher-rated characteristic aggressiveness, timing of frustration, and violence-related cues as moderators. Boys in Grades 2 and 3 (N=396) watched violent or nonviolent TV in groups of 6, and half the groups were later exposed to a cue associated with the violent TV program…

…They were frustrated either before or after TV viewing. Aggression was measured by naturalistic observation during a game of floor hockey. Groups containing more characteristically high-aggressive boys showed higher aggression following violent TV plus the cue than following violent TV alone, which in turn produced more aggression than did the nonviolent TV condition…

…There was evidence that both the violent content and the cue may have suppressed aggression among groups composed primarily of boys low in characteristic aggressiveness. Results were interpreted in terms of current information-processing theories of media effects on aggression.

(Josephson, 1987)

What participants did

flowchart LR
  A[Watch TV segments] --> C[Exposed to cue] --> B[Play Hockey]
  D[Watch TV segments] --> B

What experimenters manipulated

  • I control the independent variables
    • Boys (whose parents consented)
    • Groups of boys
    • Order of TV segments: violent_first/last
    • Timing of “frustration”–cartoons with tech problems: before/after violent segment
    • Pre-game presentation of violence “cue”: present/absent

What experimenters measured

  • The dependent (outcome) variables
    • Boys’ “characteristic aggressiveness” via teacher ratings
    • Boys’ “observed aggression” (in hockey game)

What would you predict?

What would you predict?

  • “characteristic aggressiveness” \(\rightarrow\) “observed aggression”
  • Turning predictions into statistical/quantitative tests
  • \(aggrChar + vioTV + frustr + cue \rightarrow aggrObs\).

\(b_1*charAggr + b_2*vioTV +... = aggrObs + err\)

  • where the \(b_i\) “weight” the predictors based on how well they predict \(aggrObs\)

…Groups containing more characteristically high-aggressive boys showed higher aggression following violent TV plus the cue than following violent TV alone…

\[aggrChar + vioTV + cue >\]

\[aggrChar + vioTV + noCue\]

Today

How science works (or should)

How science works (or should)

Discuss (Ritchie, 2020, Chapter 1)

Chapter 1: How Science Works

  • “Science is a social construct.” (Oreskes, 2019)
    • How so?
    • What are the consequences?

What is the “process” of science?

Figure 1 from Munafò et al. (2017)

Telling (facts/findings) AND showing

…and it has to be shown why that other theory can or cannot be the true one: And until this is shown, and until we know how it is shown, we do not understand the grounds of our opinion.

John Stuart Mill quoted in Ritchie (2020), p. 14

From replication to trust

…Only when certain events recur in accordance with rules or regularities, as is the case with repeatable experiments, can our observations be tested–in principle–by anyone. We do not take even our own observations quite seriously, or accept them as scientific observations, until we have repeated and test them…

Karl Popper quoted in Ritchie (2020), p. 23

From replication to trust

…Only by such repetitions can we convince ourselves that we are not dealing with a mere isolated ‘coincidence’…

Karl Popper quoted in Ritchie (2020), p. 23

  • In what other realm(s) do we not require someone stating an idea or finding to show us how they came to that idea or finding?
  • Are scientific experiments really repeatable by anyone?
  • In what other realm(s) do we not expect repetition or replication before trusting an idea?

Science’s social nature does come with weaknesses, however. Because scientists focus so much on trying to persuade their peers, which is the way they get those studies through peer review and oward to publication, it’s all too easy for them to disregard the real object of science: getting us closer to the truth.

Ritchie (2020), Chapter 1, pp. 14-15

Questions about Ritchie

  • Do you agree that a focus on “persuading their peers” can lead researchers to lose sight of the “real object of science”?
    • Are there other examples (outside of science), where a focus on persuading others can change a person’s focus from more important goals?
  • What’s the alternative for scientists to trying to persuade their peers?
  • What if scientific research serves multiple goals?

Discuss Nosek & Bar-Anan (2012)

  • Yoav Bar-Anon
    • Social Psychologist
    • Professor at Tel Aviv University

Scientific Utopia: I. Opening Scientific Communication

  • What’s Utopia
  • Where’s Part II?

Aims & Claims

Existing norms for scientific communication are rooted in anachronistic practices of bygone eras making them needlessly inefficient. We outline a path that moves away from the existing model of scientific communication to improve the efficiency in meeting the purpose of public science—knowledge accumulation.

Nosek & Bar-Anan (2012)

We call for six changes: (a) full embrace of digital communication; (b) open access to all published research; (c) disentangling publication from evaluation; (d) breaking the “one article, one journal” model with a grading system for evaluation and diversified dissemination outlets; (e) publishing peer review; and (f) allowing open, continuous peer review.

Nosek & Bar-Anan (2012)

We address conceptual and practical barriers to change and provide examples showing how the suggested practices are being used already. The critical barriers to change are not technical or financial; they are social. Although scientists guard the status quo, they also have the power to change it.

Nosek & Bar-Anan (2012)

Inefficiencies in scientific communication

  • No communication
  • Slow communication
  • Incomplete communication
  • Inaccurate communication
  • Unmodifiable communication

Nosek & Bar-Anan (2012), , pp. 25-26

Questions about (Nosek & Bar-Anan, 2012)

  • What are the impacts of these factors?
  • Do you agree that scientific communication should be made more efficient? Why or why not?

Questions about (Nosek & Bar-Anan, 2012)

  • Do you think all research scholarship should be open access?
    • Who should pay for the costs of publication, maintaining the data?
    • What’s wrong with enterprising people making a living by providing these services?

Questions about (Nosek & Bar-Anan, 2012)

  • What about the idea of separating publication from evaluation?
    • How can scientists discern good science from bad?
    • How can the general public?
  • What do you think reviewers would say about publishing peer review?
    • Some journals do this already, e.g. Frontiers

Common themes across Nosek & Bar-Anan (2012)

  • Goals and purposes of science
  • Utopian, idealistic aims
  • Efficiency in achieving those goals
  • High quality versus low quality science
  • Ethics, scientific integrity
  • Means of scientific communication
  • Who has access, who should
  • How are findings, articles, individuals evaluated

Next time

Friday, September 06

Work Session: Reading a paper; Evaluating its claims

Resources

Talk by Brian Nosek, (University of California Television (UCTV), 2018)

Watching the talk by Nosek is not required. But he’s a very good speaker and an inspiring person.

This talk was produced using Quarto, using the RStudio Integrated Development Environment (IDE) version 2024.4.2.764.

The source files are in R and R Markdown, then rendered to HTML using the revealJS framework.

References

PSYCH 490.012 • Fall 2024

Josephson, W. L. (1987). Television violence and children’s aggression: Testing the priming, social script, and disinhibition predictions. Journal of Personality and Social Psychology, 53(5), 882–890. https://doi.org/10.1037/0022-3514.53.5.882
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Sert, N. P. du, … Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021. https://doi.org/10.1038/s41562-016-0021
Nosek, B. A., & Bar-Anan, Y. (2012). Scientific utopia i: Opening scientific communication. Psychological Inquiry, 23(3), 217–243. https://doi.org/10.1080/1047840X.2012.692215
Oreskes, N. (2019). Why Trust Science. Princeton University Press.
Ritchie, S. (2020). Science fictions: Exposing fraud, bias, negligence and hype in science (1st ed.). Penguin Random House. Retrieved from https://www.amazon.com/Science-Fictions/dp/1847925669
University of California Television (UCTV). (2018, May). Improving openness and innovation in scholarly communication with brian nosek. Youtube. Retrieved from https://www.youtube.com/watch?v=YEwqohAjfZc