Study Of The Week: Choices, Oh My, The Choices, Sensible Medicine, 15 August 2022

The paper above describes how a study’s results depend upon how its data was analyzed. The author, Dr. Mandrola, recalls a study that was done to test the hypothesis that different types of analysis yield different results – from the same data.

Nosek and colleagues recruited 29 teams of expert researchers to analyze one data set to answer one simple question: were professional soccer referees more likely to give red cards to dark-skin-toned players than light-skin-toned players?

The 29 teams of researchers analyzed the same data in 29 different ways.

Two-thirds of the expert teams of data scientists detected a significant result and one-third found no statistical difference. Two teams of experts found results that were highly suggestive of implicit bias amongst referees.

Studies describe the method that was used to analyze data. It’s usually only one method.

As if it wasn’t already difficult to trust results.