Ten common statistical errors from all phases of research, and their fixes

Abstract

Statistical errors are surprisingly common and threaten the credibility of biomedical research.1–3 This issue has received increasing attention in recent years, including in the popular media,4–7 prompting calls for statistical reform. Debates about how to improve statistical practice have largely focused on the choice of inferential method; in particular, P values and statistical significance testing have come under scrutiny.8–12 This may give applied scientists the mistaken impression that doing better statistics is as simple as changing one’s inferential method—for example, that replacing a frequentist t-test with a Bayesian one, or a P value with a confidence interval, will transform a statistically unsound study to a statically sound one. In fact, the errors that most threaten a study’s validity usually occur long before a researcher calculates a P value. As Jeffrey Leek and Roger Peng write: “Arguing about the P value is like focusing on a single misspelling, rather than on the faulty logic of a sentence.”13 Like Leek and Peng, we believe that inappropriate study design, unclearly formulated research questions, poor data handling, and a lack of statistical thinking and numerical literacy pose even greater threats to science than misused P values. In this article, we draw attention to statistical errors that occur in all steps of the research pipeline. We present examples of 10 common mistakes that occur during four phases of research: study design; data wrangling and cleaning; data analysis; and reporting. The examples are hypothetical but are based on real cases we have encountered. We also discuss potential solutions to help researchers avoid these mistakes.