The Illusion of Practical Relevance: Rethinking Effect Size in Theory-Testing Psychology

What the paper says

Effect sizes are increasingly promoted in psychological science, sometimes even considered the primary outputs of quantitative research. While effect sizes are essential for statistical purposes related to open science efforts (replication planning, meta-analysis), their widespread interpretation as markers of practical relevance is problematic. Specifically, this article argues that effect sizes approximate practical relevance in applied research, which aims to mirror real-world environments, but not in theory-testing research, which aims to isolate causal mechanisms. Three reasons for this limitation are outlined. First, theory-testing effects often differ fundamentally from practical effects the theory aims to explain. Second, effect sizes vary strongly with design characteristics; there is no latent “true score” effect size at a theoretical level. Third, the practical impact of an effect may fade out or accumulate over time. Together, these arguments show that the magnitude of theory-testing effects does not provide reliable information about the magnitude of real-world effects. I conclude with recommendations for interpreting and reporting effect sizes in theory-testing research, emphasizing their utility for cumulative psychological science while cautioning against their uncritical interpretation as indicators of practical relevance.