Maximizing experimental power through dynamic treatment assignment

Abstract

Experimental methods have gained significant traction in agricultural economics in recent decades through the growing use of laboratory experiments, lab‐in‐the‐field studies, and experimental designs embedded in online surveys. However, replicability remains a persistent challenge and a critical benchmark for rigor in experimental economics research. Previous studies showed a strong correlation between statistical power and replicability, highlighting the importance of combining sample size with optimal study design to maximize power in experimental economics research. Despite power analysis being well established in randomized controlled trials, experimental economists have had to rely on rules of thumb (e.g., allocate equal number of subjects to each group), which may not guarantee optimal power. In this study, we propose an adaptive treatment assignment procedure to improve statistical power in experimental studies, using an experimental design that updates the researcher's beliefs about the standard deviation ratio of outcomes under different treatments. We simulate experimental data collection under different updating rules to test the expected gains in power compared to the naïve (conventional) method of equal treatment assignment. Our results show significant gains in power under our proposed adaptive procedures compared to equal treatment assignment. These gains in power are directly proportional to the standard deviation ratio between treatments. Importantly, the power achieved under our proposed adaptive treatment assignment procedures was consistently almost identical to the maximum theoretical power achievable given the study parameters. We thus provide a practical and easy‐to‐use tool that can help researchers optimize treatment assignment in their experimental studies to maximize power.