To vary or not to vary: A flexible empirical Bayes factor for testing variance components

Abstract

Random effects are the gold standard for capturing structural heterogeneity, such as individual differences or temporal dependence. Yet testing their presence is difficult because variance components are constrained to be non-negative, creating a boundary problem. This paper introduces a flexible empirical Bayes factor (EBF) for testing random effects. Instead of testing whether a variance component equals zero, the EBF evaluates the equivalent hypothesis that all random effects are zero. The approach avoids manual prior specification: the distribution of the random effects is modeled at the lower level and estimated directly from the data, yielding an "empirical" Bayes factor. Using a Savage-Dickey density ratio, the EBF requires only the full model fit, eliminating the need to estimate multiple models with alternative random-effects structures. The method enables testing a single random effect as well as multiple, potentially correlated, random effects simultaneously. Simulation studies examine the operating characteristics of the criterion. To illustrate its breadth, the EBF is applied to several widely used models in psychological research and related fields, including generalized linear crossed mixed-effects models, spatial random-effects models, dynamic structural equation models, random-intercept cross-lagged panel models, and nonlinear mixed-effects models.