Adaptive Ridge Approach to Heteroscedastic Regression

What the paper says

The adaptive ridge (AR) is an iterative scheme that has emerged in the past decade as an $$l_{2}$$ -based alternative to the traditional $$l_{1}$$ - and $$l_{0}$$ -based methods for variable estimation and model selection. In this paper, we propose using AR in heteroscedastic linear regression, where the noise variance is modelled via a log-linear function of covariates. We generalize and sharpen the existing method in three directions. (1) We prove several theoretical results of the AR estimators with a fixed number of iterations rather than focusing solely on probabilistic large-sample aspects such as the oracle property, enhancing the understanding of the evolution of AR estimates from both theoretical and computational viewpoints. (2) Thanks to the proposed log-linear variance component, we provide a computationally efficient iterative estimation method for both mean and variance structures in a heteroscedastic framework, showing new asymptotic distributional and tightness results that have been missing in the previous related studies. (3) We work with generalized noise structures and detail the conditions required for results to hold. We then examine the model selection accuracy and generalization error through simulations and real-data examples, demonstrating the practical utility of AR in the log-linear heteroscedastic setting. We will also discuss the potential applicability of AR to more complex models, hopefully paving the way for future research into scalable and adaptive regularization methods for modern statistical learning.