An Efficient Jackknife Model Averaging Method

Abstract

Model averaging, an effective method for reducing uncertainty in data modeling and learning, has been widely applied across various fields, including economics, finance, and operations research. The leave-one-out cross-validation (LOOCV) criterion is a widely used technique to choose weights in model averaging. It has achieved great success, but its application is limited by the high computational complexity, especially when the sample size is large. To address this issue, focusing on generalized linear models (GLMs), we propose a computationally efficient and approximate leave-one-out cross-validation (ALOOCV) weight choice criterion, which no longer suffers from the need to repeatedly solve optimization problems many times. Our theoretical analysis presents the discrepancy between the weight estimators produced by LOOCV and ALOOCV, which has been proven to vanish asymptotically under certain conditions. Moreover, the efficient jackknife model averaging (EJMA) procedure with ALOOCV for GLMs is proposed. The asymptotic optimality under the model misspecification and the convergence rate of the selected weights by ALOOCV are derived, and the over-consistency of the weight estimators and the consistency of averaging coefficient estimators are established if there exists at least one correct model. Additionally, to reduce the computational burden caused by an excessive number of candidate models, we provide a model screening procedure. Numerical experiments show the remarkable computational efficiency exhibited by EJMA relative to the model averaging procedure with LOOCV and the excellent prediction performance of our method over some common model averaging and selection procedures. History: Accepted by Ram Ramesh/Data Science & Machine Learning. Funding: This research was supported by the Postdoctoral Fellowship Program of CPSF [Grant GZC20231478], the National Natural Science Foundation of China [Grants 12501370, 12001534, and 12571300], and the China Postdoctoral Science Foundation [Grant 2024M761782]. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2024.0946 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2024.0946 ). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/ .