The Limits of Predicting Individual-Level Longevity: Insights From the U.S. Health and Retirement Study

Abstract

Individual-level mortality prediction is a fundamental challenge with implications for life planning, health care, social policies, and public spending. Drawing from the growing body of research on the predictability of life course events, we model and predict individual-level lifespan using 12 statistical and machine learning models and more than 150 predictors derived from the U.S. Health and Retirement Study longitudinal data. Statistical and machine learning models report comparable accuracy and relatively high discriminative performance, but they fail to account for most lifespan heterogeneity at the individual level. We observe consistent inequalities in mortality predictability and risk discrimination, with lower accuracy for men, non-Hispanic Blacks, and low-educated individuals. Additionally, people in these groups show lower accuracy in their subjective predictions of their own lifespan. Finally, top features across groups are similar, with variables related to habits, health history, and finances being relevant predictors. We conclude by highlighting the limits of predicting mortality from one of the richest longitudinal representative surveys in the United States, as well as the context-dependent inequalities across sociodemographic groups, and providing baselines and guidance for future research and public policies.