Application of Resolution Regression and Resolution Graphs in Evaluating Probability Forecasts Generated Using Binary Choice Models

Abstract

Binary choice models are widely used in econometric modeling when the dependent variable corresponds to discrete outcomes. With appropriate decision rules, these models provide predictions of binary choices generated from predicted probabilities. The accuracy of these predictions in terms of classifying probabilities to events that occurred versus those that did not is a key issue. The use of expectation-prediction success at present is the standard method used to assess the accuracy of these predictions. However, this method is limited in its ability to correctly classify probabilities in the absence of appropriate predetermined cut-off levels. We propose alternative methods to classify probabilities generated through binary choice models, namely resolution graphs and resolution regressions that measure the ability to sort predicted probabilities against observed outcomes. Using probabilities generated from the use of logit models applied to purchasing decisions of various non-alcoholic beverages made by U.S. households, we compare probability sorting power using expectation-prediction success as well as resolution graphs and resolution regressions. Based on expectation-prediction success, the logit models performed better at classifying outcomes related to purchasing isotonic drinks, regular soft drinks, diet drinks, bottled water, and tea. Based on resolution regressions, the null hypothesis of perfect sorting of probabilities was rejected for all non-alcoholic beverages. Although the logit models generated upward-sloping resolution graphs as expected, they were relatively flat compared to the 45-degree perfect sorting line. Going forward, we recommend using resolution regression and resolution graphs to capture sorting of probabilities in addition to the conventional metrics used in ascertaining the ability of binary choice models to predict out-of-sample behavior.