Legal Overfitting

Abstract

A central concern of machine learning is overfitting—which occurs when a prediction model includes too many explanatory variables and predicts noise. The problem with overfitting is that it leads the model to poorer out-of-sample predictions because it misattributes causal significance to irrelevant variables. We argue that a similar phenomenon reduces the quality of precedential reasoning. A judge or lawyer trying to reconcile prior authoritative opinions that are decided with some noise to make the best prediction about the outcome of a new case may misattribute causal significance to factors that do not offer precedential guidance. Machine learning has developed a series of estimation and diagnostic techniques to reduce the likelihood of overfitting. For example, it is standard to train models on a subset of the available data and then test how well the model predicts out-of-sample. This article argues that the quality of precedential reasoning would be improved if judges used analogous techniques in deciding cases. Lawyers might also use these techniques to improve their ability to predict what the law is. We present a decision tree estimation of copyright law’s fair use defense to illustrate the phenomenon of overfitting and how it might be limited.