Finding Distributions that Differ, with False Discovery Rate Control

Abstract

Summary We consider the problem of comparing a reference distribution with several other distributions. Given a sample from both the reference and the comparison groups, we aim to identify the comparison groups whose distributions differ from that of the reference group. Viewing this as a multiple-testing problem, we introduce a methodology that provides exact, distribution-free control of the false discovery rate. To do so, we introduce the concept of batch conformal p-values and demonstrate that they satisfy positive regression dependence across the groups (Benjamini & Yekutieli, 2001), thereby enabling control of the false discovery rate through the Benjamini–Hochberg procedure. The proof of positive regression dependence introduces a novel technique for the inductive construction of rank vectors with almost-sure dominance under exchangeability. We evaluate the performance of the proposed procedure through simulations. Despite being distribution-free, in some cases it shows performance comparable to methods with knowledge of the data-generating normal distribution, and it further has more power than direct approaches based on conformal out-of-distribution detection. Furthermore, we illustrate our methods on a hepatitis C treatment dataset, where they identify patient groups with large treatment effects, and on the Current Population Survey dataset, where they identify subpopulations with long working hours.