Validation of machine learning based scenario generators

Abstract

Machine learning (ML) methods are becoming increasingly important for designing economic scenario generators for internal models. Validating data‐driven models requires different methods than validating classical, theory‐based models. We discuss two novel aspects of such validation: first, checking the multivariate distribution of risk factors, and second, detecting unwanted memorization effects. The first task is necessary because, in ML‐based methods, dependencies are driven by data rather than derived from a financial‐mathematical theory. To address this first issue, we propose using an existing test from the literature. The second task is necessary because it cannot be ruled out that ML‐based models merely reproduce empirical data rather than generating new scenarios. For the second issue, we introduce a novel memorization ratio together with a thorough discussion. We include numerical experiments based on real market data and validate a simple autoencoder‐based scenario generator.