The impact of random tree depth—a novel randomization process for ensemble methods

What the paper says

The induction of additional randomness in parallel and sequential ensemble methods has proven to be worthwhile in many aspects. In this manuscript, we propose a novel random tree depth approach for sequential and parallel tree-based approaches. In particular, we apply the concept of a random tree depth for the representative methods of Boosting (MART) and Random Forests. Both approaches are then investigated with respect to their runtime and prediction performance. We call the resulting methods Random Depth Boosting and Random Depth Forest . Initially, an exemplary experiment on a simple data set indicates that combining Random Depth with MART can enhance prediction performance, while the impact on Random Forests remains limited. This observation aligns with a heuristically intuitive understanding of how randomizing tree depth interacts with bagging and boosting dynamics. Though a full theoretical analysis lies beyond the scope of this work, the underlying mechanisms can be well-motivated and heuristically explained. Building on these insights, a Monte Carlo simulation study investigates the effects on both artificial tree-shaped data sets with varying numbers of final partitions and on a selection of real-world classification and regression datasets. The results show that Random Depth Boosting offers relevant improvements for MART-based models. Additionally, the randomization of tree depth can reduce computation time by up to $$40\%$$ , making the approach an attractive option in time-sensitive applications.