Computational Experiments on Some Novel Tree-based Classification Algorithms

Abstract

First, we propose and implement a computational framework to benchmark the approximation capabilities of different classification tree algorithms against that of neural networks. This framework examines a tree's predictive accuracy on synthetic data generated by multi-layer neural networks of various sizes. Our computations suggest that greedily grown trees with oblique decision boundaries are an efficient learner model, capable of condensing the knowledge stored by sophisticated neural networks into a compact (depth ≤ 10) tree representation. Besides computational results, we introduce a Python-based implementation of an oblique classification tree algorithm and scripts to visualize the decision boundaries of tree classifiers in 2D. Last but not least, we generalize Leo Breiman's idea of stacking tree predictors to solve binary classification tasks. Our convex optimization formulation is efficient to execute, and experiments on several natural datasets show that stacking consistently improves the test accuracy of tree predictors built using the CART algorithm.