Learning Sparse Decision Trees via Transformer Variational Auto-Encoders
TREVIS, an approach for learning decision trees with complex objectives, is proposed. It uses a Tree Transformer Variational Auto-Encoder (TTVAE) to map decision trees to latent representations, enabling gradient-based optimization via a differentiable surrogate model. Experiments show TREVIS discovers decision trees matching predictive performance of existing near-optimal algorithms while improving structural sparsity.
A research paper introduces TREVIS, a method for learning decision trees that jointly optimizes predictive performance and structural sparsity. By mapping decision trees into a continuous latent space using a Tree Transformer Variational Auto-Encoder, TREVIS replaces the discrete search space with a continuous one, allowing gradient-based optimization. Experimental results indicate that TREVIS achieves predictive performance comparable to existing near-optimal algorithms while producing sparser trees.
The key technical contribution is the use of a Transformer-based variational auto-encoder to learn a continuous latent representation of decision trees. This enables gradient-based optimization over tree structures, which are typically discrete and combinatorial. The differentiable surrogate model allows joint optimization of predictive accuracy and structural sparsity, a non-trivial extension of standard decision tree learning.
Decision trees are widely used in high-stakes domains due to their interpretability. TREVIS's ability to produce sparser trees without sacrificing accuracy could make interpretable models more attractive in regulated industries such as finance and healthcare, where model simplicity is often required for compliance and auditability.
Sparser decision trees can reduce computational costs for inference and storage, and improve human interpretability. This may lower barriers to deploying interpretable models in production systems, especially where explainability is a regulatory or customer trust requirement.
Future work may explore applying TREVIS to other tree-based models or additional objectives such as fairness or robustness. The approach could also be extended to ensemble methods or used to generate diverse tree structures for interpretability analysis. Observing whether the method scales to larger datasets and deeper trees will be important.