Bridging KANs and Tabular Deep Learning: Feature Embeddings and Efficient Ensembling

Despite recently achieving breakthrough performance on tabular data, MLP-based approaches like TabM still miss complex feature interactions, leaving room for further improvement. Meanwhile, Kolmogorov-Arnold Networks (KANs) offer a promising alternative by switching from weights and fixed activations to learnable univariate functions on edges. This allows to capture intricate patterns in tabular data. However, their potential remains largely unexplored in the context of modern tabular deep learning – particularly regarding advanced ensembling and feature engineering techniques. We substantially enhance the application of KANs for structured data modeling. Specifically, we pioneer the use of advanced numerical embedding schemes, modern optimizers (e.g., Muon and AdEMAMix), and parameter-efficient ensembling built upon the KAN architecture. Moreover, we perform an ablation study across architecture design and key hyperparameters to derive concrete recommendations for practitioners. Through extensive evaluation on public tabular benchmarks, our enhanced KAN-based architectures achieve 5 wins across the benchmark, compared to 3 for the strongest MLP baseline, while maintaining computational efficiency.