Efficient, automated, and explainable deep learning pipelines

Over the years, deep learning techniques have achieved remarkable success in various domains, incrementally pushing frontiers. However, as the number of approaches increases, it becomes increasingly difficult to select the optimal technique for a given task. Moreover, these approaches are parametrized by hyperparameters that require extensive tuning to achieve optimal predictive performance. Selecting an optimal technique–hyperparameter configuration pair is a non-trivial task that requires domain knowledge and cannot be performed in a brute-force manner given limited resources. Additionally, deep learning methods are black-box by nature, which makes it difficult to ensure fairness and eliminate harmful biases. The majority of these challenges are addressed by the field of automated machine learning (AutoML), whose main goal is to find the optimal combination of an algorithm and its corresponding hyperparameter configuration in an end-to-end manner. Although AutoML has recently garnered more attention, most existing approaches are difficult to apply to expensive deep learning algorithms, where a single run requires extensive computational resources. In this thesis, we address several key challenges in combining AutoML with deep learning to promote wider adoption of AutoML in the current deep learning era. First, we address the adoption of automated deep learning pipelines for tabular data, a domain traditionally dominated by tree-based methods that achieve state-of-the-art results with lower resource requirements. Rather than relying on specialized architectures, our approach uses feedforward neural networks combined with per-task regularization strategies, with all components selected via end-to-end hyperparameter optimization. Experimental results show that our method attains state-of-the-art accuracy on a well-established benchmark, demonstrating the potential of automated deep learning in the tabular setting. Second, to address the black-box nature of deep learning algorithms, we propose converting the backbone architecture into a hypernetwork rather than predicting the target variable directly. The hypernetwork predicts the weights of a logistic regression model, which can then be interpreted as feature importances. This simple modification is applicable to most deep learning architectures and provides interpretability as a free lunch. Our results indicate that the proposed reformulation is on par with state-of-the-art interpretability methods and outperforms existing explainable deep learning approaches. Third and fourth, we tackle the efficiency of existing hyperparameter optimization methods by introducing a dynamic scheduling strategy that early-stops diverging or stagnant configurations to allocate resources more effectively. Our first approach employs a deep-kernel Gaussian process surrogate without assuming a specific learning curve shape. The second approach builds upon the first by using an efficient ensemble of feedforward networks as a surrogate model and by incorporating a power-law assumption for learning curves. Both methods achieve superior anytime performance compared to competing approaches and, in most cases, match the final performance of the best competitor in only 25\% of the total time. Finally, we combine all our contributions into an efficient, cost-aware, and meta-learned AutoML framework, which selects the optimal pre-trained model to fine-tune and its corresponding regularization strategies via efficient hyperparameter optimization. Our approach achieves state-of-the-art performance, outperforms large foundational models, and showcases the power of combining AutoML with deep learning approaches.