Semantic Domain Routing for Efficient Transformer Inference: Reducing FLOPs via External Layer Masking
Summary
No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.
Summarise with VIRAT Free to try with your Science Catalogs account.Abstract
Large language models are computationally expensive at inference time, activating all layers for every query regardless of its domain or complexity. We propose a lightweight external semantic router that selectively activates domain-relevant transformer layers at inference time, requiring zero modifications to the base model. Using BART-based zero-shot classification as the routing mechanism, we demonstrate that routing queries to domain-specific layer subsets reduces FLOPs by an average of 43.8% on GPT-2 Medium (345M parameters) while maintaining full model output quality on routed queries. Single-domain queries achieve up to 54.2% FLOPs savings. At a scale of 1 million queries per day, this translates to approximately 6,606 TFLOPs saved daily. We further present a self-training pipeline where the router automatically improves by labeling large question datasets without any human annotation. Our approach is modelagnostic, plug-in compatible with any pretrained transformer, and runs entirely on CPUclass hardware
Fields of Study & Subjects
Topic Modeling
Advanced Neural Network Applications
Domain Adaptation and Few-Shot Learning
Publication details
- Journal / source
- Zenodo (CERN European Organization for Nuclear Research)
- Publication date
- 28 March 2026
- Language
- EN
- Document type
- Preprint