Semantic Domain Routing for Efficient Transformer Inference: Reducing FLOPs via External Layer Masking

Read full text Summarise with VIRAT
Authors: VishnuTeja Poola
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Large language models are computationally expensive at inference time, activating all layers for every query regardless of its domain or complexity. We propose a lightweight external semantic router that selectively activates domain-relevant transformer layers at inference time, requiring zero modifications to the base model. Using BART-based zero-shot classification as the routing mechanism, we demonstrate that routing queries to domain-specific layer subsets reduces FLOPs by an average of 43.8% on GPT-2 Medium (345M parameters) while maintaining full model output quality on routed queries. Single-domain queries achieve up to 54.2% FLOPs savings. At a scale of 1 million queries per day, this translates to approximately 6,606 TFLOPs saved daily. We further present a self-training pipeline where the router automatically improves by labeling large question datasets without any human annotation. Our approach is modelagnostic, plug-in compatible with any pretrained transformer, and runs entirely on CPUclass hardware

Fields of Study & Subjects

Topic Modeling Advanced Neural Network Applications Domain Adaptation and Few-Shot Learning

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
28 March 2026
Language
EN
Document type
Preprint