Graph Neural Networks Improve Quantized Transformers by Incorporating Global Relationships

Transformers are established in natural language processing (NLP). Their capabilities improve with model size, but training and hosting larger models are computationally demanding. Quantization of model parameters can reduce cost; however, it generally leads to a significant loss in model performance. We introduce BitTransGNN which improves the performance of transformers quantized to low-bit precision by integrating their architecture with Graph Neural Networks (GNNs) during training. BitTransGNN substantially reduces and often closes the performance gap between quantized transformers and their full-precision counterparts while retaining the computational efficiency advantages provided by quantization, including lower memory and energy costs. This is possible since transformers and GNNs capture different characteristics of data. Transformers excel in modeling contextual semantics. GNNs excel in modeling very long-range relationships extracted from the entirety of texts beyond the reach of transformers' contextual window sizes. Alongside the transductive joint model, we introduce inductive variants of BitTransGNN, which transfer the knowledge of the joint framework into a standalone quantized transformer. BitTransGNN outperforms quantized transformer baselines trained without GNN integration with minimal additional overhead, and shows competitive performance against full-precision baselines across multiple language benchmarks. Its inductive variants also successfully encapsulate the information captured by the joint model into a standalone quantized transformer, achieving consistent performance gains over quantized baselines with zero additional inference cost. Furthermore, by combining advanced post-training quantization methods with BitTransGNN, we show that its improvements are complementary to those attainable through advancements in quantization.