DeBERTa-AT: A DeBERTaV3 Variant Fine-Tuned on Air Traffic Data

Large language models (LLMs) offer a powerful platform and can leverage tools to extract relevant informa- tion and provide recommendations for air traffic users. These can range from classification of voluntary safety reports to discovering shared successful corrective actions and creating more accurate transcriptions of air traffic control conversations. However, using LLMs requires leveraging the knowledge con- tained in legacy aviation datasets, which can be time-consuming and compute-intensive. This paper describes the creation of DeBERTa-AT, a variant of the DeBERTaV3 model fine-tuned on air traffic data. DeBERTa-AT continues pretraining using the replaced token detection (RTD) pretraining task with gradient- disentangled embedding sharing (GDES) to offer improved per- formance and faster training convergence on downstream tasks in the aviation field. DeBERTa-AT is evaluated on two downstream binary clas- sification tasks: aviation-specific constraint classification and document classification, with text datasets created from let- ters of agreement (LOAs). During downstream fine-tuning, im- provements were shown in performance as compared with the DeBERTaV3-base model with frozen embeddings. The results show that models combined with sample-efficient continual pre- training can offer substantially better results on domain specific data when less training data is available.