Token-Invariant Contrastive Pretraining for Robust Domain Adaptation in Text Classification

View at publisher Summarise with VIRAT
Authors: Aadhish C, Aravinthkumar P, Naagadiyanesh. N, Santhosh Kumar R
Cited by: 0
References: 14

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Domain adaptation is still an important issue in text classification, since models often break when faced with new words and semantic drifting. Current methods often only align high-level sentence or document embeddings while overlooking the evolution of individual token-level behavior, which forms the basic units of representation. This paper presents a lightweight, domain-aware Token Sorting Contrastive Pretraining (TSCP) framework to filter and prioritize domain-invariant tokens to facilitate representation learning to address distributional shifts. TSCP calculates the variance of token frequency between the source and target domains, emphasizing transferable and stable vocabulary via contrastive learning with a de-emphasis on domain-specific or noisy words. There is no modification to the architecture, nor the addition of parameters, making the approach feasible and efficient. We adopt two benchmarking datasets to evaluate TSCP: the Amazon Review Polarity dataset$(3.6 \mathrm{m}$labeled sentences collected over 18 years) and the AG's News Topic Classification dataset (having more than$\mathbf{1 m}$news passages). TSCP improves cross-domain sentiment classification accuracy on Amazon from 85.0% to 87.2% and on AG's News from 88.0% to 89.3%, compared with baselines including fine-tuned BERT and SimCSE. We conducted all experiments on an average consumer laptop (Ryzen 55500 U, 16 GB RAM) for accessibility purposes. TSCP outperforms the state-of-the-art models and emerging applications in sentiment analysis, news filtering, and content moderation. In the future, we plan to study dynamic token weighting, multilingual adaptability, and apply it to generative pretraining frameworks.

Fields of Study & Subjects

Domain Adaptation and Few-Shot Learning Sentiment Analysis and Opinion Mining Topic Modeling

Publication details

Journal / source
International Conference on Cognitive Systems and Computer Interaction (ICoSCI)
Pages
1–5
Publication date
15 January 2026
Language
EN