Learning Discriminative Syntax-Semantic Patterns With Transformer-Based Contrastive Learning

This paper proposes a dual-encoder framework that learns syntax- and semantics-aligned sentence embeddings using contrastive learning. The proposed Syntax-Semantic Contrastive Pretraining (SSCP) model employs two transformer encoders to separately model syntactic structure and semantic content, which are aligned in a shared embedding space via a symmetric contrastive objective. Across standard benchmarks, SSCP consistently outperforms strong baselines such as BERT, SimCSE, and SyntaxBERT. In particular, SSCP improves Spearman correlation on STS-B by +1.4 points over SimCSE, increases PAWS paraphrase accuracy by +2.5 points, and achieves 96.1% accuracy on TREC question classification, exceeding existing syntax-aware models. Probing experiments further show gains of up to +4–5 points on syntactic structure prediction tasks, confirming that SSCP preserves grammatical information while maintaining semantic robustness. These results demonstrate that explicitly aligning syntactic and semantic views yields representations that are more discriminative, interpretable, and robust than single-view or syntax-augmented approaches, positioning SSCP as a principled multi-view pretraining strategy for structure-aware language understanding.