BERT-JEPA: Reorganizing CLS Embeddings for Language-Invariant Semantics

Taj Gillin; Adam Lalani; Kenneth Zhang; Marcel Mateos Salles

doi:10.48550/arxiv.2601.00366

BERT-JEPA: Reorganizing CLS Embeddings for Language-Invariant Semantics

arXiv (Cornell University) 2026 Preprint Open access

Read full text

Authors: Taj Gillin, Adam Lalani, Kenneth Zhang, Marcel Mateos Salles

DOI: 10.48550/arxiv.2601.00366

Cited by: 0

Abstract

Joint Embedding Predictive Architectures (JEPA) are a novel self supervised training technique that have shown recent promise across domains. We introduce BERT-JEPA (BEPA), a training paradigm that adds a JEPA training objective to BERT-style models, working to combat a collapsed [CLS] embedding space and turning it into a language-agnostic space. This new structure leads to increased performance across multilingual benchmarks.

Fields of Study & Subjects

Topic Modeling Advanced Graph Neural Networks Machine Learning in Healthcare

Publication details

Journal / source: arXiv (Cornell University)
Publication date: 1 January 2026
Document type: Preprint