Embedding Similarity Fails to Predict LLM Comprehension: Evidence from Notation-Replacing Prompt Compression

Read full text Summarise with VIRAT
Authors: Emil Piotr Dobkowski
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

We show that SBERT cosine similarity has no discriminative powerfor notation-replacing (TYPE_B) prompt transformations.DeepSeek-V3 achieves identical task accuracy on natural languageand symbolic prompts (delta=0.000, CI=[-0.040,+0.040], n=100),while SBERT cosine (mean=0.664±0.133) is empirically decoupledfrom task performance. Ablation shows performance gaps arise fromsymbolic parsing ambiguity, not representational differences.Introduces TYPE_B as a diagnostic benchmark for compressionevaluation.

Fields of Study & Subjects

Topic Modeling Natural Language Processing Techniques Multimodal Machine Learning Applications

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
22 March 2026
Language
EN
Document type
Preprint