Detecting Non-Membership in LLM Training Data via Rank Correlations

Read full text Summarise with VIRAT
Authors: Pranav Shetty, Mirazul Haque, Zhiqiang Ma, Xiaomo Liu
Cited by: 0
References: 33

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

As large language models (LLMs) are trained on increasingly vast and opaque text corpora, determining which data contributed to training has become essential for copyright enforcement, compliance auditing, and user trust. While prior work focuses on detecting whether a dataset was used in training (membership inference), the complementary problem -- verifying that a dataset was not used -- has received little attention. We address this gap by introducing PRISM, a test that detects dataset-level non-membership using only grey-box access to model logits. Our key insight is that two models that have not seen a dataset exhibit higher rank correlation in their normalized token log probabilities than when one model has been trained on that data. Using this observation, we construct a correlation-based test that detects non-membership. Empirically, PRISM reliably rules out membership in training data across all datasets tested while avoiding false positives, thus offering a framework for verifying that specific datasets were excluded from LLM training.

Fields of Study & Subjects

Adversarial Robustness in Machine Learning Topic Modeling Privacy-Preserving Technologies in Data

Publication details

Journal / source
Conference of the European Chapter of the Association for Computational Linguistics
Pages
5414–5429
Publication date
1 January 2026
Language
EN