Continual Pretraining on Encrypted Synthetic Data for Privacy-Preserving LLMs

Read full text Summarise with VIRAT
Authors: Honghao Liu, Xuhui Jiang, Can Xu, Cehao Yang, Yiran Cheng, Lionel Ming-shuan Ni, Jian Guo
Cited by: 0
References: 33

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Preserving privacy in sensitive data while pretraining large language models on small, domain-specific corpora presents a significant challenge.In this work, we take an exploratory step toward privacy-preserving continual pretraining by proposing an entity-based framework that synthesizes encrypted training data to protect personally identifiable information (PII).Our approach constructs a weighted entity graph to guide data synthesis and applies deterministic encryption to PII entities, enabling LLMs to encode new knowledge through continual pretraining while granting authorized access to sensitive data through decryption keys.Our results on limited-scale datasets demonstrate that our pretrained models outperform base models and ensure PII security, while exhibiting a modest performance gap compared to models trained on unencrypted synthetic data.We further show that increasing the number of entities and leveraging graphbased synthesis improves model performance, and that

Fields of Study & Subjects

Privacy-Preserving Technologies in Data Topic Modeling Cryptography and Data Security

Publication details

Journal / source
Findings of the Association for Computational Linguistics: EACL
Pages
408–425
Publication date
1 January 2026
Language
EN