Cognitive Chunking for Soft Prompts: Accelerating Compressor Learning via Block-wise Causal Masking

Providing extensive context via prompting is vital for leveraging the capabilities of Large Language Models (LLMs).However, lengthy contexts significantly increase inference latency.To mitigate this issue, context compressionparticularly soft prompt compression-has emerged as a widely studied solution, which converts long contexts into shorter memory embeddings via a trained compressor.Existing methods typically compress the entire context indiscriminately into a set of memory tokens, requiring the compressor to capture global dependencies and necessitating extensive pre-training data to learn effective patterns.Inspired by the chunking mechanism in human working memory and our empirical observation of spatial specialization within memory embeddings, we propose Parallelized Iterative Compression (PIC).By simply modifying the Transformer's attention mask, PIC explicitly restricts the receptive field of memory tokens to sequential local chunks, thereby significantly reducing the difficulty of compressor training.Experiments across multiple downstream tasks demonstrate that PIC outperforms competitive baselines while dramatically expediting the training process.Finally, we provide an investigation into the emergence of spatial specialization, revealing how tuning strategies and explicit structural priors shape the latent structure of the memory embeddings.