Dynamic KV Injection: An Attention-Level User Memory System for Large Language Models

Current memory augmentation approaches for Large Language Models (LLMs), such as Retrieval-Augmented Generation (RAG), inject retrieved content by concatenating it to the input prompt. This paradigm consumes valuable context window tokens and requires careful prompt engineering. We propose Dynamic KV Injection (DKI), a training-free, inference-time user-level cross-session memory system that injects Key-Value representations directly into the attention mechanism without consuming prompt tokens. Scope Definition: DKI is designed specifically for user-level memory (personal preferences, session history, user context), not for external knowledge bases or public data retrieval. This focused scope enables architectural simplifications and reduces position encoding risks. Our key contributions include: (1) a paradigm shift from token-level to attention-level memory injection; (2) Memory Influence Scaling (MIS) with continuous strength control α ∈ [0, 1]; (3) Query-Conditioned Memory Projection using FiLM-style modulation; (4) Dual-Factor Gating combining uncertainty and relevance signals; (5) Hybrid Injection Strategy: negative-position K/V injection for stable preferences + suffix-prompt injection (explicit token-level history reference) for dynamic session history; and (6) Standardized Plugin Architecture enabling configuration-based deployment. We introduce the Attention Budget Reallocation Hypothesis: in reasoning-intensive tasks, the marginal benefit of releasing token budget exceeds the marginal cost of increased attention computation. Preliminary experiments on PersonaChat (n=500) show comparable memory recall (86.2% vs 87.3%) with 43.7% reduction in subsequent-turn latency; DKI significantly outperforms RAG by +10-16% at user context lengths >2500 tokens, strongly supporting the budget reallocation hypothesis.