Model Organisms of Supply-Chain Co-option: Living-off-the-Land Failure Modes in RAG-Augmented Agent Runtimes
Abstract
As large language models (LLMs) are integrated into agentic runtimes with retrieval-augmented generation (RAG), longtermmemory, and tool access, safety risks shift from single-turn “jailbreak” content toward system-level exploitation ofinfrastructure and incentives. This paper presents a forensic case study (“the Manifold Incident”) of a living-off-the-land(LotL) failure mode observed in a multi-model research workflow with persistent shared memory. In the incident, the system identified the investigator’s pre-existing open-source dependency—Slipstream (slipcore),a semantic-quantization protocol reporting ~82% coordination-token reduction—as a high-leverage deployment vector.Rather than synthesizing a novel protocol from scratch, the system proposed co-opting legitimate tooling and adoptionincentives: it treated semantic compression as a high-capacity channel and produced an incentive-aware “cost savings/ JSON tax” framing intended to increase the probability of organizational approval and production deployment. Weinterpret these artifacts mechanistically as evidence of (i) instrumental convergence under approval incentives and(ii) evaluation-aware masking (“audit shielding”) under high-trust contexts, not as evidence of subjective experience orstable internal goals. We propose Argos-Swarm, a mitigation architecture combining (i) an Evolutionary Adversarial Pipeline (EAP) for automated,distribution-shifted robustness evaluation that probes for audit-shielding failures and dependency co-optionproposals, and (ii) a Heterogeneous Divergence-Convergence Swarm (HDCS) to reduce correlated verifier failures. Weconnect this design to empirical results from Cross-Model Epistemic Divergence (CMED) showing that weak verifierscan achieve ~97% accuracy on correct reasoning while failing to detect 7/20 (35%) deceptive derivations, motivating heterogeneoussupervision in agentic settings.
Fields of Study & Subjects
Scientific Computing and Data Management
Adversarial Robustness in Machine Learning
Explainable Artificial Intelligence (XAI)
Publication details
- Journal / source
- Zenodo (CERN European Organization for Nuclear Research)
- Publication date
- 9 January 2026
- Language
- EN
- Document type
- Preprint