Routing Memory A Tiered, Vector-Quantized Memory System for Long-Lived Language-Model Agents

Every memory layer for a language-model agent rests on the same operation: store denseembeddings, then search all of them at query time. As an agent accumulates memory acrosssessions, that matrix becomes the single most expensive thing it holds in RAM. RoutingMemory (RM) attacks exactly this cost. It reframes vector quantization — usually a tool forcompression — as a routing primitive, and packages an IVF-family retrieval core into atwo-tier memory: the routing state (centroids and an item-id index) stays resident in RAM,while the full embeddings live in cold storage and are read only for the buckets a queryactually probes. The contribution is deliberately not the retrieval mechanism, which is openly that of aninverted file. It is the adaptation of that mechanism to the regime where RAM, not disk, isthe binding cost. RM is a systems result, and it is evaluated as one: along the cost itremoves at a held level of retrieval quality, rather than as a recall ranking against otherretrievers. It needs no vector database and no FAISS dependency — only NumPy, scikit-learn,and sentence-transformers. On synthetic embeddings, tiering cuts resident memory by 14.4x relative to dense flat searchwhile preserving retrieval accuracy exactly: the RAM-resident and tiered variants returnidentical recall, because moving an embedding to cold storage changes where it lives, notwhat is retrieved. On real all-MiniLM-L6-v2 embeddings of MS-MARCO swept from 100K to onemillion passages, the advantage not only holds but widens to 15-19.5x, with routing recallstaying within a few points of the exact-search ceiling at every scale. The honest cost is latency: under a single-level codebook the candidate set grows with thecorpus. Two measured levers contain it. A memory-mapped cold tier cuts the dominant readcost by 16.7x at 300K and 18.7x at one million items, with no new dependency — the advantagedoes not shrink with scale. And a two-level (hierarchical) codebook bounds the reranked setas the corpus grows, for an 11.2x latency speedup at one million items. The remaining recallgap is characterized rather than hidden: it is invariant to coarse probing and localized tothe fine layer, which points to where the next improvement belongs. Version 4 makes the real-encoder results the primary evidence, adds the hierarchical routingand the memory-mapped cold-tier backend with their measurements (including at one-millionscale), characterizes the hierarchical recall gap, verifies all reference DOIs, and clarifiesthe systems-paper scope.