Multistage Data-Centric Dense Retrieval

Dense retrieval is the backbone of today’s compound generative AI systems. It relies on high-dimensional vector representations of items (text passage, image, video frame, etc.) and employs a nearest-neighbor search (NNS) algorithm to retrieve the top-k most semantically similar items to a query. However, current exact and approximate NNS algorithms fall short of providing both the speed and accuracy required for retrieval-augmented generative AI systems. In this work, we discuss DReX, a dataset-agnostic, accurate, and scalableDenseRetrievalAcceleration scheme enabled through a data-centric hardware architecture that leverages in-DRAM and near-DRAM processing. We analyze key design decisions behind DReX and explore alternative Pareto-optimal configurations that trade off performance and cost across the design space.