DKC-LLM: Dynamic Knowledge Caching for Large Language Models in Business Applications

Large Language Models (LLMs) often face severe latency and computational cost constraints which hinder their adoption in real-time enterprise applications. Retrieval-Augmented Generation (RAG) frameworks, while improving factual accuracy, further increase inference delays owing to additional retrieval and context integration steps. To address these challenges, we propose Dynamic Knowledge Caching for Large Language Models (DKC-LLM), a novel framework that integrates dynamic semantic caching with an adaptive cache management strategy, which detects and compensates for semantic drift to accelerate response generation while ensuring accuracy and freshness. We evaluate DKC-LLM on two distinct question-answering benchmarks: BankFAQs, a domain-specific dataset with 5,000 selected queries and HotpotQA, an open-domain dataset with 5,000 selected queries, totaling 10,000 evaluation queries. Experimental results demonstrate that DKC-LLM achieves 20-30% lower latency, i.e., 5.3–6.8 ms vs. RAG’s 8–10 ms, a 63-70.5% cache hit rate, and a 60% reduction in LLM compute overhead, while maintaining 83-90% response accuracy. Additionally, DKC-LLM reduced hallucination rates by 75%, outperforming baseline RAG by up to 15 percentage points. These findings posit that DKC-LLM is a cost-efficient, low-latency, and high-accuracy solution for real-time, high-frequency business scenarios such as customer support and enterprise information retrieval.