Attention-Aware Multi-Level Caching for Efficient Multimodal Vision-Language Inference

Multimodal vision-language models (VLMs) have achieved remarkable capabilities but suffer from high inference latency, particularly due to repeated visual encoding operations. While caching techniques have proven effective for text-only large language models, existing approaches fail to address the unique characteristics of multimodal inference: heterogeneous token types with different computational costs, cross-modal dependencies, and the distinction between expensive visual encoding versus lightweight text generation. We present a novel multi-level caching architecture that employs attention-based importance scoring and cross-modal cache awareness to optimize multimodal inference. Our Phase 1 prototype on Apple Silicon with MLX validates output-level caching (L2), achieving 52.8% hit rate and$2.12 \times$real-world speedup through 1291ms effective latency versus 2733ms baseline. Evaluated on SmolVLM2-2.2B across 250 visual question answering queries with three cache policies (LRU, importance-based, cross-modal), all policies demonstrate statistically identical performance$(p