Toward Cost-Efficient LLM Serving: A System-Level Memory Optimization Approach

Serving large-scale language models (LLMs) requires significant system resources, where GPU memory limits, system bottlenecks, and I/O delays collectively introduce substantial latency. We present MBALL (Memory BALLooner), a lightweight framework that jointly optimizes resource allocation across VRAM, DRAM, and NVMe storage to enable cost-efficient and sustainable inference. MBALL integrates task-aware scheduling and cooperative memory management to mitigate contention in real deployments. Experiments with state-of-the-art LLM workloads show that MBALL improves throughput by up to 60%, reduces latency by 35%, and lowers VRAM usage by 15%; results represent means over n=5 runs with 95% confidence intervals. MBALL provides a flexible design applicable across diverse industrial domains, including smart logistics, real-time customer services, and AI-driven IoT platforms, and offers empirical insights that can inform future system-level LLM optimization research. This paper describes the MBALL architecture and its core components, evaluates its performance against strong baselines, and demonstrates how it enables scalable and practical AI services in real deployments.