RAMS: Runtime Adaptive Memory Scaling for Tiny Deep Learning on IoT Devices

Deploying Tiny Deep Learning (TinyDL) on Internet of Things (IoT) devices is gaining popularity. To accommodate the limited memory, recent methods split tensors into fine-grained parts and plan memory offline to minimize its footprint. However, they fail to adapt to dynamic memory, missing the opportunity to utilize temporarily available memory for faster inference. Additionally, existing approaches focus solely on minimizing memory size while neglecting cache usage characteristics, resulting in frequent cache misses and increased latency. In this paper, we propose RAMS, an efficient framework supporting runtime adaptive memory scaling to fully utilize the dynamic memory. We also propose a cache-friendly memory management approach that minimizes cache miss times. RAMS includes an offline planner to minimize the memory footprint essential for inference and an online manager to determine memory sizes and generate layouts for size-controllable tensors based on available memory. RAMS significantly reduces inference latency while maintaining a compact memory footprint. Extensive experiments on commercial devices running RTOS and Android systems demonstrate that, compared to the state-of-the-art methods, RAMS can efficiently reduce latency by up to 1.57× and 1.48× compared to TFLM and TinyTS, respectively using a comparable memory footprint, while reducing power consumption by 67.74% and 15.97%.