Intelligent Hybrid Caching for Sustainable Big Data Processing: Leveraging NVM to Enable Green Digital Transformation

Apache Spark has gained widespread adoption for large-scale data processing. However, conventional caching methods inadequately address the dual challenges of performance bottlenecks and escalating energy consumption in data-intensive workloads. This paper introduces a sustainable computing framework that integrates Directed Acyclic Graph (DAG) dependency analysis with garbage collection (GC) behavior monitoring to optimize data placement between DRAM and non-volatile memory (NVM). The proposed Intelligent Hybrid Caching Management Framework (IHCMF) dynamically predicts data access patterns and migrates cache blocks based on cost–benefit analysis, achieving a 37.5% execution time reduction over default Spark configurations in SparkBench evaluations. By improving throughput-per-watt and projecting potential benefits from NVM’s near-zero idle power and extended hardware lifespan, IHCMF provides a scalable, cost-effective caching solution for resource-constrained edge computing environments. This work demonstrates that high-performance computing can be reconciled with environmental sustainability through intelligent memory management.