MARS: Memory-Aware Metadata Management for Scalable Hadoop Systems

Effective management of metadata is emerging as an increasingly important task in large-scale distributed storage systems. In particular, in the Hadoop HDFS environment, due to the structural characteristics of maintaining all file system metadata in a single NameNode memory, memory overload and performance degradation problems occur as the data scale increases. This study proposes a new metadata management framework called Memory-Aware Routing and Scaling (MARS) to address these limitations. MARS is designed to analyze the access frequency of metadata and the system memory state in real time, keep high-priority items in main memory, and distribute and store low-priority items in auxiliary storage. The proposed system adopts dynamic routing algorithms and priority-based distributed storage structures without relying on fixed cache size or static thresholds to maximize memory utilization and enable stable processing without performance degradation. As a result of the experiment, MARS reduced memory usage by about 42% compared to the existing HDFS. In addition, it was confirmed that the delay time of metadata query was improved by 35%, and the metadata loss rate in the context of memory shortage was also reduced by more than 60%. These results show that MARS can effectively enhance scalability and fault response capabilities without significantly changing the existing HDFS structure. This study presents a practical research direction for adaptive memory-based storage structures in an environment where data-driven applications are increasing.