Hybrid Data Lake localization optimization based on complex networks

As enterprise organizations increasingly adopt hybrid and multi-cloud architectures to manage the massive growth of Big Data, the Data Lake (DL) has evolved from a simple raw data repository into a complex, logically centralized structure. However, this distributed evolution introduces significant governance challenges, particularly regarding the financial impact of data gravity and the varying costs of storage, computation, and data egress across different providers. This paper proposes a Entity Migration Recommender System that leverages complex network theory to optimize data localization and minimize operational expenses in hybrid Data Lake environments. By modeling data entities as vertices and their interdependencies as edges within a directed graph, the research quantifies costs by integrating storage rates, compute time, and egress fees. A quantitative experimental simulation was conducted using Brazilian healthcare data (DataSus) and Apache Spark metadata to validate the model's efficacy. The experimental results demonstrate that the iterative application of the migration algorithm achieved a 16.2% reduction in total infrastructure costs compared to the initial configuration.The findings indicate that while storage and compute fees are primary drivers for migration, the "interaction factor"—the recursive impact of locality changes on egress costs—impacts in data gravity and requires an iterative algorithmic approach to reach an optimal steady state.