Big data energy systems: A survey of practices and associated challenges

Energy systems generate vast amounts of data in extremely short time intervals, creating challenges for efficient data management. Traditional data management methods often struggle with scalability and accessibility, limiting their usefulness. More advanced solutions, such as NoSQL databases and cloud-based platforms, have been adopted to address these issues. Still, even these advanced solutions can encounter bottlenecks, which can impact the efficiency of data storage, retrieval, and analysis. This review explores the research trends in big data management for energy systems, highlighting the practices, opportunities and challenges. Additionally, the regulatory data requirements are highlighted using the chosen reference architectures. The review, in particular, explores the limitations of current big data storage and integration solutions and examines how new technologies are applied to the energy sector. Novel insights into emerging technologies - including data spaces, peer-to-peer overlays, and blockchains - are provided, along with practical recommendations to improve data sharing and regulatory compliance.