Active Data Lakes: Regaining Physical Data Independence Without Losing Interoperability
Abstract
Data lakes aim to avoid vendor lock-in and enable interoperability between different query engines on a single copy of data. While early data lakes were only collections of files in various formats, they have since evolved to incorporate some features traditionally associated with relational databases. Today, Apache Parquet is the de facto standard file format for relational data in data lakes. This standardization is fundamental to interoperability, but it comes at the cost of physical data independence because query engines integrate tightly with Parquet. As a result, adoption of novel approaches in the areas of file formats, access paths, and storage media has been limited. We propose the Active Data Lake architecture as a way to restore physical data independence and demonstrate its potential experimentally through three example optimizations.
Fields of Study & Subjects
Data Quality and Management
Research Data Management Practices
Advanced Database Systems and Queries
Publication details
- Journal / source
- Proceedings of the VLDB Endowment
- Publisher
- Association for Computing Machinery (ACM)
- Volume
- 19
- Issue
- 6
- Pages
- 1372–1385
- Publication date
- 1 February 2026
- Language
- EN