Profiling Minimal Data Dependency Combinations

Data profiling describes the activity of inferring structural metadata, such as functional dependencies, inclusion dependencies, and unique column combinations, from (relational) datasets. Because structural metadata is often not stored explicitly, data profiling plays a crucial role in various data management tasks, including data discovery, cleaning, integration, normalization, and querying. Due to the importance of structural metadata and, in particular, data dependencies, researchers have been actively exploring new types of metadata and efficient algorithms for their automatic discovery. In the past, however, each type of metadata has been considered mostly in isolation. This poses a serious challenge to many use cases that actually require specific combinations of data dependencies because deriving these combinations from individually profiled metadata is as difficult as the initial metadata discovery. In this article, we investigate the interaction of data dependencies in (complex) combinations and define minimality and completeness as two essential properties that enable the automatic profiling of data dependency combinations. A notion of minimal dependency combinations and complete dependency combination result sets is a prerequisite for the (automatic) discovery of dependency combinations, because these properties enable search space pruning and effectively restrict the profiling to manageable and meaningful result sizes. Due to the enormous search space of dependency combinations, we also propose a minimality constraint formalism as a novel search space pruning technique. This technique expresses the minimality of any data dependency combination in terms of already well-known, type-specific minimality constraints. Furthermore, we apply a practical, graph-based constraint inference algorithm to automatically derive query-specific minimality constraints for any given metadata query. In an experimental evaluation, we assess the effectiveness of the derived minimality constraints and provide a first impression of the possibilities and challenges that arise when profiling (complex) metadata patterns. Our study covers both theoretical and practical aspects for the profiling of data dependency combinations and is a necessary step towards the development of a holistic data profiling system that efficiently answers metadata pattern queries.