Taming Multidimensional Analytics Costs: An Empirical Study of HLL and KLL Sketches in Cloud Data Warehouses

View at publisher
Authors: Gurmeet Singh, Kushal Thakkar
Cited by: 0
References: 7

Abstract

Analytics dashboards that track unique users and percentiles across multiple dimensions create a costly problem for data teams: each query requires scanning the entire dataset, leading to spiraling costs and slow performance. We present an empirical study of a novel architectural pattern that stores compact probabilistic summaries (HyperLogLog (HLL) and Karnin-Lang-Liberty (KLL) sketches) once, then merges them to answer any dimensional query without rescanning data. Testing six configurations on Stack Overflow data (12.46 GB), we achieve 93-96% reduction in data scanning with less than 1% error. Our data-driven guidelines recommend HLL precision 20 and KLL precision 10,000 for production deployments. This approach transforms expensive multi-dimensional analytics into cost-effective operations suitable for real-time dashboards across cloud data warehouses including BigQuery, Snowflake, Redshift, and Presto.

Fields of Study & Subjects

Advanced Database Systems and Queries Big Data and Business Intelligence Cloud Computing and Resource Management

Publication details

Journal / source
IEEE Software
Volume
43
Issue
3
Pages
53–61
Publication date
3 February 2026