EBCC: an Error Bounded Climate-data Compressor

As the resolution of weather and climate simulations increases, the amount of data produced is growing rapidly from hundreds of terabytes to tens of petabytes. The huge size becomes a limiting factor for broader adoption, and its fast growth rate will soon exhaust all available storage devices. To address these issues, we present EBCC (Error Bounded Climate-data Compressor). It follows a two-layer compression approach: a base compression layer using JPEG2000 to capture the bulk of the data with a high compression ratio, and a residual compression layer using wavelet transform and SPIHT (Set Partitioning In Hierarchical Trees) encoding to efficiently eliminate long-tail extreme errors. EBCC outperforms other methods in the benchmarks at relative error targets ranging from 0.1% to 10%. In the energy budget closure and Lagrangian trajectory benchmarks, it can achieve more than 100× compression while keeping errors within the natural variability derived from ERA5 uncertainty members. We implement EBCC as a standalone C library which is seamlessly integrated with NetCDF and Zarr pipelines.