自动利用数据相关性，大幅压缩表格文件且几乎不增加查询开销。
Lightweight Correlation-Aware Table Compression
- 基于虚拟层自动发现并利用数据列间相关性进行压缩
- 在data-gov数据集上比Parquet减少40%存储空间
- 无需人工设置相关性，兼容现有开放格式
数据湖对关系型数据的广泛采用，要求存储格式具备高效扫描性能和良好的压缩率。现有格式虽通过轻量编码实现快速扫描，但在减少存储占用方面已接近瓶颈。近期相关性感知压缩方案可进一步缩小文件大小，但现有方法或带来显著扫描开销，或需手动指定相关性，实用性受限。本文提出Virtual框架，无缝集成于现有开放格式，自动利用数据相关性，在保持极低扫描性能损耗的同时实现显著压缩增益。在data-gov数据集上的实验表明，Virtual相比Apache Parquet最多可减少40%文件大小。
原文摘要 · Abstract (English)
The growing adoption of data lakes for managing relational data necessitates efficient, open storage formats that provide high scan performance and competitive compression ratios. While existing formats achieve fast scans through lightweight encoding techniques, they have reached a plateau in terms of minimizing storage footprint. Recently, correlation-aware compression schemes have been shown to reduce file sizes further. Yet, current approaches either incur significant scan overheads or require manual specification of correlations, limiting their practicability. We present $\texttt{Virtual}$, a framework that integrates seamlessly with existing open formats to automatically leverage data correlations, achieving substantial compression gains while having minimal scan performance overhead. Experiments on data-gov datasets show that $\texttt{Virtual}$ reduces file sizes by up to 40% compared to Apache Parquet.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。