通过表元数据预测湖仓数据合并收益，提升查询效率。
Smart Compaction: Predicting Compaction Utility from Lakehouse Table Metadata
- 基于17个元数据特征训练模型，精准预测文件压缩比。
- 发现单个分区文件数超4即需合并，无需复杂模型。
- 适用于数据湖场景中频繁查询但不依赖全表扫描的用户。
开放湖仓表格式随时间积累大量小文件，导致查询性能下降。现有合并策略依赖阈值，但何种元数据特征决定合并价值尚不明确。本文提出一个开源仿真框架，生成2,376个Apache Iceberg表，覆盖文件大小三个数量级；从清单文件中提取17个元数据特征（无需读取数据），并使用XGBoost预测连续的文件减少率（R²=0.998，RMSE=0.013）。二分类合并决策被证明仅通过单一分区阈值max_files_per_partition > 4即可准确区分，无需学习模型。在96个TPC-H表上进行跨模式验证，未重新训练也表现良好（R²=0.976）。查询基准测试显示，合并对元数据密集型查询有显著提升，但会因降低任务并行度而拖慢全表扫描聚合。所有代码与数据均公开。
原文摘要 · Abstract (English)
Open lakehouse table formats accumulate small data files over time, which degrades query performance. Deciding when compaction is worthwhile remains threshold-driven, but which metadata features actually determine compaction utility is not well understood. We present an open simulation framework that generates 2,376 Apache Iceberg tables spanning three orders of magnitude in file size, extracts 17 metadata features from manifest files without reading data, and trains XGBoost to predict the continuous file-reduction ratio (R2 = 0.998, RMSE= 0.013). The binary compaction decision turns out to be trivially separable by a single partition-level threshold max_files_per_partition> 4, requiring no learned model. Cross-schema validation on 96 TPC-H tables confirms generalisation without retraining (R2 = 0.976). A query benchmark reveals that compaction benefits metadata-heavy queries but can slow full-scan aggregations by reducing task parallelism. All code and data are publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。