arXiv:2605.09928cs.IRcs.DB2026-05

用图模型实现快速可扩展的通用压缩,比现有方法快多个数量级。

OpenZL: Using Graphs to Compress Smaller and Faster

  • 将压缩设计为有向无环图结构,模块化编码器可灵活组合。
  • 在真实数据集上压缩率和速度均优于主流通用压缩器。
  • 开发周期从数月缩短至数天,适合工业级应用快速迭代。

过去几十年,压缩技术虽大幅提升无损压缩率,但处理时间显著增加,难以在工业界普及。实际应用中,针对特定场景的压缩器因能利用数据结构与语义信息而表现更优,但存在适用性差、开发维护困难的问题。本文提出一种新压缩策略——图模型压缩,将压缩过程建模为模块化编码器构成的有向无环图。OpenZL 实现该框架,将数据压缩为自描述的通用二进制格式,任意配置均可由统一解码器还原。实验表明,OpenZL 在多种真实数据集上压缩率与速度均超越当前最优通用压缩器;相比以压缩率为导向的深度学习压缩方案,其压缩率相当,但速度提升多个数量级。Meta 内部部署显示,压缩尺寸与速度均有持续优化,开发周期从数月缩短至数天。OpenZL 为现代数据密集型应用提供了高效、可扩展且易维护的压缩解决方案。

原文摘要 · Abstract (English)

In the last few decades, research techniques have improved lossless compression ratios by significantly increasing processing time. However, these techniques have not gained popularity in industry because production systems require high throughput and low resource utilization. Instead, real world improvements in compression are increasingly realized by building application-specific compressors which can exploit knowledge about the structure and semantics of the data being compressed. Application-specific compressor systems outperform even the best generic compressors, but these techniques have severe drawbacks -- they are inherently limited in applicability, are hard to develop, and are difficult to maintain and deploy. In this work, we show that these challenges can be overcome with a new compression strategy. We propose the "graph model" of compression, a new theoretical framework for representing compression as a directed acyclic graph of modular codecs. OpenZL implements this framework and compresses data into a self-describing wire format, any configuration of which can be decompressed by a universal decoder. OpenZL's design enables rapid development of application-specific compressors with minimal code. Experimental results demonstrate that OpenZL achieves superior compression ratios and speeds compared to state-of-the-art general-purpose compressors on a variety of real-world datasets. Compared to ratio-focused deep-learning compressors, OpenZL is competitive on ratio while being many orders of magnitude faster. Internal deployments at Meta have also shown consistent improvements in size and/or speed, with development timelines reduced from months to days. OpenZL thus represents a significant advance in practical, scalable, and maintainable data compression for modern data-intensive applications.

数据压缩图模型高效编码工业部署

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。