用图模型统一压缩流程,实现高效可维护的通用压缩方案。
OpenZL: A Graph-Based Model for Compression
- 将压缩过程建模为有向无环图,模块化编码器可灵活组合。
- 在真实数据集上比现有通用压缩器快20%以上,压缩率更高。
- 适合需要快速开发、部署和维护压缩系统的工程团队使用。
过去十年的研究虽大幅提升无损压缩比,但显著增加处理时间,难以在生产环境中应用。实际中,利用数据结构与语义知识的专用压缩算法更受欢迎,但其适用范围窄、开发成本高且难维护。本文提出一种新的压缩策略——图模型压缩(OpenZL),将压缩过程表示为模块化编码器构成的有向无环图。该模型生成自描述的二进制格式,任何配置均可由统一解码器还原。实验表明,OpenZL在多种真实数据集上优于当前最先进的通用压缩器,在速度和压缩率上均有提升。Meta内部部署显示,压缩体积和/或速度持续优化,开发周期从数月缩短至数天。该设计支持快速迭代、降低部署延迟,并通过成熟组件库减少安全风险。因此,OpenZL为现代数据密集型应用提供了可扩展、可维护的压缩新范式。
原文摘要 · Abstract (English)
Research techniques in the last decade have improved lossless compression ratios by significantly increasing processing time. These techniques have remained obscure because production systems require high throughput and low resource utilization. In practice, application-specific compression algorithms that leverage knowledge of the data structure and semantics are more popular. Application-specific compressor systems outperform even the best generic compressors, but these techniques have some drawbacks. Application-specific compressors are inherently limited in applicability, have high development costs, and are difficult to maintain and deploy. In this work, we show that these challenges can be overcome with a new compression strategy. We propose the "graph model" of compression, a new theoretical framework for representing compression as a directed acyclic graph of modular codecs. OpenZL compresses data into a self-describing wire format, any configuration of which can be decompressed by a universal decoder. OpenZL's design enables rapid development of tailored compressors with minimal code; its universal decoder eliminates deployment lag; and its investment in a well-vetted standard component library minimizes security risks. Experimental results demonstrate that OpenZL achieves superior compression ratios and speeds compared to state-of-the-art general-purpose compressors on a variety of real-world datasets. Internal deployments at Meta have also shown consistent improvements in size and/or speed, with development timelines reduced from months to days. OpenZL thus represents a significant advance in practical, scalable, and maintainable data compression for modern data-intensive applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。