用大模型实现高效文本压缩,速度提升54倍。
FineZip : Pushing the Limits of Large Language Models for Practical Lossless Text Compression
- 结合在线记忆与动态上下文,大幅降低大模型压缩耗时。
- 压缩10MB文本仅需4小时,较之前快54倍,压缩率提升50%。
- 为大模型实用化文本压缩迈出关键一步,适合研究者参考。
尽管语言建模目标与压缩密切相关,但现代大语言模型(LLM)并未被用于实际文本压缩系统。本文深入分析了神经网络与基于Transformer的压缩技术,对比传统方法与基于LLM的压缩方案。尽管基于LLM的方法显著优于传统方法,但其极不实用:使用Llama3-8B的LLMZip压缩10MB文本需9.5天,虽压缩比大幅提升。为此,我们提出FineZip——一种融合在线记忆与动态上下文的新颖LLM压缩系统,将相同任务压缩时间降至约4小时,相比LLMZip提速54倍,且压缩性能相当。FineZip大幅超越传统算法,压缩率提升约50%。本工作首次推动了基于大模型的无损文本压缩走向现实,但仍不足以应对大规模压缩需求。我们希望该研究为未来创新铺平道路。
原文摘要 · Abstract (English)
While the language modeling objective has been shown to be deeply connected with compression, it is surprising that modern LLMs are not employed in practical text compression systems. In this paper, we provide an in-depth analysis of neural network and transformer-based compression techniques to answer this question. We compare traditional text compression systems with neural network and LLM-based text compression methods. Although LLM-based systems significantly outperform conventional compression methods, they are highly impractical. Specifically, LLMZip, a recent text compression system using Llama3-8B requires 9.5 days to compress just 10 MB of text, although with huge improvements in compression ratios. To overcome this, we present FineZip - a novel LLM-based text compression system that combines ideas of online memorization and dynamic context to reduce the compression time immensely. FineZip can compress the above corpus in approximately 4 hours compared to 9.5 days, a 54 times improvement over LLMZip and comparable performance. FineZip outperforms traditional algorithmic compression methods with a large margin, improving compression ratios by approximately 50\%. With this work, we take the first step towards making lossless text compression with LLMs a reality. While FineZip presents a significant step in that direction, LLMs are still not a viable solution for large-scale text compression. We hope our work paves the way for future research and innovation to solve this problem.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。