用大模型自身预测能力实现文本无损压缩,效果远超传统方法。
Lossless Compression of Large Language Model-Generated Text via Next-Token Prediction
- 利用大模型的下一步词预测能力,实现对生成文本的高效压缩。
- 在14个模型、8个数据集上压缩率超20倍,远高于Gzip的3倍。
- 方法通用性强,适用于不同规模模型和多种领域数据。
随着大语言模型(LLMs)在各领域广泛应用,其生成数据量快速增长,高效且无损压缩成为现代文本管理系统的关键需求。与传统机器生成内容不同,LLM生成文本结构复杂、多样性高,难以被传统压缩器有效处理。本文首次系统研究专用于LLM生成数据的无损压缩技术。由于LLMs通过下一步词预测训练,其输出具有高度可预测性,因此可作为自身输出的高效压缩器。在14个代表性大模型和8个跨领域数据集上的实验表明,基于大模型的预测压缩方法实现了超过20倍的压缩率,显著优于广泛使用的通用压缩工具Gzip的3倍压缩率。该优势在不同模型规模和数据类型下均保持稳定,证明了基于大模型的方法在生成式AI工作负载下的鲁棒性与实用性。
原文摘要 · Abstract (English)
As large language models (LLMs) continue to be deployed and utilized across domains, the volume of LLM-generated data is growing rapidly. This trend highlights the increasing importance of effective and lossless compression for such data in modern text management systems. However, compressing LLM-generated data presents unique challenges compared to traditional human- or machine-generated content. Traditional machine-generated data is typically derived from computational processes or device outputs, often highly structured and limited to low-level elements like labels or numerical values. This structure enables conventional lossless compressors to perform efficiently. In contrast, LLM-generated data is more complex and diverse, requiring new approaches for effective compression. In this work, we conduct the first systematic investigation of lossless compression techniques tailored specifically to LLM-generated data. Notably, because LLMs are trained via next-token prediction, we find that LLM-generated data is highly predictable for the models themselves. This predictability enables LLMs to serve as efficient compressors of their own outputs. Through extensive experiments with 14 representative LLMs and 8 LLM-generated datasets from diverse domains, we show that LLM-based prediction methods achieve remarkable compression rates, exceeding 20x, far surpassing the 3x rate achieved by Gzip, a widely used general-purpose compressor. Furthermore, this advantage holds across different LLM sizes and dataset types, demonstrating the robustness and practicality of LLM-based methods in lossless text compression under generative AI workloads.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。