用大模型做有损文本压缩，预测错才存修正信息，省了大量比特。
The Lossy Horizon: Error-Bounded Predictive Coding for Lossy Text Compression (Episode I)
- 用掩码语言模型预测被遮蔽内容，仅当预测错误时存储少量修正数据。
- 在相同压缩率下，比基线方法提升20%以上重建质量，且比特数更低。
- 适合追求高压缩率与高质量恢复的文本压缩场景，尤其适合大模型部署。
大型语言模型（LLMs）可作为强大的概率模型实现近似最优的无损压缩。本文探索其在有损压缩领域的应用，即以重建保真度换取更高的压缩比。提出一种名为误差有界预测编码（EPC）的有损文本编解码器，利用掩码语言模型（MLM）作为解码器。与仅存储原始子集令牌的传统方法不同，EPC允许模型预测被遮蔽的内容，并仅在模型最高预测不正确时存储最小的、基于排名的修正信息。这形成了一条残差通道，实现连续的率失真控制。通过精确的比特计数和率失真分析，将EPC与更简单的预测遮蔽（PM）基线及基于变换的向量量化加残差块（VQ+RE）方法进行对比，结果表明EPC始终优于PM，在显著更低的比特率下提供更优的重建保真度，更高效地利用了模型内在知识。
原文摘要 · Abstract (English)
Large Language Models (LLMs) can achieve near-optimal lossless compression by acting as powerful probability models. We investigate their use in the lossy domain, where reconstruction fidelity is traded for higher compression ratios. This paper introduces Error-Bounded Predictive Coding (EPC), a lossy text codec that leverages a Masked Language Model (MLM) as a decompressor. Instead of storing a subset of original tokens, EPC allows the model to predict masked content and stores minimal, rank-based corrections only when the model's top prediction is incorrect. This creates a residual channel that offers continuous rate-distortion control. We compare EPC to a simpler Predictive Masking (PM) baseline and a transform-based Vector Quantisation with a Residual Patch (VQ+RE) approach. Through an evaluation that includes precise bit accounting and rate-distortion analysis, we demonstrate that EPC consistently dominates PM, offering superior fidelity at a significantly lower bit rate by more efficiently utilising the model's intrinsic knowledge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。