用压缩理论抑制大模型重复生成，实现零温度采样无退化。
LZ Penalty: An information-theoretic repetition penalty for autoregressive language models
- 基于LZ77压缩算法的码长设计惩罚项，抑制冗余生成。
- 在零温度贪婪解码下，重复率降至0%，优于现有方法。
- 适合追求高效推理且避免重复输出的开源模型用户。
我们提出LZ惩罚，一种专用于减少自回归语言模型退化重复的惩罚机制，不损失模型能力。该惩罚基于LZ77通用无损压缩算法中的码长。从预测-压缩对偶性视角看，使用该惩罚进行解码等价于从去除高度可压缩信息后的残差分布中采样。实验表明，LZ惩罚使开源推理模型在零温度贪婪解码下仍能保持高性能且无退化重复。而行业标准的频率惩罚和重复惩罚效果不佳，重复率高达4%。
原文摘要 · Abstract (English)
We introduce the LZ penalty, a penalty specialized for reducing degenerate repetitions in autoregressive language models without loss of capability. The penalty is based on the codelengths in the LZ77 universal lossless compression algorithm. Through the lens of the prediction-compression duality, decoding the LZ penalty has the interpretation of sampling from the residual distribution after removing the information that is highly compressible. We demonstrate the LZ penalty enables state-of-the-art open-source reasoning models to operate with greedy (temperature zero) decoding without loss of capability and without instances of degenerate repetition. Both the industry-standard frequency penalty and repetition penalty are ineffective, incurring degenerate repetition rates of up to 4%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。