测试时动态融合通用与神经压缩模型，提升文本无损压缩率。
Test-Time Steering for Lossless Text Compression via Weighted Product of Experts
- 测试时通过加权专家集成，自适应融合通用模型与预训练语言模型。
- 在不微调的情况下，压缩率优于单一模型，且保证不低于最优模型。
- 兼容任意自回归语言模型，适合跨数据分布的实用压缩场景。
无损压缩在数据快速增长的时代至关重要。传统通用压缩器如gzip计算开销低、速度快、适用范围广，但压缩率常低于现代神经压缩器。后者虽利用大规模训练数据更精确建模数据分布，却难以泛化到未见数据。为此，本文提出一种基于加权专家集成（wPoE）的测试时调整框架。推理时，该方法自适应融合通用压缩模型与预训练神经语言模型，确保压缩率不低于任一单独模型的最佳表现。大量实验表明，该方法无需微调即可提升文本压缩性能，并可无缝集成于任意自回归语言模型，为多种数据分布提供高效实用的压缩解决方案。
原文摘要 · Abstract (English)
Lossless compression techniques are crucial in an era of rapidly growing data. Traditional universal compressors like gzip offer low computational overhead, high speed, and broad applicability across data distributions. However, they often lead to worse compression rates than modern neural compressors, which leverage large-scale training data to model data distributions more effectively. Despite their advantages, neural compressors struggle to generalize to unseen data. To address this limitation, we propose a novel framework that performs Test-Time Steering via a Weighted Product of Experts (wPoE). At inference, our method adaptively combines a universal compression model with a pretrained neural language model, ensuring the compression rate is at least as good as that of the best individual model. Extensive experiments demonstrate that our approach improves the performance of text compression without requiring fine-tuning. Furthermore, it seamlessly integrates with any autoregressive language model, providing a practical solution for enhancing text compression across diverse data distributions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。