arXiv:2412.16642cs.CLcs.AI2024-12AAAI被引 11

用轻量RWKV模型实现高速无损文本压缩,比gzip节省48%比特。

L3TC: Leveraging RWKV for Learned Lossless Low-Complexity Text Compression

  • 采用RWKV模型作为核心,兼顾速度与压缩率。
  • 通过异常词元绕过机制提升压缩效率,减少冗余编码。
  • 高秩重参数化策略提升训练能力,推理复杂度不变。

基于学习的概率模型可与熵编码结合用于数据压缩,但因学习模型复杂度高,其在文本压缩中的实际应用受限。为此,本文提出一种新型学习型无损低复杂度文本压缩方法L3TC。实验表明,RWKV模型在解码速度上表现最优,且压缩比适中,最适合作为本方法的骨干网络。提出一种异常词元感知的分词器,使用有限词汇表覆盖高频词元,同时允许异常词元跳过预测与编码。提出一种新颖的高秩重参数化策略,在不增加推理复杂度的前提下增强模型训练能力。实验验证,本方法相比gzip压缩器可节省48%比特;性能接近其他学习型压缩器,但模型参数减少50倍;更重要的是,L3TC是所有学习型压缩器中最快的,支持每秒数兆字节的实时解码。代码已开源。

原文摘要 · Abstract (English)

Learning-based probabilistic models can be combined with an entropy coder for data compression. However, due to the high complexity of learning-based models, their practical application as text compressors has been largely overlooked. To address this issue, our work focuses on a low-complexity design while maintaining compression performance. We introduce a novel Learned Lossless Low-complexity Text Compression method (L3TC). Specifically, we conduct extensive experiments demonstrating that RWKV models achieve the fastest decoding speed with a moderate compression ratio, making it the most suitable backbone for our method. Second, we propose an outlier-aware tokenizer that uses a limited vocabulary to cover frequent tokens while allowing outliers to bypass the prediction and encoding. Third, we propose a novel high-rank reparameterization strategy that enhances the learning capability during training without increasing complexity during inference. Experimental results validate that our method achieves 48% bit saving compared to gzip compressor. Besides, L3TC offers compression performance comparable to other learned compressors, with a 50x reduction in model parameters. More importantly, L3TC is the fastest among all learned compressors, providing real-time decoding speeds up to megabytes per second. Our code is available at https://github.com/alipay/L3TC-leveraging-rwkv-for-learned-lossless-low-complexity-text-compression.git.

文本压缩RWKV低复杂度无损压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。