用去噪Transformer和熵编码实现高效文本压缩,体积减半且语义几乎无损。
TextEconomizer: Enhancing Lossy Text Compression with Denoising Transformers and Entropy Coding

- 采用编码器-解码器结构,结合去噪Transformer与熵编码提升压缩效率。
- 压缩比达5.39倍,参数量仅同类模型的1/153,保持高语义保真度。
- 适合资源受限场景,如移动设备、数字档案存储等低内存环境使用。
有损文本压缩在保留核心语义的同时减小数据规模,适用于摘要生成、自动分析和数字存档。尽管基于Transformer的语言模型已占主导地位,但将上下文向量与熵编码融入序列到序列生成仍研究不足。关键挑战在于从编码器输出中识别最具信息量的上下文向量,并通过熵编码提升存储效率,同时在噪声文本下维持高质量输出。我们提出TextEconomizer,一种配对变压器神经网络的编码器-解码器框架,可将可变长度输入减少50%至80%,无需预先了解数据集维度。该模型通过熵编码实现具有竞争力的压缩比,且在BLEU、ROUGE、METEOR及语义相似度评分上表现接近完美。TextEconomizer参数量约为同类模型的1/153,实现5.39倍压缩比而不牺牲语义质量。我们还评估了基于LSTM的自编码器,达到67倍压缩比,参数减少196倍;以及改进的LLaMAFormer,参数量为ICAE的1/263,仍保持良好文本质量。TextEconomizer显著优于现有基于Transformer的模型,在内存效率与高保真输出间取得平衡,标志着有损压缩在最优空间利用上的突破。
原文摘要 · Abstract (English)
Lossy text compression reduces data size while preserving core meaning, making it well-suited for summarization, automated analysis, and digital archives. Despite the dominance of transformer-based models in language modeling, integrating context vectors and entropy coding into Sequence-to-Sequence (Seq2Seq) generation remains underexplored. A key challenge lies in identifying the most informative context vectors from encoder output and incorporating entropy coding to enhance storage efficiency while maintaining high-quality outputs, even under noisy text. We introduce TextEconomizer, an encoder-decoder framework paired with a transformer neural network that reduces variable-sized inputs by 50% to 80% without prior knowledge of dataset dimensions. Our model achieves competitive compression ratios via entropy coding while delivering near-perfect text quality, assessed by BLEU, ROUGE, METEOR, and semantic similarity scores. TextEconomizer operates with approximately 153x fewer parameters than comparable models, achieving a 5.39x compression ratio without sacrificing semantic quality. We also evaluate an LSTM-based autoencoder achieving a state-of-the-art 67x compression ratio with 196x fewer parameters, and LLaMAFormer, a modified transformer with 263x fewer parameters than ICAE while maintaining competitive text quality. TextEconomizer significantly surpasses existing transformer-based models in balancing memory efficiency and high-fidelity outputs, marking a breakthrough in lossy compression with optimal space utilization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。