用大模型重建删掉部分的文本,实现高效语义压缩。
Text-Preserving Lossy Text Compression: A Study of Strategic Deletion and LLM Reconstruction
- 通过删除策略保留关键文本骨架,由LLM重构原文。
- 在保留率10%~90%下,频率删除法效果稳定且速度快。
- 适合作为低资源场景下的文本压缩方案,跨语言有效。
传统无损文本压缩虽完整保存每字节,但在自然语言场景中压缩收益有限。本文研究有损语义文本压缩:编码器有策略地删减文本内容,再由大语言模型(LLM)从剩余片段重建原内容。我们在BBC新闻数据集上评估了多种删除策略,包括均匀步长删除、基于词长的删除(WordLen)、基于词频的删除(WordFreq)、LP优化删除(Opt)、基于GPT-2意外度的熵删除,以及频率与意外度结合的混合方法。在保留率 $\r_{keep} \in [0.1,0.9]$ 下,主要发现有三:第一,WordFreq作为低成本基线表现优异,仅依赖静态词频查表,却能媲美更复杂的语义方法,且编码速度远快;第二,语义与混合方法在中等压缩率下优势明显,而词频删除在最低保留率下更具鲁棒性;第三,QLoRA微调得到的本地解码器性能强劲,可媲美Gemini 2.0 Flash,尤其在解码器单体比较中表现突出。额外的英、中文实验表明该框架具备跨领域迁移能力,但最佳删除规则仍依赖数据集特性。
原文摘要 · Abstract (English)
Traditional lossless text compression preserves every byte, but its gains on natural language are often modest in realistic operating regimes. We study \emph{lossy semantic text compression}, where the encoder strategically deletes parts of the text and a large language model (LLM) reconstructs the original content from the retained skeleton. We benchmark a progression of deletion strategies, including uniform step deletion, word-length-guided deletion (WordLen), word-frequency-guided deletion (WordFreq), LP-optimized deletion (Opt), entropy-based deletion using GPT-2 surprisal, and hybrid methods that combine frequency and surprisal signals. Evaluation on the BBC News dataset across retention rates $\r_{keep} \in [0.1,0.9]$ shows three main findings. First, WordFreq is a strong low-cost baseline: despite using only a static frequency lookup, it remains competitive with much more expensive semantic methods while being far faster at the encoder. Second, semantic and hybrid methods provide their clearest gains at mild-to-moderate compression, whereas word-frequency deletion is often more robust at the lowest retention rates. Third, QLoRA fine-tuning yields a strong local decoder that is competitive with Gemini 2.0 Flash and is often strongest in decoder-only comparisons. Additional English and Chinese experiments show that the overall framework transfers across domains, while the best deletion rule remains dataset-dependent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。