无需训练的双阶段压缩框架，让大模型用更少tokens完成长文本推理。
DSPC: Dual-Stage Progressive Compression Framework for Efficient Long-Context Reasoning
- 分两阶段压缩：先按语义筛选句子，再按注意力等指标删减无用词元。
- 仅用3倍少的token，Longbench少样本任务准确率达49.17，超现有最佳方法7.76。
- 适合需要高效处理长上下文、无额外训练资源的部署场景。
大语言模型在自然语言处理任务中表现卓越，但为提升输出准确性，提示词长度持续增加，导致计算成本上升。为缓解提示膨胀问题，提示压缩被提出，但多数方法需训练小型辅助模型，带来额外计算开销。为此，本文提出无需训练的双阶段渐进式压缩框架（DSPC）。第一阶段采用基于TF-IDF的语义相关性过滤，剔除低价值句子；第二阶段结合注意力贡献、跨模型损失差异和位置重要性评估词元重要性，精准剪枝低效词元同时保留语义。我们在LLaMA-3.1-8B-Instruct与GPT-3.5-Turbo上验证该方法，在受限词元预算下实现稳定提升。例如，在Longbench数据集的少样本任务中，仅使用3倍少的词元，即达49.17分，优于最优基线LongLLMLingua 7.76分。
原文摘要 · Abstract (English)
Large language models (LLMs) have achieved remarkable success in many natural language processing (NLP) tasks. To achieve more accurate output, the prompts used to drive LLMs have become increasingly longer, which incurs higher computational costs. To address this prompt inflation problem, prompt compression has been proposed. However, most existing methods require training a small auxiliary model for compression, incurring a significant amount of additional computation. To avoid this, we propose a two-stage, training-free approach, called Dual-Stage Progressive Compression (DSPC). In the coarse-grained stage, semantic-related sentence filtering removes sentences with low semantic value based on TF-IDF. In the fine-grained stage, token importance is assessed using attention contribution, cross-model loss difference, and positional importance, enabling the pruning of low-utility tokens while preserving semantics. We validate DSPC on LLaMA-3.1-8B-Instruct and GPT-3.5-Turbo under a constrained token budget and observe consistent improvements. For instance, in the FewShot task of the Longbench dataset, DSPC achieves a performance of 49.17 by using only 3x fewer tokens, outperforming the best state-of-the-art baseline LongLLMLingua by 7.76.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。