动态压缩长文本，让大模型更准地处理关键信息。
ATACompressor: Adaptive Task-Aware Compression for Efficient Long-Context Processing in LLMs
- 根据任务需求选择性压缩，只留重要部分。
- 在三个问答数据集上，压缩率更高且准确率领先。
- 适合需要高效处理长文本的AI应用开发人员。
大语言模型处理长上下文时常出现‘中间信息丢失’问题，关键内容因长度过长被稀释或忽略。现有压缩方法难以兼顾信息保留与效率。本文提出自适应任务感知压缩器（ATACompressor），可根据任务需求动态调整压缩策略。其选择性编码器仅压缩任务相关的上下文片段，确保关键信息保留的同时减少冗余内容。自适应分配控制器可感知有效内容长度，并相应调节压缩率，优化资源利用。在HotpotQA、MSMARCO和SQUAD-show三个问答数据集上的实验表明，该方法在压缩效率和任务表现上均优于现有方法。此外，通过一系列消融实验与分析，深入揭示了各组件的关键作用。
原文摘要 · Abstract (English)
Long-context inputs in large language models (LLMs) often suffer from the "lost in the middle" problem, where critical information becomes diluted or ignored due to excessive length. Context compression methods aim to address this by reducing input size, but existing approaches struggle with balancing information preservation and compression efficiency. We propose Adaptive Task-Aware Compressor (ATACompressor), which dynamically adjusts compression based on the specific requirements of the task. ATACompressor employs a selective encoder that compresses only the task-relevant portions of long contexts, ensuring that essential information is preserved while reducing unnecessary content. Its adaptive allocation controller perceives the length of relevant content and adjusts the compression rate accordingly, optimizing resource utilization. We evaluate ATACompressor on three QA datasets: HotpotQA, MSMARCO, and SQUAD-showing that it outperforms existing methods in terms of both compression efficiency and task performance. Our approach provides a scalable solution for long-context processing in LLMs. Furthermore, we perform a range of ablation studies and analysis experiments to gain deeper insights into the key components of ATACompressor.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。