压缩长文本上下文，让模型高效推理代码和多文档。
LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning
- 用轻量交叉注意力层+分块策略压缩长输入
- 在10万+词长任务上准确率超基线，跨模型迁移成功
- 适合需要处理超长文本的代码、文档分析场景
随着真实应用对10万+词输入的需求增加，上下文长度与推理效率之间的差距已成为关键瓶颈。上下文压缩可在保持任务准确性的同时降低预填充开销。然而，现有无训练的基于注意力的方法在代码推理等高难度长上下文任务中仍存在显著不足。本文提出LongAttnComp，是AttnComp在长上下文场景下的改进版本，通过微调轻量级交叉注意力评分层，并引入词粒度分块、基于词预算的top-p算法、位置重排及格式无关查询解析器。我们设计了两阶段微调方案：第一阶段使用NIAH风格数据构建通用检索基础，第二阶段引入多跳推理和复杂任务数据以拓展长上下文覆盖范围。在InfiniteBench Code-Debug任务上，LongAttnComp达到或超过全上下文准确率，显著优于无训练基线，并成功迁移到三个模型家族中的四个目标模型。在LongBench v2上，两阶段方案大幅缩小了第一阶段的性能差距，同时保持Code-Debug表现。
原文摘要 · Abstract (English)
As real-world applications increasingly require processing inputs of 100k+ tokens, the gap between context length and inference efficiency has become a critical bottleneck. Context compression offers a way to reduce prefill costs while preserving task accuracy. However, existing training-free attention-based methods leave substantial gaps in demanding long-context tasks such as code reasoning. We present LongAttnComp, a long-context adaptation of AttnComp that fine-tunes a lightweight cross-attention scoring layer and introduces tokenlevel chunking, a token-budget top-p algorithm, positional reordering, and a formatagnostic query parser. We further design a two-stage fine-tuning recipe for the compressor: Stage 1 builds a general retrieval foundation from NIAH-style data, and Stage 2 extends it with multi-hop and reasoning data for broader long-context task coverage. On InfiniteBench Code-Debug, LongAttnComp matches or exceeds full-context accuracy, substantially outperforms training-free baselines, and transfers across four target models from three families. On LongBench v2, the two-stage recipe largely closes the Stage 1 gap on multi-document reasoning while preserving Code-Debug performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。