arXiv:2605.09463cs.CL2026-05被引 2

让大模型压缩长文本时不再依赖位置,而是按语义自动选择重点内容。

Beyond Position Bias: Shifting Context Compression from Position-Driven to Semantic-Driven

  • 通过语义中心动态选择关键信息,取代固定位置插入
  • 在14个基准上显著提升推理速度与跨领域鲁棒性
  • 适合需要高效处理长文本的部署场景

大型语言模型在各类任务中表现优异,但在长上下文场景下存在计算开销高和信息冗余问题。软提示压缩虽能缓解成本,但现有方法仍受制于位置偏差:主要依赖固定位置插入可学习标记或按物理分块分组,导致性能不稳定与语义割裂。为此,我们提出语义一致性上下文压缩(SeCo),将压缩机制从位置驱动转向语义驱动。SeCo不依赖物理分块,而是通过选取查询相关标记作为语义中心,并以一致性加权方式聚合其余标记,从而天然保持语义连贯性并消除位置偏差。在两个主干模型上的14个基准测试表明,SeCo在下游任务表现、推理延迟和跨领域鲁棒性方面均具持续优势。代码已公开于https://anonymous.4open.science/r/seco-EE5E。

原文摘要 · Abstract (English)

Large Language Models (LLMs) have demonstrated exceptional performance across diverse tasks. However, their deployment in long-context scenarios faces high computational overhead and information redundancy. While soft prompt compression has emerged as a promising way to mitigate these costs by compressing sequences into compact embeddings, existing paradigms remain fundamentally constrained by position bias: they primarily rely on learnable tokens insertion at fixed positions or group tokens according to their physical token layout, thereby inducing performance instability and semantic fragmentation. To overcome this bottleneck, we propose Semantic Consistency Context Compression (SeCo), a method that shifts context compression from position-driven to semantic-driven. Rather than constraint by physical token layout, SeCo dynamically anchors compression directly in the semantic space by selecting query-relevant tokens as semantic centers and aggregating remaining tokens via consistency-weighted merging. This design inherently preserves semantic consistency while eliminating position bias. Extensive experiments on 14 benchmarks across two backbone models demonstrate that SeCo consistently shows superiority in downstream tasks, inference latency, and out-of-domain robustness. The code is available at https://anonymous.4open.science/r/seco-EE5E.

上下文压缩语义驱动大模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。