通过分析注意力模式实现高效上下文压缩，提升长文本问答性能。
Sentinel: Decoding Context Utilization via Attention Probing for Efficient LLM Context Compression
- 基于冻结大模型的注意力模式，轻量级探测上下文使用行为。
- 0.5B小模型实现5倍压缩，性能媲美7B模型。
- 仅用单次前向传播，支持中英多语言和跨领域泛化。
检索增强生成（RAG）常因长且噪声大的检索上下文而受限。现有压缩方法多依赖启发式相关性估计或监督压缩模型，而非大模型推理时对上下文的实际利用方式。我们提出Sentinel，一种轻量级句级压缩框架，通过分析冻结大模型头级别注意力模式，解码推理时的上下文利用行为。为使监督信号与检索依赖回答行为对齐，Sentinel 使用仅在检索上下文可用时才能正确作答的QA样本训练轻量级探测器。压缩过程仅需一次非自回归前向传播，无需专门压缩训练或自回归评分。实验发现，即使在小型代理模型中，有效上下文利用信号仍可被获取。在LongBench上，采用0.5B代理模型的Sentinel实现最高5×压缩，同时问答性能与基于7B模型构建的压缩方法相当。尽管仅在英文QA数据上训练，Sentinel在中文及跨领域设置下也表现出良好泛化能力。
原文摘要 · Abstract (English)
Retrieval-augmented generation (RAG) often suffers from long and noisy retrieved contexts. Existing context compression methods typically rely on heuristic relevance estimation or supervised compression models rather than on how LLMs utilize retrieved context during inference. We propose Sentinel, a lightweight sentence-level compression framework that decodes inference-time contextual utilization behaviors from head-wise attention patterns of frozen LLMs. To ground supervision in retrieval-dependent answering behavior, Sentinel trains a lightweight probe using QA examples where the model succeeds only when retrieved context is available. Sentinel performs compression using only a single non-autoregressive forward pass without dedicated compression training or autoregressive scoring. Empirically, we find that effective contextual utilization signals remain accessible even in compact proxy models. On LongBench, Sentinel with a 0.5B proxy model achieves up to 5$\times$ compression while attaining question-answering performance competitive with compression methods built on 7B-scale models. Despite being trained only on English QA data, Sentinel also generalizes effectively to Chinese and out-of-domain settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。