硬压缩导致指代断裂,丢失关键信息使答案失效
Relevant but Incomplete: Referential Dangling as a Paradigm-Level Failure Mode in Hard Prompt Compression

- 独立选词导致依赖文本被拆分,关键信息残缺
- 压缩比30%时,3个数据集上34%-54%的题目出现指代断裂
- 无需标注即可自动补全缺失段落,提升准确率4.7点
硬提示压缩通过独立评分并保留最高得分单元来降低长上下文推理成本。我们发现该方法存在结构性缺陷:独立选择可能将相关证据对拆开,保留一个而删除另一个。当保留内容包含答案但删除内容定义了理解答案所需实体时,即产生指代断裂。在压缩比为0.30时,Beaver(使用Qwen3-0.6B嵌入)在三个多跳问答数据集的桥接例题中,有34%-54%出现答案路径不完整。在共享HotpotQA桥接集上,所有六种硬压缩器均出现指代断裂,最高达60%;LongBench-v2单文档问答中每篇文档至少存在一处指代断裂。在Qwen3-8B上评估,重新插入缺失支持段落(同时移除非支持段落以维持预算),准确率提升29-34个百分点(p < 0.0001),恢复至少88%与保留双支持段落的差距。更强的答主模型也无法弥补损失:在MuSiQue上,GPT-5.5在压缩上下文下的准确率比保留双段落时低8.8分。最后,我们训练了一个小型分类器,在推理时根据是否需解释保留文本排序被删句子,并自动补回前几名,无需支持标注。在HotpotQA上使用Qwen3-8B,此方法将准确率提升4.7分,压缩比仅从0.30升至0.31。硬压缩应同时优化相关性与指代完整性。
原文摘要 · Abstract (English)
Hard prompt compression reduces long-context inference cost by independently scoring tokens, sentences, or chunks and retaining the highest-scoring units under a budget. We identify a structural failure in this procedure: independent selection can split dependent evidence pairs, retaining one member while deleting the other. When retained text contains an answer but deleted text defines the entity needed to interpret it, we call the result referential dangling. At a compression ratio of 0.30, Beaver, which ranks coherent chunks using Qwen3-0.6B embeddings, leaves the answer path incomplete in 34-54% of bridge examples across three multi-hop question answering datasets. On a shared HotpotQA bridge set, all six hard compressors we test exhibit dangling at rates up to 60%, and every document in LongBench-v2 Single-Document QA contains at least one dangling reference. On dangling examples evaluated with Qwen3-8B, reinserting the missing supporting paragraph while removing nonsupporting paragraphs to maintain the token budget improves accuracy by 29-34 percentage points (p < 0.0001), recovering at least 88% of the gap to contexts retaining both supporting paragraphs. Stronger answer models do not absorb the loss: on MuSiQue, GPT-5.5 is 8.8 points less accurate on compressed contexts than on contexts retaining both supporting paragraphs. Finally, we train a compact classifier to rank omitted sentences by whether they are needed to interpret retained text and reinsert the top-ranked candidates without support annotations at inference. On HotpotQA with Qwen3-8B, this automatic restoration improves accuracy by 4.7 points while changing the compression ratio only from 0.30 to 0.31. Hard compressors should optimize both relevance and referential completeness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。