用因果证据集训练模型，比注意力机制更准地筛选长文本关键信息
Learning What Matters: Supervising Global Context Pruning with Causal Evidence Sets
- 用已知证据的因果关系替代注意力做剪枝指导
- 在多跳检索任务中，因果方法稳定达到99%以上准确率
- 适用于需提前压缩上下文的场景，尤其大模型效果显著
剪枝长上下文需决定保留哪些内容块，传统方法依赖教师模型注意力生成选择器。但我们在检索任务中发现，注意力与真实因果依赖不一致：教师关注过时信息且不同训练轮次表现差异大。基于注意力训练的选择器在多跳任务中准确率从36%到98%波动。而使用已知因果证据集训练的选择器，每轮均达99%以上。即使不标注，仅通过冻结教师模型并掩码上下文即可恢复因果证据集。该现象存在于预训练冻结模型中，且在上下文需提前剪枝时，因果方法优于注意力，32B模型下仍保持优势。
原文摘要 · Abstract (English)
Pruning a long context means committing to the blocks a model will keep, and the usual selector is distilled from a dense teacher's attention. That assumes attention shows which context the answer depends on. We test the assumption on retrieval tasks where the evidence is known exactly, by masking context and measuring whether the answer changes. Attention and causal dependence disagree. Teachers attend to outdated facts that the answer does not depend on, and they attend differently across training runs that use the same evidence. Selectors trained on that attention copy both failures. On a multi-hop retrieval task, a selector distilled from attention routes at 36% to 98% depending on the training run. The same selector trained on causal evidence sets reaches 99% or better on every run. Dense accuracy does not tell the teachers apart. Masking the frozen teacher recovers the causal sets of these tasks without annotations. Frozen pretrained models show the same conflict, and selectors supervised with known evidence labels beat attention-based eviction through 32B when context must be pruned before the question arrives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。