纠正位置偏差无法替代多轮重排，单次排序效果仍有差距
Position Bias Correction is Insufficient for One-Pass Attention Sorting
- 用注意力分布估计偏差曲线，单次修正原始注意力得分
- 单次排序准确率仅达迭代重排的63%，差距未完全弥补
- 适合关注长文本生成效率与精度平衡的研究者
长上下文语言模型存在位置偏差问题，中间位置信息利用率低。注意力重排通过多轮迭代根据注意力模式重新排序文档来缓解此问题，但显著增加部署开销。本文假设位置偏差是主要瓶颈，提出去偏单次注意力重排方法：从低注意力多数文档中估计每提示的位置偏差曲线，并通过减法或除法修正原始注意力得分，实现单次排序。在两个模型上的实验表明，在 LLaMA-2-7B-32K-Instruct 上，去偏后单次排序准确率与未校准版本相同（94.83% 包含准确率）；在 YaRN-Llama-2-7b-64k 上，去偏提升 8.67 个百分点，但仍落后于迭代重排 14.84 个百分点，仅弥补了 37% 的差距。结果说明，仅靠位置偏差校正不足以达到迭代重排效果，重复重排带来的收益超出偏差修正本身。
原文摘要 · Abstract (English)
Long-context language models suffer from position bias, where information in middle positions is underutilized. Attention Sorting addresses this by iteratively reordering documents based on attention patterns, but its multiple sort-and-generate cycles increase deployment cost. We hypothesize that position bias is the primary bottleneck and propose Debiased One-Pass Attention Sorting, which estimates a per-prompt position-bias curve from the low-attention majority of documents and uses it to correct raw attention scores (via subtraction or division) to enable single-pass sorting. Our experiments on two models refute this hypothesis in the tested setting: on LLaMA-2-7B-32K-Instruct, debiasing produces identical results to uncalibrated single-pass sorting (94.83\% containment accuracy), while on YaRN-Llama-2-7b-64k, debiasing improves accuracy by 8.67 percentage points but remains 14.84pp behind iterative sorting, closing only 37\% of the gap. These results suggest that position-bias correction is insufficient to match iterative sorting, and that repeated reordering provides additional benefits beyond bias correction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。