利用模型位置偏见,用黑盒方法高效定位长文本中的关键信息。
GOLD PANNING: Strategic Context Shuffling for Needle-in-Haystack Reasoning
- 通过重排文档将重要信息置于诊断性强的位置,实现信号锚定。
- 在长文本问答中,比传统方法少30%~65%的查询次数即可准确定位目标。
- 无需白盒访问,适合无法获取内部机制的大模型,尤其适用于实际部署场景。
大型语言模型在长上下文的‘针中找针’问题中表现出显著的位置偏差,倾向于优先关注信息位置而非其相关性。现有缓解方法依赖白盒访问,但对多数先进模型不可行。本文提出GOLD PANNING,一种黑盒贝叶斯框架,在推理时通过主动搜索长上下文:(i) 重新排列文档,将高置信度内容集中于诊断性强的位置(信号锚定);(ii) 从模型输出更新对文档相关性的信念。不同于传统主动学习聚焦不确定性降低,GOLD PANNING采用‘一旦标记,持续保留’的锚定策略,以维持弱线索。通过基于模型诊断性分布的迭代分配,可证明在$O(\log N)$轮内从$N$个文档中识别目标,具备多文档场景下的可扩展性。在针中找针检索与长上下文问答任务中,GOLD PANNING以30%–65%更少的查询次数达到与排列自洽性相当的定位效果,且在校准不匹配下仍有效,表明粗粒度位置排序即可带来性能提升。结果表明,模型的固有偏见不必是缺陷,反而可作为控制工具加以利用。
原文摘要 · Abstract (English)
Large language models (LLMs) exhibit pronounced position bias in long-context needle-in-haystack problems, systematically prioritizing the location of information over its relevance. While current mitigations rely on white-box access, this is effectively impossible for many state-of-the-art models. We introduce GOLD PANNING, a black-box Bayesian framework that performs inference-time active search over long contexts by (i) reordering documents to concentrate high-belief items in highly diagnostic positions (signal anchoring) and (ii) updating beliefs over document relevance from model outputs. Unlike conventional active learning, which prioritizes uncertainty reduction, GOLD PANNING leverages anchoring -- once flagged, keep it in sight -- to preserve weak cues. We implement this using iterative assignment derived from the model's diagnosticity profile, which provably identifies a target among $N$ documents in $O(\log N)$ rounds, ensuring scalability to many-document settings.On needle-in-a-haystack retrieval and long-context QA, GOLD PANNING matches Permutation Self-Consistency's target identification with $30--65%$ fewer queries and remains effective under calibration mismatch, suggesting coarse positional ordering drives performance gains. These results demonstrate that inherent model biases need not be failures, but can be used as tools for control.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。