解决语言模型对列表输入顺序敏感的问题，提升真实场景下的可靠性。
RoToR: Towards More Reliable Responses for Order-Invariant Inputs
- 通过最小修改位置编码实现零样本顺序不变性
- 在LitM、KGQA等任务上零样本性能显著优于基线
- 自适应框架可同时处理顺序无关与敏感的输入
缓解语言模型对列表输入的位置偏差是重要挑战（如‘丢失中间’问题）。尽管已有零样本顺序不变模型，但其在实际列表任务中表现有限。本文首次识别并克服两大限制：(1) 训练与推理时位置标识分配不一致导致的分布偏移；(2) 难以适应实际任务中顺序不变与敏感输入混合的情况。为此提出：(1) RoToR，一种对真正顺序不变输入仅做最小位置编码修改的零样本模型；(2) Selective Routing，可自适应处理两类输入的框架。在Lost in the middle (LitM)、知识图谱问答 (KGQA) 与 MMLU 基准上，实验表明该方法可在零样本条件下有效应对实际列表任务（代码开源：https://github.com/soyoung97/RoToR）。
原文摘要 · Abstract (English)
Mitigating positional bias of language models (LMs) for listwise inputs is a well-known and important problem (e.g., lost-in-the-middle). While zero-shot order-invariant LMs have been proposed to solve this issue, their success on practical listwise problems has been limited. In this work, as a first contribution, we identify and overcome two limitations to make zero-shot invariant LMs more practical: (1) training and inference distribution mismatch arising from modifying positional ID assignments to enforce invariance, and (2) failure to adapt to mixture of order-invariant and sensitive inputs in practical listwise problems. Then, to overcome these issues we propose (1) RoToR, a zero-shot invariant LM for genuinely order-invariant inputs with minimal modifications of positional IDs, and (2) Selective Routing, an adaptive framework that handles both order-invariant and order-sensitive inputs in listwise tasks. On the Lost in the middle (LitM), Knowledge Graph QA (KGQA), and MMLU benchmarks, we show that RoToR with Selective Routing can effectively handle practical listwise input tasks in a zero-shot manner (https://github.com/soyoung97/RoToR)
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。