用强化学习提升大模型对输入顺序的鲁棒性,同时保持高准确率。
Towards Order Fairness: Mitigating LLMs Order Sensitivity through Dual Group Advantage Optimization

- 引入双组优势优化框架,平衡输出准确性和顺序稳定性。
- 在RAG、数学推理等任务上实现更高顺序公平性,性能优于基线。
- 提出新评估指标,揭示旧方法虚假稳定性,适合关注可靠性研究者。
大型语言模型存在顺序偏差,其表现受输入元素排列顺序影响,限制了其在上下文学习和检索增强生成(RAG)等场景的应用。现有方法通过统计或搜索获取最优排序,增加推理开销且未解决模型内在偏差;或通过多顺序微调缓解,但常以牺牲准确性为代价,导致模型陷入一致但错误的幻觉。本文提出双组优势优化(DGAO),首次将强化学习用于缓解大模型的顺序敏感性。DGAO同时计算并平衡组内相对准确率优势与组间相对稳定性优势,奖励生成稳定且正确输出的策略,惩罚对顺序敏感或错误的响应。我们还提出一致性率和过度自信率两个新指标,揭示以往方法的伪稳定性,并指导更全面评估。大量实验表明,DGAO在保持甚至提升性能的同时,显著改善了RAG、数学推理和分类任务中的顺序公平性。代码已开源:https://github.com/Hyalinesky/DGAO。
原文摘要 · Abstract (English)
Large Language Models (LLMs) suffer from order bias, where their performance is affected by the arrangement order of input elements. This unfairness limits the model's applications in scenarios such as in-context learning and Retrieval-Augmented Generation (RAG). Recent studies attempt to obtain optimal or suboptimal arrangements based on statistical results or using dataset-based search, but these methods increase inference overhead while leaving the model's inherent order bias unresolved. Other studies mitigate order sensitivity through supervised fine-tuning using augmented training sets with multiple order variants, but often at the cost of accuracy, trapping the model in consistent yet incorrect hallucinations. In this paper, we propose \textbf{D}ual \textbf{G}roup \textbf{A}dvantage \textbf{O}ptimization (\textbf{DGAO}), which aims to improve model accuracy and order stability simultaneously. DGAO calculates and balances intra-group relative accuracy advantage and inter-group relative stability advantage, rewarding the policy model for generating order-stable and correct outputs while penalizing order-sensitive or incorrect responses. This marks the first time reinforcement learning has been used to mitigate LLMs' order sensitivity. We also propose two new metrics, Consistency Rate and Overconfidence Rate, to reveal the pseudo-stability of previous methods and guide more comprehensive evaluation. Extensive experiments demonstrate that DGAO achieves superior order fairness while improving performance on RAG, mathematical reasoning, and classification tasks. Our code is available at: https://github.com/Hyalinesky/DGAO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。