通过排列一致性优化，让大模型在选择题中不被选项位置影响判断。
Mitigating Selection Bias in Large Language Models via Permutation-Aware GRPO
- 构建题目多种排列组合，用相对优势和一致性奖励训练模型。
- 在7个基准测试中显著降低选择偏差，性能保持高水平。
- 适合需要公平评估的AI系统，如教育测评与智能问答。
用于多选题和成对评估的大语言模型常因选项位置、标签符号等非语义因素产生选择偏差。现有推理时去偏方法成本高且可能损害推理能力，而点式训练忽略了同一题目在不同排列下应有统一答案。为此，我们提出排列感知组相对策略优化（PA-GRPO），通过强制排列一致的语义推理来缓解选择偏差。PA-GRPO为每个实例生成多个候选排列，采用两种互补机制优化模型：(1) 跨排列优势，以同一实例所有排列的平均奖励为基准计算优势；(2) 一致性感知奖励，鼓励模型在不同排列下做出一致决策。实验表明，PA-GRPO在7个基准上优于强基线，显著降低选择偏差，同时保持高整体性能。代码已开源（https://github.com/ECNU-Text-Computing/PA-GRPO）。
原文摘要 · Abstract (English)
Large language models (LLMs) used for multiple-choice and pairwise evaluation tasks often exhibit selection bias due to non-semantic factors like option positions and label symbols. Existing inference-time debiasing is costly and may harm reasoning, while pointwise training ignores that the same question should yield consistent answers across permutations. To address this issue, we propose Permutation-Aware Group Relative Policy Optimization (PA-GRPO), which mitigates selection bias by enforcing permutation-consistent semantic reasoning. PA-GRPO constructs a permutation group for each instance by generating multiple candidate permutations, and optimizes the model using two complementary mechanisms: (1) cross-permutation advantage, which computes advantages relative to the mean reward over all permutations of the same instance, and (2) consistency-aware reward, which encourages the model to produce consistent decisions across different permutations. Experimental results demonstrate that PA-GRPO outperforms strong baselines across seven benchmarks, substantially reducing selection bias while maintaining high overall performance. The code is available on github (https://github.com/ECNU-Text-Computing/PA-GRPO).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。