用强化学习让大模型摆脱偏见，学会独立思考
Making Bias Non-Predictive: Training Robust LLM Reasoning via Reinforcement Learning
- 通过让偏见线索对奖励无预测性，训练模型不依赖表面提示
- 在Qwen3-4B上提升准确率与抗干扰能力，且不牺牲真值判断性能
- 训练后模型可泛化到权威、分散注意力等未见过的偏见类型
大型语言模型日益承担推理与评估角色，但仍易受认知偏见影响——面对共识声称或权威引用等误导性提示时，其推理过程常被扭曲。现有基于提示或监督微调的方法难以泛化，因其仅改变表层行为，未触及使偏见线索具有吸引力的优化目标。本文提出**认识独立性训练（EIT）**，一种基于核心原则的强化学习框架：要习得独立性，偏见线索必须对奖励无预测性。EIT通过平衡冲突策略实现，即偏见信号同时支持正确与错误答案，配合惩罚偏见服从但不奖励偏见一致的奖励设计。在Qwen3-4B上的实验表明，EIT在对抗性偏见下显著提升准确率与鲁棒性，且当偏见与真相一致时仍保持原有性能。值得注意的是，仅在顺从偏见（bandwagon）上训练的模型，能泛化至未见过的偏见类型如权威偏见和分心偏见，表明EIT诱导了可迁移的认识独立性而非特定偏见启发式。EIT进一步在不同基准（MedQA, HellaSwag）、模型家族（Llama-3.2-3B）和规模（Qwen3-8B）间表现优异，优于分布偏移方法（GroupDRO, IRM），且无需环境标签。代码与数据已公开。
原文摘要 · Abstract (English)
Large language models (LLMs) increasingly serve as reasoners and automated evaluators, yet they remain susceptible to cognitive biases -- often altering their reasoning when faced with spurious prompt-level cues such as consensus claims or authority appeals.} Existing mitigations via prompting or supervised fine-tuning fail to generalize, as they modify surface behavior without changing the optimization objective that makes bias cues attractive. We propose \textbf{Epistemic Independence Training (EIT)}, a reinforcement learning framework grounded in a key principle: to learn independence, bias cues must be made non-predictive of reward. EIT operationalizes this through a balanced conflict strategy where bias signals are equally likely to support correct and incorrect answers, combined with a reward design that penalizes bias-following without rewarding bias agreement. Experiments on Qwen3-4B demonstrate that EIT improves both accuracy and robustness under adversarial biases, while preserving performance when bias aligns with truth. Notably, models trained only on bandwagon bias generalize to unseen bias types such as authority and distraction, indicating that EIT induces transferable epistemic independence rather than bias-specific heuristics. \revised{EIT further generalizes across benchmarks (MedQA, HellaSwag), model families (Llama-3.2-3B), and scales (Qwen3-8B), and outperforms distribution-shift methods (GroupDRO, IRM) without requiring environment labels.} Code and data are available at https://anonymous.4open.science/r/bias-mitigation-with-rl-BC47
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。