通过因果分解提升大模型奖励模型的鲁棒性，避免过度依赖回答长度和讨好语气。
Robust Reward Modeling for Large Language Models via Causal Decomposition
- 设计解码器将回答映射到提示意图嵌入，用重构误差正则化奖励模型训练。
- 在数学、帮助性与安全性任务中准确率达0.877，比基线提升显著。
- 适合需要减少虚假线索干扰的对齐研究者与部署场景。
奖励模型在对齐大语言模型中至关重要，但常过度依赖响应长度和过于讨好的语气等虚假线索。现有方法多通过惩罚或控制特定伪特征来缓解，却未明确引导模型基于提示意图建立偏好。本文提出一种解码器，将候选回答映射至输入提示的潜在意图嵌入，并利用重构误差作为信号正则化奖励模型训练。理论证明该信号强调与提示相关的有效信息，抑制无关捷径。在数学、帮助性与安全基准测试中，该方法选出更短且不讨好型的回答，准确率达0.877。将此信号引入Gemma-2-2B-it与Gemma-2-9B-it的奖励模型训练后，RewardBench准确率从0.832提升至0.868。在Best-of-N选择中，既提高长度控制下的胜率，又生成更短输出，且在可控重写测试中对长度增加和轻微离题仍保持稳健。
原文摘要 · Abstract (English)
Reward models are central to aligning large language models, yet they often overfit to spurious cues such as response length and overly agreeable tone. Most prior work weakens these cues directly by penalizing or controlling specific artifacts, but it does not explicitly encourage the model to ground preferences in the prompt's intent. We learn a decoder that maps a candidate answer to the latent intent embedding of the input. The reconstruction error is used as a signal to regularize the reward model training. We provide theoretical evidence that this signal emphasizes prompt-dependent information while suppressing prompt-independent shortcuts. Across math, helpfulness, and safety benchmarks, the decoder selects shorter and less sycophantic candidates with 0.877 accuracy. Incorporating this signal into RM training in Gemma-2-2B-it and Gemma-2-9B-it increases RewardBench accuracy from 0.832 to 0.868. For Best-of-N selection, our framework increases length-controlled win rates while producing shorter outputs, and remains robust to lengthening and mild off-topic drift in controlled rewrite tests.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。