用因果方法减少大模型奖励模型的啰嗦偏好。
Mitigating Length Bias in RLHF through a Causal Lens
- 构造内容相似但长度不同的响应对，分离质量与冗长性。
- 训练后奖励模型更关注内容，输出平均缩短18%。
- 适合改进对话系统和生成模型的对齐效果。
强化学习从人类反馈（RLHF）广泛用于使大语言模型（LLMs）与人类偏好对齐。然而，经由RLHF训练的奖励模型常表现出长度偏差——将冗长误认为高质量。本文提出一种因果框架来分析并缓解这一问题。核心是反事实数据增强方法，生成旨在分离内容质量与冗长性的响应对。具体包括：(1) 内容相似但长度不同的响应对；(2) 长度相近但内容不同的响应对。实验表明，该方法有效降低奖励分配中的长度偏差，使策略模型输出更简洁、聚焦内容。结果证明，该方法显著提升奖励建模在RLHF流程中的鲁棒性与内容敏感性。
原文摘要 · Abstract (English)
Reinforcement learning from human feedback (RLHF) is widely used to align large language models (LLMs) with human preferences. However, RLHF-trained reward models often exhibit length bias -- a systematic tendency to favor longer responses by conflating verbosity with quality. We propose a causal framework for analyzing and mitigating length bias in RLHF reward modeling. Central to our approach is a counterfactual data augmentation method that generates response pairs designed to isolate content quality from verbosity. These counterfactual examples are then used to train the reward model, enabling it to assess responses based on content quality independently of verbosity. Specifically, we construct (1) length-divergent pairs with similar content and (2) content-divergent pairs of similar length. Empirical evaluations show that our method reduces length bias in reward assignment and leads to more concise, content-focused outputs from the policy model. These findings demonstrate that the proposed approach effectively reduces length bias and improves the robustness and content sensitivity of reward modeling in RLHF pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。