用简单领域数据训练通用奖励模型，让大模型在无偏好数据的领域也能对齐人类意图。
Aligning LLMs with Domain Invariant Reward Models
- 通过双损失优化，让奖励模型在不同领域间保持偏好一致性。
- 跨语言、降噪、少样本到全量等4种场景下性能均提升，最高准确率提高7.5个百分点。
- 适合缺乏标注数据的领域，如小语种或复杂任务场景下的模型对齐。
在缺乏偏好数据的目标领域中对齐大语言模型（LLM）面临挑战。本文提出一种利用更易获取人类偏好数据的简单源领域反馈，来学习目标领域奖励模型的方法。核心思想是：尽管领域差异大，但人类偏好包含可被奖励模型捕捉的领域无关概念。我们提出 extit{method}框架，通过双损失优化——域损失最小化源与目标分布差异，源损失优化源域偏好。该方法在4个不同设置中验证：(1) 跨语言迁移（准确率从0.621升至0.661），(2) 清洁到噪声（0.671→0.703），(3) 少样本到全量（0.845→0.920），(4) 简单到复杂任务（相关性0.508→0.556）。代码、模型和数据已开源。
原文摘要 · Abstract (English)
Aligning large language models (LLMs) to human preferences is challenging in domains where preference data is unavailable. We address the problem of learning reward models for such target domains by leveraging feedback collected from simpler source domains, where human preferences are easier to obtain. Our key insight is that, while domains may differ significantly, human preferences convey \emph{domain-agnostic} concepts that can be effectively captured by a reward model. We propose \method, a framework that trains domain-invariant reward models by optimizing a dual loss: a domain loss that minimizes the divergence between source and target distribution, and a source loss that optimizes preferences on the source domain. We show \method is a general approach that we evaluate and analyze across 4 distinct settings: (1) Cross-lingual transfer (accuracy: $0.621 \rightarrow 0.661$), (2) Clean-to-noisy (accuracy: $0.671 \rightarrow 0.703$), (3) Few-shot-to-full transfer (accuracy: $0.845 \rightarrow 0.920$), and (4) Simple-to-complex tasks transfer (correlation: $0.508 \rightarrow 0.556$). Our code, models and data are available at \url{https://github.com/portal-cornell/dial}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。