从真实用户交互中直接学习奖励模型，无需人工标注偏好对。
WildReward: Learning Reward Models from In-the-Wild Human Interactions
- 用真实对话数据提取隐式反馈，跳过人工标注偏好对
- 18.6万高质量样本训练，性能媲美甚至超越传统模型
- 用户多样性越高，模型越强，适合在线强化学习场景
奖励模型（RMs）对大语言模型（LLMs）训练至关重要，但通常依赖大规模人工标注的偏好对。随着LLMs广泛应用，真实环境中的用户交互成为隐式奖励信号的丰富来源。本文探索是否可直接从真实交互中构建奖励模型。基于WildChat数据源，提出一套提取可靠人类反馈的流程，获得18.6万条高质量样本，通过序数回归直接在用户反馈上训练WildReward，无需偏好对。大量实验表明，WildReward在性能上可与传统模型媲美甚至更优，具备更好的校准性与跨样本一致性。还发现用户多样性提升能显著增强模型表现。将WildReward用于在线DPO训练，在多个任务上均取得显著改进。代码与数据已公开于https://github.com/THU-KEG/WildReward。
原文摘要 · Abstract (English)
Reward models (RMs) are crucial for the training of large language models (LLMs), yet they typically rely on large-scale human-annotated preference pairs. With the widespread deployment of LLMs, in-the-wild interactions have emerged as a rich source of implicit reward signals. This raises the question: Can we develop reward models directly from in-the-wild interactions? In this work, we explore this possibility by adopting WildChat as an interaction source and proposing a pipeline to extract reliable human feedback, yielding 186k high-quality instances for training WildReward via ordinal regression directly on user feedback without preference pairs. Extensive experiments demonstrate that WildReward achieves comparable or even superior performance compared to conventional reward models, with improved calibration and cross-sample consistency. We also observe that WildReward benefits directly from user diversity, where more users yield stronger reward models. Finally, we apply WildReward to online DPO training and observe significant improvements across various tasks. Code and data are released at https://github.com/THU-KEG/WildReward.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。