用无标注数据迭代优化奖励模型，省钱省力还有效。
Semi-Supervised Reward Modeling via Iterative Self-Training
- 用伪标签+高置信度筛选，从无标注数据中挖掘有用信息
- 在多种模型上表现接近纯标注数据训练的水平
- 适合资源有限但想提升奖励模型的团队
奖励模型（RM）捕捉人类价值观与偏好，在基于人类反馈的强化学习（RLHF）中对对齐预训练大语言模型（LLMs）起核心作用。传统训练依赖大量人工标注的偏好数据，面临可扩展性和成本挑战。为此，我们提出半监督奖励建模（SSRM），利用无标注数据增强RM训练。给定无标注数据集，SSRM包含三个关键迭代步骤：对无标注样本进行伪标签生成，通过置信度阈值筛选高可信样本，并在精炼数据集上进行监督微调。在多种模型配置的广泛实验中，我们证明了SSRM能显著提升奖励模型性能，且无需额外标注成本。值得注意的是，SSRM可达到与同等规模全标注数据训练模型相当的性能。总体而言，SSRM大幅降低了对大规模人工标注数据的依赖，从而显著减少训练高效奖励模型所需的成本和时间。
原文摘要 · Abstract (English)
Reward models (RM) capture the values and preferences of humans and play a central role in Reinforcement Learning with Human Feedback (RLHF) to align pretrained large language models (LLMs). Traditionally, training these models relies on extensive human-annotated preference data, which poses significant challenges in terms of scalability and cost. To overcome these limitations, we propose Semi-Supervised Reward Modeling (SSRM), an approach that enhances RM training using unlabeled data. Given an unlabeled dataset, SSRM involves three key iterative steps: pseudo-labeling unlabeled examples, selecting high-confidence examples through a confidence threshold, and supervised finetuning on the refined dataset. Across extensive experiments on various model configurations, we demonstrate that SSRM significantly improves reward models without incurring additional labeling costs. Notably, SSRM can achieve performance comparable to models trained entirely on labeled data of equivalent volumes. Overall, SSRM substantially reduces the dependency on large volumes of human-annotated data, thereby decreasing the overall cost and time involved in training effective reward models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。