用弱监督扩充语言模型对齐数据，降低人工标注成本。
Reward Modeling with Weak Supervision for Language Models
- 通过启发式规则和标签模型，用噪声数据自动标注偏好数据。
- 小数据集上弱监督显著提升奖励模型性能，大数据集效果递减。
- 用大模型生成并弱标注新数据，是扩展偏好数据的有效路径。
大型语言模型（LLMs）的快速发展使其在各类任务中广泛应用，其中基于人类反馈的强化学习（RLHF）是其训练的关键环节，用于对齐模型输出与用户意图。在RLHF过程中，奖励模型通过人工标注者或AI系统判断的响应偏好进行训练，再通过强化学习优化语言模型。本文提出利用弱监督策略扩展RLHF数据集并提升奖励模型性能。弱监督采用噪声或不精确的数据标注，减少对昂贵人工标注的依赖。通过分析现有RLHF数据集，识别与响应偏好相关的启发式规则，编写简单标注函数，并使用标签模型对未标注数据进行弱标注。实验表明，弱监督在小数据集上能显著提升奖励模型性能，但在原始标注数据较大的情况下效果减弱。此外，使用大语言模型生成响应并进行弱标注，为扩展偏好数据提供了有前景的方法。
原文摘要 · Abstract (English)
Recent advancements in large language models (LLMs) have led to their increased application across various tasks, with reinforcement learning from human feedback (RLHF) being a crucial part of their training to align responses with user intentions. In the RLHF process, a reward model is trained using responses preferences determined by human labelers or AI systems, which then refines the LLM through reinforcement learning. This work introduces weak supervision as a strategy to extend RLHF datasets and enhance reward model performance. Weak supervision employs noisy or imprecise data labeling, reducing reliance on expensive manually labeled data. By analyzing RLHF datasets to identify heuristics that correlate with response preference, we wrote simple labeling functions and then calibrated a label model to weakly annotate unlabeled data. Our evaluation show that while weak supervision significantly benefits smaller datasets by improving reward model performance, its effectiveness decreases with larger, originally labeled datasets. Additionally, using an LLM to generate and then weakly label responses offers a promising method for extending preference data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。