让大模型更真实地学习人类反馈，避免被自私标签者误导。
Online Learning from Strategic Human Feedback in LLM Fine-Tuning
- 设计动态加权机制，识别并奖励诚实的反馈者。
- 实现次线性后悔率，比传统平均法快得多。
- 适合需要高质量人工反馈的模型对齐场景。
强化学习从人类反馈（RLHF）已成为对齐大型语言模型（LLM）与人类偏好的关键步骤。然而，人类标注者具有自私倾向且偏好多样，可能在在线反馈中策略性地歪曲意见，以影响系统聚合结果。现有方法简单地对每时段反馈取平均，无法识别最可靠的标注者，导致 $T$ 个时间片下线性后悔 $ ext{O}(T)$。我们首次研究了在大模型微调中应对策略性人类标注者的在线学习机制。通过构建新的动态贝叶斯博弈，动态调整标注者的权重，确保其反馈真实，并实现次线性后悔 $ ext{O}(T^{1/2})$。模拟结果表明，该机制显著优于现有基准方案。
原文摘要 · Abstract (English)
Reinforcement learning from human feedback (RLHF) has become an essential step in fine-tuning large language models (LLMs) to align them with human preferences. However, human labelers are selfish and have diverse preferences. They may strategically misreport their online feedback to influence the system's aggregation towards their own preferences. Current practice simply averages labelers' feedback per time and fails to identify the most accurate human labeler, leading to linear regret $\mathcal{O}(T)$ for $T$ time slots. To our best knowledge, we are the first to study online learning mechanisms against strategic human labelers in the LLM fine-tuning process. We formulate a new dynamic Bayesian game and dynamically adjust human labelers' weights in the preference aggregation, ensuring their truthful feedback and sublinear regret $\mathcal{O}(T^{1/2})$. Simulation results demonstrate our mechanism's great advantages over the existing benchmark schemes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。