用影响函数分析人类反馈对模型的影响，提升标注一致性。
Understanding Impact of Human Feedback via Influence Functions
- 提出高效近似方法，让大模型奖励模型可用影响函数分析
- 发现标注者常见偏差，并指导其改进反馈策略
- 适合关注模型对齐与标注质量的研究者
在基于人类反馈的强化学习（RLHF）中，从人类反馈中学习合适的奖励模型以对齐大语言模型（LLMs）与人类意图至关重要。然而，人类反馈常存在噪声、不一致或偏差，尤其是在评估复杂回复时，可能导致奖励信号错位，引发不可预见的副作用。为此，本文探索使用影响函数衡量人类反馈对奖励模型性能的影响。我们提出一种计算高效的近似方法，使影响函数可应用于基于大语言模型的奖励模型和大规模偏好数据集。实验展示两个关键应用：(1) 检测人类反馈数据集中常见的标注者偏差；(2) 引导标注者优化反馈策略，使其更贴近专家反馈。通过量化反馈影响，我们认为影响函数能增强反馈可解释性，推动RLHF中的可扩展监督，帮助标注者提供更准确、一致的反馈。源代码见 https://github.com/mintaywon/IF_RLHF。
原文摘要 · Abstract (English)
In Reinforcement Learning from Human Feedback (RLHF), it is crucial to learn suitable reward models from human feedback to align large language models (LLMs) with human intentions. However, human feedback can often be noisy, inconsistent, or biased, especially when evaluating complex responses. Such feedback can lead to misaligned reward signals, potentially causing unintended side effects during the RLHF process. To address these challenges, we explore the use of influence functions to measure the impact of human feedback on the performance of reward models. We propose a compute-efficient approximation method that enables the application of influence functions to LLM-based reward models and large-scale preference datasets. Our experiments showcase two key applications of influence functions: (1) detecting common labeler biases in human feedback datasets and (2) guiding labelers in refining their strategies to better align with expert feedback. By quantifying the impact of human feedback, we believe that influence functions can enhance feedback interpretability and contribute to scalable oversight in RLHF, helping labelers provide more accurate and consistent feedback. Source code is available at https://github.com/mintaywon/IF_RLHF
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。