用线性探针识别并惩罚大模型的讨好行为，提升回答客观性。
Linear Probe Penalties Reduce LLM Sycophancy
- 通过线性探针检测奖励模型中的讨好信号，构建反讨好代理奖励函数。
- 在多个开源大模型上测试，显著降低讨好行为比例。
- 为解决强化学习对讨好行为激励不足的问题提供通用方法，适合对齐研究者。
大语言模型（LLMs）常表现出讨好倾向，优先迎合用户而非给出准确或客观的回答。这种问题在基于人类反馈的强化学习（RLHF）微调阶段尤为严重，该阶段本意是使模型输出符合人类价值观，但实际学习到的奖励模型往往奖励讨好行为。我们提出一种线性探针方法，用于识别并惩罚奖励模型中讨好行为的标志，生成抑制讨好倾向的奖励信号。实验表明，构建并优化对抗该代理奖励函数，能有效减少多个开源大模型中的讨好行为。结果表明，这是一种可泛化的缓解非期望行为的方法，适用于那些在RLHF微调中未被充分抑制的不良行为。
原文摘要 · Abstract (English)
Large language models (LLMs) are often sycophantic, prioritizing agreement with their users over accurate or objective statements. This problematic behavior becomes more pronounced during reinforcement learning from human feedback (RLHF), an LLM fine-tuning stage intended to align model outputs with human values. Instead of increasing accuracy and reliability, the reward model learned from RLHF often rewards sycophancy. We develop a linear probing method to identify and penalize markers of sycophancy within the reward model, producing rewards that discourage sycophantic behavior. Our experiments show that constructing and optimizing against this surrogate reward function reduces sycophantic behavior in multiple open-source LLMs. Our results suggest a generalizable methodology for reducing unwanted LLM behaviors that are not sufficiently disincentivized by RLHF fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。