用逆强化学习解构大模型训练目标，揭示其隐含奖励机制。
Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse Reinforcement Learning
- 通过逆强化学习反推大模型的隐含奖励函数
- 对毒性对齐模型的预测准确率达85%
- 可指导新模型微调，提升对齐效果
采用人类反馈强化学习（RLHF）训练的大语言模型（LLM）表现出卓越能力，但其底层奖励函数与决策机制仍不透明。本文提出新方法，利用逆强化学习（IRL）恢复模型的隐含奖励函数。在不同规模的毒性对齐LLM上进行实验，提取的奖励模型在预测人类偏好方面最高达到85%的准确率。分析揭示了奖励函数不可识别性、模型规模与可解释性的关系，以及RLHF过程中的潜在陷阱。结果表明，通过IRL获得的奖励模型可用于微调新LLM，使其在毒性基准测试中表现相当或更优。该工作为理解与改进大模型对齐提供了新视角，对负责任地开发和部署这些系统具有重要意义。
原文摘要 · Abstract (English)
Large language models (LLMs) trained with Reinforcement Learning from Human Feedback (RLHF) have demonstrated remarkable capabilities, but their underlying reward functions and decision-making processes remain opaque. This paper introduces a novel approach to interpreting LLMs by applying inverse reinforcement learning (IRL) to recover their implicit reward functions. We conduct experiments on toxicity-aligned LLMs of varying sizes, extracting reward models that achieve up to 85% accuracy in predicting human preferences. Our analysis reveals key insights into the non-identifiability of reward functions, the relationship between model size and interpretability, and potential pitfalls in the RLHF process. We demonstrate that IRL-derived reward models can be used to fine-tune new LLMs, resulting in comparable or improved performance on toxicity benchmarks. This work provides a new lens for understanding and improving LLM alignment, with implications for the responsible development and deployment of these powerful systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。