用贝叶斯优化学习可解释的逐标记奖励，提升大模型对齐效率
Learning Explainable Dense Reward Shapes via Bayesian Optimization
- 基于SHAP/LIME等可解释方法生成逐标记奖励
- 在多个任务上比基线更快找到最优策略，性能提升明显
- 理论保证：保持原奖励下的最优策略不变
当前大型语言模型对齐中的强化学习人类反馈（RLHF）通常为序列分配标量奖励，以最后一个词作为整个序列质量的代理指标。这导致反馈稀疏且词级信用分配不佳。本文将奖励塑造视为聚焦于词级信用分配的优化问题。提出一种奖励塑造函数，利用SHAP和LIME等可解释方法从奖励模型中估计每个词的奖励。为学习该函数参数，采用结合贝叶斯优化与策略训练的双层优化框架，以应对词奖励估计中的噪声。实验表明，更均衡的词级奖励分配能提升下游任务表现，并加速训练中最优策略的发现。此外，理论上证明：满足特征可加性属性的可解释方法能保持原始奖励下的最优策略。
原文摘要 · Abstract (English)
Current reinforcement learning from human feedback (RLHF) pipelines for large language model (LLM) alignment typically assign scalar rewards to sequences, using the final token as a surrogate indicator for the quality of the entire sequence. However, this leads to sparse feedback and suboptimal token-level credit assignment. In this work, we frame reward shaping as an optimization problem focused on token-level credit assignment. We propose a reward-shaping function leveraging explainability methods such as SHAP and LIME to estimate per-token rewards from the reward model. To learn parameters of this shaping function, we employ a bilevel optimization framework that integrates Bayesian Optimization and policy training to handle noise from the token reward estimates. Our experiments show that achieving a better balance of token-level reward attribution leads to performance improvements over baselines on downstream tasks and finds an optimal policy faster during training. Furthermore, we show theoretically that explainability methods that are feature additive attribution functions maintain the optimal policy as the original reward.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。