用嵌入相似度做奖励，让小模型更稳定地学习大模型的语义意图。
Parent-Guided Semantic Reward Model (PGSRM): Embedding-Based Reward Functions for Reinforcement Learning of Transformer Language Models
- 用父模型输出嵌入与子模型输出的余弦相似度作为奖励信号
- 在5个任务上实现更平滑的奖励提升和更稳定的PPO训练动态
- 无需人工标注或额外训练，适合资源有限的小型语言模型对齐
我们提出父模型引导的语义奖励模型（PGSRM），一种用于Transformer语言模型强化学习的轻量级奖励框架。PGSRM用父模型参考输出嵌入与子模型生成输出之间的余弦相似度替代二值正确性信号、人工偏好数据或训练好的奖励模型，生成无需人工标注且无需额外训练的密集语义奖励。我们在五个语言任务上应用PGSRM，结果表明其相比二值奖励基线能实现更平滑的奖励提升和更稳定的PPO动态，表明基于嵌入的语义奖励是小型Transformer模型中父模型引导对齐的一种实用替代方案。
原文摘要 · Abstract (English)
We introduce the Parent-Guided Semantic Reward Model (PGSRM), a lightweight reward framework for reinforcement learning (RL) of transformer language models. PGSRM replaces binary correctness signals, human preference data, and trained reward models with a simple signal: cosine similarity between a parent model's reference output embedding and a child model's generated output for the same input. This yields a dense, semantically meaningful reward with no human annotation or additional model training. We apply PGSRM on five language tasks and find that it produces smoother reward improvement and more stable PPO dynamics than a binary reward baseline, suggesting that embedding-based semantic rewards are a practical alternative to RLHF-style reward modeling for parent-guided alignment in smaller transformer models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。