不依赖奖励模型，仅用提示词实现在线对齐优化。
Just Say What You Want: Only-prompting Self-rewarding Online Preference Optimization
- 仅通过提示词生成偏好数据，避免外部反馈开销。
- 训练后期增强负样本难度，提升对细微偏好的捕捉能力。
- 在小模型上有效，适合资源有限的对齐场景。
针对在线强化学习人类反馈（RLHF）中反馈成本高的问题，本文提出一种仅依赖提示词的自奖励在线优化方法。该方法无需依赖判别器或额外奖励模型即可生成偏好数据，降低交互成本。通过细粒度控制正负样本间的最优性差距，在训练后期主动生成更难的负样本，帮助模型更好地理解细微的人类偏好。在Mistral-7B和Mistral-Instruct-7B两个基础模型上进行实验，显著提升参考模型性能，使AlpacaEval 2.0的长度控制胜率达到34.5%。
原文摘要 · Abstract (English)
We address the challenge of online Reinforcement Learning from Human Feedback (RLHF) with a focus on self-rewarding alignment methods. In online RLHF, obtaining feedback requires interaction with the environment, which can be costly when using additional reward models or the GPT-4 API. Current self-rewarding approaches rely heavily on the discriminator's judgment capabilities, which are effective for large-scale models but challenging to transfer to smaller ones. To address these limitations, we propose a novel, only-prompting self-rewarding online algorithm that generates preference datasets without relying on judgment capabilities. Additionally, we employ fine-grained arithmetic control over the optimality gap between positive and negative examples, generating more hard negatives in the later stages of training to help the model better capture subtle human preferences. Finally, we conduct extensive experiments on two base models, Mistral-7B and Mistral-Instruct-7B, which significantly bootstrap the performance of the reference model, achieving 34.5% in the Length-controlled Win Rates of AlpacaEval 2.0.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。