无需人工标注，模型自动生成偏好数据优化自身表现。
SGPO: Self-Generated Preference Optimization based on Self-Improver
- 模型自生成偏好数据，通过自我改进机制实现在线优化。
- 在AlpacaEval 2.0和Arena-Hard上优于DPO及基线方法。
- 适合追求低成本、高效率对齐的LLM研发团队使用。
大语言模型虽在多样化数据集上进行了广泛预训练，但在实际部署中仍需与人类偏好对齐以确保可靠性和实用性。传统对齐方法多采用离策略学习并依赖人工标注数据集，这限制了其广泛应用，并在训练中引入分布偏移问题。为此，我们提出基于自改进器的自生成偏好优化（SGPO）框架，采用策略内自改进机制。具体而言，改进器对策略模型输出进行优化，自动生成用于直接偏好优化（DPO）的偏好数据。改进器与策略模型融合为单一模型，通过参考监督微调结果，学习对当前回复进行渐进但可察觉的改进，以生成更高质量的偏好数据。在AlpacaEval 2.0和Arena-Hard上的实验表明，所提方法在不使用外部偏好数据的情况下，显著优于DPO及基线自改进方法。
原文摘要 · Abstract (English)
Large language models (LLMs), despite their extensive pretraining on diverse datasets, require effective alignment to human preferences for practical and reliable deployment. Conventional alignment methods typically employ off-policy learning and depend on human-annotated datasets, which limits their broad applicability and introduces distribution shift issues during training. To address these challenges, we propose Self-Generated Preference Optimization based on Self-Improver (SGPO), an innovative alignment framework that leverages an on-policy self-improving mechanism. Specifically, the improver refines responses from a policy model to self-generate preference data for direct preference optimization (DPO) of the policy model. Here, the improver and policy are unified into a single model, and in order to generate higher-quality preference data, this self-improver learns to make incremental yet discernible improvements to the current responses by referencing supervised fine-tuning outputs. Experimental results on AlpacaEval 2.0 and Arena-Hard show that the proposed SGPO significantly improves performance over DPO and baseline self-improving methods without using external preference data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。