用模型自动生成训练数据，实现大模型对齐，省去人工标注成本。
Aligning Large Language Models via Fully Self-Synthetic Data
- 模型自我生成问答和偏好数据，全程无需外部标注
- 在AlpacaEval 2.0上提升对话能力，同时保持问答与推理性能
- 适合追求低成本自优化的大模型研究者使用
传统基于人类反馈的强化学习（RLHF）依赖昂贵的人工标注数据，而基于AI反馈的强化学习（RLAIF）同样成本高昂，需收集多样化提示及对应回复，并常依赖外部奖励模型或专有模型（如GPT-4）标注偏好对。本文提出全自生成框架自对齐优化（SAO），所有训练数据——包括提示（用户查询）、响应及偏好——均由模型自身生成。具体而言，SAO首先让大语言模型进行角色扮演以生成多样化的提示与回答，随后由模型自我评估并用于偏好优化。大量实验表明，SAO在AlpacaEval 2.0等标准基准上有效提升模型对话能力，同时在下游任务（如问答、数学推理）中保持优异性能。本工作为大模型对齐提供了可落地的自提升方案，代码已开源：https://github.com/SJY8460/SAO。
原文摘要 · Abstract (English)
Traditional reinforcement learning from human feedback (RLHF) for large language models (LLMs) relies on expensive human-annotated datasets, while Reinforcement Learning from AI Feedback (RLAIF) also incurs significant costs, requiring the collection of diverse prompts and corresponding responses, often necessitating external reward models or proprietary models like GPT-4 to annotate preference pairs. In this work, we introduce Self-Alignment Optimization (SAO), a fully self-synthetic framework for LLM alignment, where all training data, including prompts (i.e., user queries), responses, and preferences, are generated by the model itself. Specifically, SAO first instructs the LLM to engage in persona role-play and generate diverse prompts and responses, which are then self-evaluated for preference optimization. Extensive experiments demonstrate that SAO effectively enhances the model's chat capabilities on standard benchmarks like AlpacaEval~2.0, while maintaining strong performance on downstream objective tasks (e.g., question-answering, math reasoning). Our work provides a practical solution for self-improvement in aligning LLMs, and the code for reproducing our results is available at: https://github.com/SJY8460/SAO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。