用对抗生成方式让大模型自我对齐，无需人工标注。
Self-Generative Adversarial Fine-Tuning for Large Language Models
- 让模型同时扮演生成者和判别者，在内部完成对齐训练。
- 在多个基准上达到当前最优性能，优于传统方法。
- 适合需要低成本高质量对齐的AI研发团队使用。
将大型语言模型（LLMs）对齐的微调通常依赖监督微调或人类反馈强化学习，但二者受限于高质量标注数据的成本与稀缺性。近期的自对弈和合成数据方法虽降低了依赖，却常基于启发式假设或无根基的自我评估，易导致偏差累积和性能漂移。本文提出自生成对抗大模型（SGALM），一种统一的微调框架，将对齐建模为单一语言模型内的生成对抗游戏。SGALM无需外部奖励模型，即可联合演化生成与判别能力。理论与实证结果表明，该方法实现当前最佳性能，兼具有效对齐算法与稳健合成数据引擎双重功能。
原文摘要 · Abstract (English)
Fine-tuning large language models (LLMs) for alignment typically relies on supervised fine-tuning or reinforcement learning from human feedback, both limited by the cost and scarcity of high-quality annotations. Recent self-play and synthetic data approaches reduce this dependence but often rely on heuristic assumptions or ungrounded self-evaluation, which can cause bias accumulation and performance drift. In this paper, we propose Self-Generative Adversarial LLM (SGALM), a unified fine-tuning framework that formulates alignment as a generative adversarial game within a single LLM. SGALM jointly evolves generation and discrimination capabilities without external reward models. Theoretical and empirical results demonstrate that SGALM achieves state-of-the-art performance, serves as an effective alignment algorithm and a robust synthetic data engine.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。