用多个开源模型协作生成高质量数据，提升大模型对齐效果
Improving Model Alignment Through Collective Intelligence of Open-Source LLMS
- 通过多模型协作生成对齐数据，替代依赖人工标注
- 使小模型在硬评测中胜率从19.5%提升至48.3%
- 支持模型自我迭代，推动开源模型能力边界
构建有益且无害的大语言模型需要基于人类指令与反馈的有效对齐方法，这通常依赖高质量的人工标注数据。然而，构建此类数据集成本高、难以扩展，且可能存在多样性与泛化能力不足的问题。为此，我们提出混合智能体对齐（MoAA），利用多种语言模型的集体优势生成高质量对齐数据。通过MoAA，我们提升了监督微调与偏好优化的效果，性能优于仅使用单一模型（如GPT-4o）生成对齐数据的方式。评估结果表明，该方法将LLaMA-3.1-8B-Instruct在Arena-Hard上的胜率从19.5%提升至48.3%，在AlpacaEval2上从22.33%提升至57.23%，展示了通过可扩展、多样化的合成数据实现模型对齐的新方向。此外，我们证明了MoAA可构建自进化流水线，经由MoAA生成数据微调后的模型超越自身初始能力，验证了该方法可在不依赖更强外部监督的前提下推进开源大模型的发展。数据与代码将公开。
原文摘要 · Abstract (English)
Building helpful and harmless large language models (LLMs) requires effective model alignment approach based on human instructions and feedback, which necessitates high-quality human-labeled data. Constructing such datasets is often expensive and hard to scale, and may face potential limitations on diversity and generalization. To address these challenges, we introduce Mixture of Agents Alignment (MoAA), that leverages the collective strengths of various language models to provide high-quality data for model alignment. By employing MoAA, we enhance both supervised fine-tuning and preference optimization, leading to improved performance compared to using a single model alone to generate alignment data (e.g. using GPT-4o alone). Evaluation results show that our approach can improve win rate of LLaMA-3.1-8B-Instruct from 19.5 to 48.3 on Arena-Hard and from 22.33 to 57.23 on AlpacaEval2, highlighting a promising direction for model alignment through this new scalable and diverse synthetic data recipe. Furthermore, we demonstrate that MoAA enables a self-improvement pipeline, where models finetuned on MoA-generated data surpass their own initial capabilities, providing evidence that our approach can push the frontier of open-source LLMs without reliance on stronger external supervision. Data and code will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。