仅用9000条标注数据实现大模型对齐，效果接近主流方法。
ALMA: Alignment with Minimal Annotation
- 通过少样本提示生成与多检查点响应，自动生成高质量合成数据。
- 10轮自举训练后，在AlpacaEval 2.0上比基准低0.1%。
- 适合资源有限但需高效对齐的团队使用。
现有大语言模型对齐方法通常依赖数百万人工标注或外部对齐模型生成合成数据。本文提出ALMA：仅需9000条标注样本（不足传统方法的1%）即可实现有效对齐。ALMA通过新方法生成大量高质量合成对齐数据：基于少样本学习的多样化提示生成、利用多个模型检查点的多样化响应生成，以及通过分数聚合与自蒸馏增强判别器（奖励模型）。仅使用预训练Llama3基础模型、5000条SFT样本和4000条判别器标注，ALMA在多种对齐基准上表现接近Llama3-Instruct（如AlpacaEval 2.0得分相差仅0.1%）。该方法采用多轮自举式数据生成与训练流程，持续优化达10轮，突破以往通常3轮的上限。结果表明，基础模型已具备充分对齐知识，而有效的合成数据生成方法可充分释放其潜力。
原文摘要 · Abstract (English)
Recent approaches to large language model (LLM) alignment typically require millions of human annotations or rely on external aligned models for synthetic data generation. This paper introduces ALMA: Alignment with Minimal Annotation, demonstrating that effective alignment can be achieved using only 9,000 labeled examples -- less than 1% of conventional approaches. ALMA generates large amounts of high-quality synthetic alignment data through new techniques: diverse prompt synthesis via few-shot learning, diverse response generation with multiple model checkpoints, and judge (reward model) enhancement through score aggregation and self-distillation. Using only a pretrained Llama3 base model, 5,000 SFT examples, and 4,000 judge annotations, ALMA achieves performance close to Llama3-Instruct across diverse alignment benchmarks (e.g., 0.1% difference on AlpacaEval 2.0 score). These results are achieved with a multi-round, self-bootstrapped data synthesis and training recipe that continues to improve for 10 rounds, surpassing the typical 3-round ceiling of previous methods. These results suggest that base models already possess sufficient knowledge for effective alignment, and that synthetic data generation methods can expose it.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。