通过聚合多个草稿生成更优答案，小模型也能超越大模型。
From Drafts to Answers: Unlocking LLM Potential via Aggregation Fine-Tuning
- 训练时学习合并多个草稿生成最终答案，提升推理质量。
- 仅用64k数据微调的Llama3.1-8B在AlpacaEval 2上取得41.3%胜率。
- 适合追求高效推理性能、资源有限的部署场景。
增大数据量和模型规模已被证明能有效提升大语言模型性能。除了训练阶段的扩展，近期研究发现增加推理阶段的计算资源也能进一步改善表现。本文提出聚合微调（Aggregation Fine-Tuning, AFT），一种监督微调范式，使模型学会将多个草稿响应（即提案）整合为一个优化后的答案。推理时采用提议-聚合策略，通过迭代生成提案并聚合，进一步提升性能。基准测试显示，AFT训练模型显著优于标准SFT模型。值得注意的是，仅用64k数据微调的Llama3.1-8B-Base模型，在AlpacaEval 2上达到41.3%的胜率，显著超过更大的Llama3.1-405B-Instruct和GPT4。该提议-聚合框架结合序列优化与并行采样，灵活扩展推理阶段计算量。总体而言，AFT为在不增加数据或模型规模的前提下释放LLM潜力提供了新路径。
原文摘要 · Abstract (English)
Scaling data and model size has been proven effective for boosting the performance of large language models. In addition to training-time scaling, recent studies have revealed that increasing test-time computational resources can further improve performance. In this work, we introduce Aggregation Fine-Tuning (AFT), a supervised finetuning paradigm where the model learns to synthesize multiple draft responses, referred to as proposals, into a single, refined answer, termed aggregation. At inference time, a propose-and-aggregate strategy further boosts performance by iteratively generating proposals and aggregating them. Empirical evaluations on benchmark datasets show that AFT-trained models substantially outperform standard SFT. Notably, an AFT model, fine-tuned from Llama3.1-8B-Base with only 64k data, achieves a 41.3% LC win rate on AlpacaEval 2, surpassing significantly larger LLMs such as Llama3.1-405B-Instruct and GPT4. By combining sequential refinement and parallel sampling, the propose-and-aggregate framework scales inference-time computation in a flexible manner. Overall, These findings position AFT as a promising approach to unlocking additional capabilities of LLMs without resorting to increasing data volume or model size.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。