通过密集偏好优化融合异构模型，提升大模型性能
FuseRL: Dense Preference Optimization for Heterogeneous Model Fusion
- 分两阶段融合多模型：先加权监督微调初始化，再基于多源输出优化偏好
- 在AlpacaEval-2和Arena-Hard上达到8B模型顶尖水平
- 适合需要融合多个模型能力的高阶LLM研发场景
异构模型融合通过整合结构各异的多个模型的知识与能力，提升大语言模型性能。然而，现有方法通常仅从源模型中选择每个提示的最佳输出，导致源知识利用不充分且优化信号稀疏。为此，我们提出FuseRL，一种包含两阶段框架FuseSFT与FusePO的新方法，以最大化源模型潜力。FuseSFT通过在每提示的多样化输出上进行加权监督微调（SFT），融合异构源模型优势，建立稳健初始化。FusePO则基于多个源模型的输出优化加权偏好，实现更优对齐性能。大量实验表明，该框架在多种偏好对齐方法（包括RLOO、DPO和SimPO）下均有效。以Llama-3.1-8B-Instruct为目标模型，在AlpacaEval-2和Arena-Hard基准上达到8B模型当前最优表现。进一步分析显示，FuseSFT可正则化训练过程以减少过拟合，而FusePO引入密集且多样化的优化信号。
原文摘要 · Abstract (English)
Heterogeneous model fusion enhances the performance of LLMs by integrating the knowledge and capabilities of multiple structurally diverse models. However, existing approaches often rely solely on selecting the best output for each prompt from source models, which underutilizes their full potential due to limited source knowledge and results in sparse optimization signals. To address this limitation, we propose FuseRL, a novel two-stage framework comprising FuseSFT and FusePO to maximize the utilization of source LLMs. FuseSFT establishes a robust initialization by integrating the strengths of heterogeneous source models through weighted supervised fine-tuning (SFT) on diverse outputs for each prompt. FusePO optimizes weighted preferences based on the outputs of multiple source models to enable superior alignment performance. Extensive experiments demonstrate the effectiveness of our framework across various preference alignment methods, including RLOO, DPO, and SimPO. Using Llama-3.1-8B-Instruct as the target model, our approach achieves state-of-the-art performance among 8B LLMs on the AlpacaEval-2 and Arena-Hard benchmarks. Further analysis suggests that FuseSFT regularizes the training process to reduce overfitting, while FusePO introduces dense and diverse signals for preference optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。