通过集成多目标强化学习，高效灵活地微调大模型。
EMORL: Ensemble Multi-Objective Reinforcement Learning for Efficient and Flexible LLM Fine-Tuning
- 用多个独立模型分别优化不同目标，再聚合隐藏状态实现协同。
- 训练消耗降低至1.75万±1650数据点，耗时仅6573±147秒。
- 适合需要多目标平衡与可解释性的大模型微调场景。
大语言模型微调中的多目标强化学习虽有潜力，但仍面临目标冲突、效率低、可扩展性差和可解释性弱等挑战。本文提出集成多目标强化学习（EMORL）框架，通过独立微调多个模型并优化其聚合方式，在提升效率与灵活性的同时，首次将各模型的隐藏状态进行融合，引入多目标上下文信息。该方法基于分层网格搜索算法寻找最优加权组合。在顾问反思生成任务上，采用文本分类模型评分并提供奖励。在PAIR与Psych8k数据集上的实验表明，相比现有基线，EMORL显著降低且稳定训练开销（17,529±1,650 数据点，6,573±147.43 秒），同时提升可扩展性与可解释性，并在多目标表现上保持相当水平。
原文摘要 · Abstract (English)
Recent advances in reinforcement learning (RL) for large language model (LLM) fine-tuning show promise in addressing multi-objective tasks but still face significant challenges, including competing objective balancing, low training efficiency, poor scalability, and limited explainability. Leveraging ensemble learning principles, we introduce an Ensemble Multi-Objective RL (EMORL) framework that fine-tunes multiple models with individual objectives while optimizing their aggregation after the fine-tuning to improve efficiency and flexibility. Our method is the first to aggregate the hidden states of individual models, incorporating contextual information from multiple objectives. This approach is supported by a hierarchical grid search algorithm that identifies optimal weighted combinations. We evaluate EMORL on counselor reflection generation tasks, using text classification models to score the generations and provide rewards during RL fine-tuning. Through comprehensive experiments on the PAIR and Psych8k datasets, we demonstrate the advantages of EMORL against existing baselines: significantly lower and more stable training consumption ($17,529\pm 1,650$ data points and $6,573\pm 147.43$ seconds), improved scalability and explainability, and comparable performance across multiple objectives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。