用强化学习动态调整大模型集成权重,提升多任务表现。
RLAE: Reinforcement Learning-Assisted Ensemble for LLMs
- 构建马尔可夫决策过程,让智能体根据上下文实时调整模型权重。
- 在多个任务上最高提升3.3%准确率,且延迟更低。
- 无需重新训练即可跨任务泛化,适合实际部署场景。
集成大型语言模型(LLMs)能够有效融合不同模型的优势,是提升各类任务性能的有前景方法。然而,现有方法通常依赖固定加权策略,无法适应模型能力随上下文动态变化的特点。本文提出一种基于强化学习的集成框架RLAE,将LLM集成重新建模为马尔可夫决策过程(MDP)。该方法引入一个强化学习智能体,根据输入上下文和中间生成状态动态调整集成权重,并使用与最终输出质量直接相关的奖励进行训练。我们实现了单智能体和多智能体强化学习版本($ ext{RLAE}_ ext{PPO}$ 和 $ ext{RLAE}_ ext{MAPPO}$),在多种任务上的大量评估显示,相比传统集成方法,性能最高提升达3.3%准确率点。此外,该方法展现出优异的跨任务泛化能力,无需重训练,同时实现更低的时间延迟。
原文摘要 · Abstract (English)
Ensembling large language models (LLMs) can effectively combine diverse strengths of different models, offering a promising approach to enhance performance across various tasks. However, existing methods typically rely on fixed weighting strategies that fail to adapt to the dynamic, context-dependent characteristics of LLM capabilities. In this work, we propose Reinforcement Learning-Assisted Ensemble for LLMs (RLAE), a novel framework that reformulates LLM ensemble through the lens of a Markov Decision Process (MDP). Our approach introduces a RL agent that dynamically adjusts ensemble weights by considering both input context and intermediate generation states, with the agent being trained using rewards that directly correspond to the quality of final outputs. We implement RLAE using both single-agent and multi-agent reinforcement learning algorithms ($\text{RLAE}_\text{PPO}$ and $\text{RLAE}_\text{MAPPO}$ ), demonstrating substantial improvements over conventional ensemble methods. Extensive evaluations on a diverse set of tasks show that RLAE outperforms existing approaches by up to $3.3\%$ accuracy points, offering a more effective framework for LLM ensembling. Furthermore, our method exhibits superior generalization capabilities across different tasks without the need for retraining, while simultaneously achieving lower time latency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。