用强化学习动态组合小模型，让大模型推理更省资源、更快
HierRouter: Coordinated Routing of Specialized Large Language Models via Reinforcement Learning
- 用强化学习构建分层路由，按上下文选合适的小模型
- 在6个任务上提升响应质量2.4倍，额外开销极小
- 适合资源有限的实时部署场景，如移动端或边缘计算
大型语言模型（LLMs）在诸多任务中表现卓越，但计算与内存开销巨大，限制了其在资源受限或实时场景中的部署。为此，我们提出HierRouter，一种分层路由方法，通过动态组合一组专用轻量级语言模型来构建推理流水线。该方法被建模为有限时域马尔可夫决策过程（MDP），采用基于近端策略优化（PPO）的强化学习代理，迭代选择多跳推理中每阶段应调用的模型。代理根据不断演化的上下文和累积成本做出上下文感知的路由决策。在三个开源候选LLM及六个基准上的实验表明，相比独立使用单个模型，HierRouter在问答、代码生成和数学推理等任务中将响应质量提升高达2.4倍，平均仅增加极少额外推理开销。结果表明，分层路由在实现高效、高性能的LLM推理方面具有巨大潜力。所有代码见：https://github.com/Nikunj-Gupta/hierouter。
原文摘要 · Abstract (English)
Large Language Models (LLMs) deliver state-of-the-art performance across many tasks but impose high computational and memory costs, limiting their deployment in resource-constrained or real-time settings. To address this, we propose HierRouter, a hierarchical routing approach that dynamically assembles inference pipelines from a pool of specialized, lightweight language models. Formulated as a finite-horizon Markov Decision Process (MDP), our approach trains a Proximal Policy Optimization (PPO)-based reinforcement learning agent to iteratively select which models to invoke at each stage of multi-hop inference. The agent conditions on the evolving context and accumulated cost to make context-aware routing decisions. Experiments with three open-source candidate LLMs across six benchmarks, including QA, code generation, and mathematical reasoning, show that HierRouter improves response quality by up to 2.4x compared to using individual models independently, while incurring only a minimal additional inference cost on average. These results highlight the promise of hierarchical routing for cost-efficient, high-performance LLM inference. All codes can be found here https://github.com/ Nikunj-Gupta/hierouter.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。