arXiv:2510.08439cs.LGcs.AI2025-10被引 11

用强化学习让大模型自动选型,省钱又高效。

xRouter: Training Cost-Aware LLMs Orchestration System via Reinforcement Learning

  • 通过强化学习训练路由模块,自动决定调用哪个模型
  • 在多个测试中实现显著降本,同时保持任务完成率
  • 适合需要低成本部署复杂推理任务的场景

现代大模型部署面临成本与性能的广泛权衡:高端模型推理能力强但昂贵,轻量模型便宜却在复杂任务上脆弱。静态升级规则和关键词启发式方法无法有效利用这一谱系,且难以跨任务自适应。我们提出xRouter,一种基于工具调用的路由系统,其学习型路由器可直接回答或调用一个或多个外部模型。路由器通过显式成本感知奖励进行端到端强化学习训练,无需人工设计路由规则。系统完整实现了强化学习框架,包括奖励与成本核算,以及部署和评估流程。在多种基准测试中,xRouter实现了出色的成本-性能平衡(例如,在任务完成率相当的情况下大幅降低开销),并提供了关于哪些因素真正有助于学习型路由的实证洞察,涵盖模型可训练性,以及在小型开源模型中激发复杂编排行为的难度。我们希望这些发现及开放实现能为推进学习型、成本感知的大模型协同提供实用基础。

原文摘要 · Abstract (English)

Modern LLM deployments confront a widening cost-performance spectrum: premium models deliver strong reasoning but are expensive, while lightweight models are economical yet brittle on complex tasks. Static escalation rules and keyword heuristics under-utilize this spectrum and fail to adapt across task types. We present xRouter, a tool-calling-based routing system in which a learned router can either answer directly or invoke one or more external models. The router is trained end-to-end with reinforcement learning using an explicit, cost-aware reward that encodes cost-performance trade-offs, eliminating the need for hand-engineered routing rules. Our implementation encompasses the full reinforcement learning framework, including reward and cost accounting, as well as the deployment and evaluation pipelines. Across diverse benchmarks, xRouter achieves strong cost-performance trade-offs (e.g., substantial cost reductions at comparable task completion rates), and provides empirical insights into what reliably helps learned routing and what does not, ranging from model trainability to the difficulty of eliciting sophisticated orchestration behaviors in small open models. We hope these findings and our open implementation will serve as a practical substrate for advancing learned, cost-aware LLM orchestration.

大模型调度强化学习成本优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。