用在线反馈训练大模型路由，可实时调节性能与成本权衡。
Learning to Route LLMs from Bandit Feedback: One Policy, Many Trade-offs
- 基于上下文带偏见的强化学习，仅用选择结果反馈训练。
- 相比强基线模型，性能提升至少12.46%，节省成本超2.45%。
- 部署时可动态调节性能/成本偏好，无需重新训练。
高效使用大语言模型对大规模部署至关重要：缺乏自适应路由会导致过度支付强模型或因弱模型导致性能下降。为每个查询选择合适模型本质上是一个在线决策问题——模型能力各异，价格波动，用户对准确性和成本的需求不同。然而，大多数路由策略依赖于所有候选模型的离线标注，这一假设在实际部署中不成立，因为只能观测到所选模型的结果。我们提出 BaRP（带偏见反馈路由），一种在训练和部署中均遵循部分反馈限制的方法，同时支持可调偏好推理：运营者可在测试阶段灵活调节性能与成本的权衡，而无需重新训练。将该问题建模为基于提示特征和用户偏好向量的上下文带偏见问题，在训练中模拟在线反馈环境，使路由决策能根据新提示自适应调整，而非依赖全信息离线监督。大量实验表明，该方法持续优于强基线离线路由器至少12.46%，且在最大规模的LLM上仍提升2.45%以上，并对未见过的任务表现出稳健泛化能力。
原文摘要 · Abstract (English)
Efficient use of large language models (LLMs) is critical for deployment at scale: without adaptive routing, systems either overpay for strong models or risk poor performance from weaker ones. Selecting the right LLM for each query is fundamentally an online decision problem: models differ in strengths, prices fluctuate, and users value accuracy and cost differently. Yet most routers are trained offline with labels for all candidate models, an assumption that breaks in deployment, where only the outcome of the chosen model is observed. We bridge this gap with BaRP, a Bandit-feedback Routing with Preferences approach that trains under the same partial-feedback restriction as deployment, while supporting preference-tunable inference: operators can dial the performance/cost trade-off at test time without retraining. Framed as a contextual bandit over prompt features and a user preference vector, our method simulates an online feedback setting during training and adapts its routing decisions to each new prompt, rather than depending on full-information offline supervision. Comprehensive experiments show that our method consistently outperforms strong offline routers by at least 12.46% and the largest LLM by at least 2.45%, and generalizes robustly for unseen tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。