智能路由模型，按用户需求平衡成本与质量。
IPR: Intelligent Prompt Routing with User-Controlled Quality-Cost Trade-offs
- 根据预测质量动态选模型，支持用户自定义容忍度。
- 成本降低43.9%，响应延迟低于150ms，质量不降。
- 新模型接入仅需数小时，适合大规模商业系统。
在大规模商业系统中，如何在保持响应质量的前提下，将查询路由到最经济的大型语言模型，是性能与成本权衡的核心挑战。本文提出IPR——一种质量约束的智能提示路由框架，基于预测响应质量与用户设定的容忍度，动态选择最优模型。IPR包含三大创新：(1) 模块化架构搭配轻量级质量估计算法，利用150万条带校准质量评分的提示数据训练，实现跨模型家族的细粒度质量预测；(2) 用户可控的路由机制，通过参数τ∈[0,1]明确控制质量-成本权衡；(3) 使用冻结编码器+模型专用适配器的可扩展设计，将新模型集成时间从数天缩短至数小时。为系统训练与评估，我们构建了工业级基准IPRBench，包含150万例数据，覆盖11个主流大模型。部署于主流云平台后，IPR实现43.9%成本下降，质量与最强的Claude系列持平，请求处理延迟低于150毫秒。系统及产品详情已公开于https://aws.amazon.com/bedrock/intelligent-prompt-routing/
原文摘要 · Abstract (English)
Routing incoming queries to the most cost-effective LLM while maintaining response quality poses a fundamental challenge in optimizing performance-cost trade-offs for large-scale commercial systems. We present IPR\, -- \,a quality-constrained \textbf{I}ntelligent \textbf{P}rompt \textbf{R}outing framework that dynamically selects optimal models based on predicted response quality and user-specified tolerance levels. IPR introduces three key innovations: (1) a modular architecture with lightweight quality estimators trained on 1.5M prompts annotated with calibrated quality scores, enabling fine-grained quality prediction across model families; (2) a user-controlled routing mechanism with tolerance parameter $τ\in [0,1]$ that provides explicit control over quality-cost trade-offs; and (3) an extensible design using frozen encoders with model-specific adapters, reducing new model integration from days to hours. To rigorously train and evaluate IPR, we curate an industrial-level dataset IPRBench\footnote{IPRBench will be released upon legal approval.}, a comprehensive benchmark containing 1.5 million examples with response quality annotations across 11 LLM candidates. Deployed on a major cloud platform, IPR achieves 43.9\% cost reduction while maintaining quality parity with the strongest model in the Claude family and processes requests with sub-150ms latency. The deployed system and additional product details are publicly available at https://aws.amazon.com/bedrock/intelligent-prompt-routing/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。