用数学方法精准决定何时该花大代价评估模型能力,提升多模型系统效率。
Pandora's AI Model Routing Box: Efficient Allocation with Costly Value Estimation
- 基于潘多拉盒子理论,设计动态评估策略,权衡评估成本与收益。
- 实验显示仅需少量调用昂贵评估器,即可达到全量评估的路由效果。
- 适合需要高效调度多个模型的AI系统开发者或研究者参考。
由多个模型、架构或推理设置组成的异构AI系统可通过将查询路由给最擅长且成本最低的专家来提升质量与效率。但路由需估计各专家的期望回报,而这种价值估计本身有代价:低成本估算(如基于嵌入的预测)快速但噪声大,高精度估算(如接入检索结果或部分推理轨迹的微调模型)则昂贵。本文将此权衡形式化为经典的潘多拉盒子问题,在高斯信号模型下,得到闭式信息价值表达式,可判断对每个专家及输入是否值得进一步细化评估。提出中心化策略Pandora's Router;扩展至去中心化场景,即每个专家自主决定是否投入自评以争取任务,称为Pandora's Bidder。在三个领域测试——标准多大模型基准、检索增强专家、具可变推理路径的大模型——结果表明,Pandora's Router在路由质量上媲美穷尽评估,但调用昂贵估算器的次数显著减少。在去中心化设置中,当竞争性估值准确时,信息价值推理提升了分配效率;但若估值噪声大,则可能使策略性专家获益,损害他人利益。
原文摘要 · Abstract (English)
Heterogeneous AI systems composed of multiple models, architectures, harnesses, or inference-time settings can improve quality and efficiency by routing queries to the specialist who can answer most effectively at the lowest cost. Routing requires estimating each specialist's expected return, but this value estimation has a cost. Cheap estimators (e.g., embedding-based predictors) are fast but noisy, while accurate estimators (e.g., fine-tuned models with access to retrieval results or partial reasoning traces) are expensive. We formalize this tradeoff as an instance of Pandora's Box, the classical problem of optimal search with costly inspection. Under a Gaussian signal model, the resulting policies have closed-form value-of-information expressions that determine, for each specialist and input, whether refining the value estimate is worth its cost. We call the centralized policy Pandora's Router. We extend this to a decentralized setting, Pandora's Bidder, where specialists independently decide whether to invest in self-assessment before accepting an offered price to claim a query. Experiments across three domains---a standard multi-LLM benchmark, retrieval-augmented specialists, and LLMs with variable inference-time reasoning---show that Pandora's Router matches the routing quality of exhaustive estimation, while querying the expensive estimator far less often. In the decentralized setting, value-of-information reasoning improves allocative efficiency when competing estimates are accurate; when competing estimates are noisy, however, it can increase the strategic specialist's utility at the expense of others.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。