用历史成功数据优化工具选择，零成本降低大模型推理延迟
Outcome-Aware Tool Selection for Semantic Routers: Latency-Constrained Learning Without LLM Inference
- 基于历史成功查询聚类，动态调整工具嵌入向量
- 在MetaTool上NDCG@5提升至0.940，ToolBench达0.848
- 无需额外参数或算力，适合高并发低延迟场景
大型语言模型推理网关中的语义路由器负责在关键请求路径中选择工具，每毫秒的延迟在数百万请求下会被放大。本文提出结果感知的工具选择（OATS），通过将工具嵌入向量向其历史成功查询的中心点插值，实现离线优化，部署时无额外参数、延迟或GPU开销。在MetaTool（199~工具，4,287~查询）上，NDCG@5从0.869提升至0.940；在ToolBench（2,413个API）上，从0.834提升至0.848。还评估了两种学习扩展：2,625参数的MLP重排序器和197K参数的对比适配器。当结果数据稀疏时，MLP性能下降或持平；对比适配器在MetaTool上取得0.931的NDCG@5。所有方法均在相同的30%保留测试集上评估。实际建议是优先采用零成本优化，仅在数据密度足够时才引入学习组件。所有机制均可在单数字毫秒级CPU预算内运行。
原文摘要 · Abstract (English)
Semantic routers in LLM inference gateways select tools in the critical request path, where every millisecond of added latency compounds across millions of requests. We propose Outcome-Aware Tool Selection (OATS), which interpolates tool embeddings toward the centroid of queries where they historically succeed -- an offline process that adds no parameters, latency, or GPU cost at serving time. On MetaTool (199~tools, 4,287~queries), this improves NDCG@5 from 0.869 to 0.940; on ToolBench (2,413~APIs), from 0.834 to 0.848. We also evaluate two learned extensions: a 2,625-parameter MLP re-ranker and a 197K-parameter contrastive adapter. The MLP re-ranker hurts or matches baseline when outcome data is sparse relative to the tool set; the contrastive adapter provides comparable gains on MetaTool (NDCG@5: 0.931). All methods are evaluated on the same held-out 30\% test split. The practical takeaway is to start with the zero-cost refinement and add learned components only when data density warrants it. All mechanisms run within single-digit millisecond CPU budgets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。