用校准不确定性实现大模型流水线路由，降低推理成本31%。
UCCI: Calibrated Uncertainty for Cost-Optimal LLM Cascade Routing
- 通过等单调回归将分词级不确定性转化为查询错误概率，实现精准校准。
- 在7.5万次实体识别任务中，成本降低31%，错误率保持0.91的微平均F1。
- 无需手动调参，适合生产环境部署，实测延迟与真实成本匹配。
LLM流水线和模型路由可通过将简单请求交给小模型、复杂请求升级到大模型来降低推理成本，但现有路由器多使用未经校准的置信度分数，且需针对每个任务手动调整阈值。本文提出UCCI，一种以校准为先的路由机制：通过等单调回归将分词级边际不确定性映射为每查询的错误概率，并基于约束性成本最小化选择升级阈值。在三个显式假设下，对校准得分的阈值策略为成本最优，且等单调校准在期望校准误差（ECE）上达到O(n^{-1/3})的样本复杂度。在4B与12B指令微调模型于H100 GPU上处理7.5万次命名实体识别任务的生产环境中，UCCI将推理成本降低31%（95%置信区间：[27%, 35%]），同时保持微平均F1为0.91，将ECE从0.12降至0.03。在相同运行点，UCCI优于熵阈值法、分割共形路由及类似FrugalGPT的可学习阈值方法。所有流水线结果均基于实际模型输出的端到端路由与真实测量的H100延迟，而非基于全局准确率或名义API价格的模拟路由。
原文摘要 · Abstract (English)
LLM cascades and model routing promise lower inference cost by sending easy queries to a small model and escalating hard ones to a large model, but most deployed routers use uncalibrated confidence scores and require per-workload threshold tuning. We present UCCI, a calibration-first router that maps token-level margin uncertainty to a per-query error probability via isotonic regression and selects the escalation threshold by constrained cost minimization. Under three explicit assumptions, threshold policies on the calibrated score are cost-optimal, and isotonic calibration achieves O(n^{-1/3}) sample complexity for expected calibration error (ECE). On a production named entity recognition workload of 75,000 queries served by 4B and 12B instruction-tuned LLMs on H100 GPUs, UCCI cuts inference cost by 31% (95% CI: [27%, 35%]) at micro-F1 = 0.91 while reducing ECE from 0.12 to 0.03. At the same operating point, UCCI beats entropy thresholding, split-conformal routing, and a FrugalGPT-style learned threshold. All cascade results use end-to-end routing on actual model outputs and measured H100 latency, not simulated routing from global accuracies or nominal API prices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。