arXiv:2607.25018cs.LG2026-07

用置信集大小决定大模型推理分流,无需调参就能保证准确率。

Conformal Cascade: Distribution-Free Accuracy Guarantees for Multi-Tier LLM Inference

  • 以校准后的答案集合大小作为分流依据,小集合则接受结果
  • 在18个测试集上,多数场景下准确率高于传统方法,尤其在复杂推理任务中优势明显
  • 不需训练模型,仅需接口访问,适合部署在实际系统中

大型语言模型级联通过将简单问题交给小模型处理、复杂问题转给大模型来降低推理成本。现有级联依赖置信度阈值决策,但大模型置信分数常失准,阈值需针对不同模型对和领域分别调整,且无法提供准确率的严格保障。本文提出Conformal Cascade(CC),采用校准后预测集大小作为分流规则:当预测集收缩为单一答案时接受,否则延迟至更高级模型。该方法提供分布无关的有限样本准确率保证。通过每层的并集界,接受层级的预测集覆盖正确答案的概率不低于 $1 - Kα$;在选择保持条件下,可收紧至 $1 - α$。我们进一步将预期级联成本表征为 $α$ 和校准集接受率的显式函数。在涵盖科学、医学、常识和标准化考试的18个多项选择基准上,基于四个开源模型家族构建的双层级联中,CC在多数模型-基准组合上优于最优校准启发式方法,尤其在多数投票不可靠的推理密集型任务中提升显著;在较易任务上,绝大多数查询由小模型处理,准确率无损失。开放生成任务需额外聚类步骤,留待后续研究。方法无需模型训练,仅需黑盒API访问。

原文摘要 · Abstract (English)

Large language model (LLM) cascades reduce inference cost by routing easy queries to a small model and deferring hard queries to a larger one. Production cascades govern this deferral through a confidence threshold, but LLM confidence scores are miscalibrated, the threshold must be tuned per model pair and per domain, and no setting yields a formal bound on cascade accuracy. We introduce \textbf{Conformal Cascade} (CC), a multi-tier inference framework that uses conformal prediction set size as the deferral rule: accept when the calibrated set collapses to a single answer, defer otherwise. The procedure delivers a distribution-free, finite-sample accuracy guarantee. By a per-tier union bound, the prediction set at the accepting tier covers the correct answer with probability at least $1 - Kα$ for any user-specified $α$; under a selection-preservation condition (consistent with, but not strictly implied by, our marginal coverage results), the bound tightens to $1 - α$. We further characterise expected cascade cost as an explicit function of $α$ and the calibration-set acceptance rate. Across 18 multiple-choice benchmarks spanning science, medicine, commonsense, and standardized exams, evaluated on two-tier cascades drawn from four open-weight model families, CC strictly improves over the strongest calibration-tuned heuristic cascade on the majority of family--benchmark pairs, with the largest gains on reasoning-heavy benchmarks where majority vote is unreliable; on easier benchmarks the cascade commits the vast majority of queries to the small model at no accuracy cost. Extension to open-ended generation requires an answer-clustering step that we leave for future work. The method requires no model training and only black-box API access.

大模型推理置信度校准级联架构概率保证

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。