提出PASC方法，实现多阶段NLP系统联合置信度保障，精度更高且计算更快。
PASC: Pipeline-Aware Conformal Prediction with Joint Coverage Guarantees for Multi-Stage NLP and LLM Pipelines
- 将多阶段预测的联合覆盖问题转化为单一标量最大非符合度计算
- 在三阶段流水线中实现96.4%端到端覆盖，优于贝叶斯修正法和独立校准
- 适用于大模型与智能体系统，适合需要可靠不确定性评估的研究者
现代NLP与大模型系统多为流水线结构：命名实体识别（NER）→实体消歧（NED）→实体分类，检索增强生成（检索器→阅读器），以及规划→工具→批评的代理链。错误在各阶段累积，但现有不确定性量化方法要么独立校准每阶段（无联合覆盖），要么使用贝叶斯联合界（有联合覆盖但过于保守）。本文提出PASC（Pipeline-Aware Split Conformal），将多阶段联合覆盖问题简化为对联合最大非符合度的单标量共形预测问题。PASC提供有限样本、无需分布假设的保证：所有K个阶段同时被覆盖的概率至少为1−α，且近乎紧致，仅差1/(n+1)因子。在CoNLL-2003上的三阶段流水线实验中，PASC实现96.4%端到端覆盖，高于贝叶斯修正法的93.4%和独立共形预测的86.5%，平均预测集大小相同（1.083）。在WNUT-17推文和WikiNEuRal维基数据的分布偏移下，PASC仍保持目标覆盖率，而独立共形预测降至59%。PASC仅需一次分位数计算，运行速度比贝叶斯修正快1.7倍，可扩展至K=6阶段，而独立共形预测端到端覆盖率跌至0.53。该方法直接适用于复合大模型系统与代理流水线。
原文摘要 · Abstract (English)
Modern NLP and LLM systems are pipelines: named entity recognition (NER) -> entity disambiguation (NED) -> entity typing, retrieval-augmented generation (retriever -> reader), and agentic chains of planner -> tool -> critic. Errors compound across stages, but existing uncertainty quantification methods either calibrate each stage independently (no joint coverage) or apply a Bonferroni union bound (joint coverage, but conservative). We present PASC (Pipeline-Aware Split Conformal), which reduces multi-stage joint coverage to a single scalar conformal prediction problem on the joint maximum nonconformity score. PASC provides a finite-sample distribution-free guarantee that all K stages are simultaneously covered with probability at least 1 - alpha, and is nearly tight up to a 1/(n+1) factor. On a three-stage NER -> NED -> entity-typing pipeline over CoNLL-2003, PASC achieves 96.4% end-to-end coverage versus 93.4% for Bonferroni and 86.5% for independent CP, at identical average prediction set size (1.083). Under distribution shift to WNUT-17 Twitter and WikiNEuRal Wikipedia data, PASC empirically maintains the target coverage in the tested shift settings while independent CP collapses to 59%. PASC requires a single quantile computation, runs 1.7x faster than Bonferroni, and scales to K = 6 stages where independent CP drops to 0.53 end-to-end coverage. The same joint-maximum-score reduction applies directly to compound LLM systems and agent pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。