为大模型结构化输出提供可靠风险控制，揭示何时能认证、何时不可能。
When Can Conformal Risk Control Certify LLM Outputs? Bounds, Impossibility, and Adaptation for Structured Generation
- 提出可验证的可行性判据，提前判断能否用置信度控制保证输出可靠性
- 在低方差场景下，改进的置信度界使可认证配置提升37%，数据少时更鲁棒
- 实测证明可减少风险超限，适合部署前评估与应对数据分布偏移
部署于命名实体识别、JSON提取、问答和分类等任务的大语言模型缺乏可靠的保证机制，传统启发式拒绝策略的误差达7.5%至12.5%。本文刻画了置信度风险控制（CRC）在结构化生成中可应用的边界：首先证明一个不可能性定理——当基础风险μ > α时，任何无分布假设的方法必须放弃至少((μ-α)/(1-α))比例的样本；由此导出一个闭式可行性检验，可预先判断是否适用CRC。其次，分析了霍夫丁、经验伯恩斯坦及基于投注的e-CRC三种置信界，在低方差/大数据条件下性能逐级提升：霍夫丁到伯恩斯坦带来最大增益（+37%可认证配置），而e-CRC在校准数据稀缺时仍可实现10%认证率（对比霍夫丁的0%）。最后，通过跨数据集偏移验证自适应共形推断（ACI），将风险目标违反率从71%降至21%，剩余失败恰好落在不可能性边界预测位置。在六个开源模型（3B–72B参数）、八个数据集、四类任务、六种非一致性评分下，硬性NER/QA/CLS任务在α=0.10时不可认证；放宽至α=0.30–0.40后可实现实用认证（47% NER，40% QA，60% CLS）。框架提供三步部署方案：先验可行性检查，再选置信界与评分，最后缓解分布偏移。
原文摘要 · Abstract (English)
Large language models (LLMs) deployed for structured generation (NER, JSON extraction, QA, and classification) lack formal reliability guarantees, and standard heuristic abstention policies miss user-specified risk targets by 7.5--12.5%. We characterize when conformal risk control (CRC) can certify structured LLM outputs and when it provably cannot. First, we prove an impossibility result: when the base risk (μ> α), any distribution-free method must abstain on at least ((μ-α)/(1-α)) examples, yielding a closed-form feasibility test: one can check whether CRC will work before running it. Second, we analyze a certification hierarchy across Hoeffding, empirical Bernstein, and a betting-based e-CRC bound, with strict gains in low-variance/large-sample regimes: the Hoeffding-to-Bernstein step delivers the largest gain (+37% certified configurations), while e-CRC adds value when calibration data is scarce (10% certification at 20% data versus 0% for Hoeffding). Third, we validate adaptive conformal inference (ACI) under cross-dataset shift, reducing risk-target violations from 71% to 21%, with residual failures concentrated exactly where the impossibility bound predicts. Across six open-weight models (3B--72B parameters), eight datasets, four tasks, and six nonconformity scores, hard NER/QA/CLS configurations are uncertifiable at (α= 0.10); relaxing to (α= 0.30--0.40) unlocks practical certification (47% NER, 40% QA, 60% CLS). The framework gives a three-step deployment recipe: check feasibility, select the bound and score, then mitigate shift.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。