让大模型在不同群体中都可靠，避免某些群体被错误暴露。
Hierarchical Group-Conditional Conformal Risk Control for Selective Prediction in Language Models
- 构建分层群体条件的风险控制框架，按群体层级应用阈值。
- 实测零违规率，在多个数据集上保持严格风险约束。
- 适合关注公平性与可靠性、无需重训练的部署场景。
大语言模型服务于具有领域、主题难度和语言风格差异的异质群体。符合性风险控制（CRC）可为选择性预测提供严格的总体风险保障，但不保证各子群体的风险控制：模型可能满足整体预算，却系统性地使某些子群体面临更高出错风险。在群体构成轻微变化下，标准CRC在高达47%的试验中违反预算。本文提出HG-CRC（分层群体条件符合性风险控制），一种后处理校准框架，可同时在用户定义的群体层次结构所有节点上施加风险保障。其采用节点间的邦弗朗尼校正，并使用叶节点优先策略：优先采用最具体的适用阈值，若细粒度阈值不确定或拒绝样本，则退回到更粗粒度节点。仅需一个独立校准集，无需重新训练。我们在三个模型（Qwen3-4B、Llama-3.1-8B-Instruct、Gemma-3-4B）和两个基准（ARC Challenge、MMLU-Pro）上评估了八种配置，涵盖独立同分布泛化、异质性、混合/领域/提示/难度偏移、标签噪声及量化等。主要结果：对高精度模型（Qwen3-4B、Llama-3.1-8B），HG-CRC在ARC Challenge上实现0%违规率和WGER=0；在500次自助采样中，这些零为经验上限（真实率不超过0.6%）。结果具基准依赖性：在MMLU-Pro上，这些模型要么完全拒绝，要么（Llama）维持WGER=0.014。Gemma-3-4B在此处校准不佳，但通过增加拒答实现平稳退化。与全局CRC相比，参与成本上升22至37个百分点。消融实验显示，层级深度对预算清空至关重要：移除难度层级后，违规率回升至约11%。邦弗朗尼校正是理论保证所必需，尽管其实际影响仅在节点较多时显著。
原文摘要 · Abstract (English)
Large language models serve heterogeneous populations structured by domain, topic difficulty, and linguistic style. Conformal risk control (CRC) gives rigorous marginal risk guarantees for selective prediction with abstention, but marginal guarantees do not imply per-group ones: a model can meet the population budget while systematically over-exposing subgroups to errors. Under mild shift in group composition, standard CRC violates the budget in up to 47% of trials. We propose HG-CRC (Hierarchical Group-Conditional CRC), a post-hoc calibration framework enforcing simultaneous risk guarantees across all nodes of a user-defined group hierarchy. It applies a Bonferroni correction over nodes and a leaf-first policy that uses the most specific applicable threshold, falling back to coarser nodes when a finer one is uncertified or rejects the example. It needs only a held-out calibration set, with no retraining. We evaluate on three models (Qwen3-4B, Llama-3.1-8B-Instruct, Gemma-3-4B) and two benchmarks (ARC Challenge, MMLU-Pro) across eight configurations probing IID generalization, heterogeneity, mixture/domain/prompt/difficulty shift, label noise, and quantization. Main result: HG-CRC reaches an empirical 0% violation rate and WGER=0 on ARC Challenge for high-accuracy models (Qwen3-4B, Llama-3.1-8B). At 500 bootstrap trials these zeros are empirical upper bounds (true rate up to 0.6%), not certified. Results are benchmark-specific: on MMLU-Pro these models abstain entirely or (Llama) retain WGER=0.014. Gemma-3-4B, poorly calibrated here, degrades gracefully by abstaining. Participation cost vs. global CRC is 22 to 37 points. Ablations show hierarchical depth clears the budget: removing difficulty level returns violations to about 11%. Bonferroni is needed for the theoretical guarantee, though its empirical effect matters only with many nodes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。