arXiv:2510.10193cs.AI2025-10被引 6

提出SAFER框架,让大模型在开放问答中更安全地输出可信答案。

SAFER: Risk-Constrained Sample-then-Filter in Large Language Models

  • 先采样后过滤:分两阶段控制风险,确保答案集覆盖正确答案
  • 在有限采样下实现严格误覆盖率控制,失败则直接放弃回答
  • 适配多种任务和数据划分,适合高风险场景的可信推理

随着大语言模型(LLMs)在真实世界开放式问答等高风险场景中的广泛应用,确保其输出的可信性变得至关重要。现有选择性置信预测(SCP)方法通过构建具有受限误覆盖率的预测集提供统计保证,但先前工作假设所有问题的可接受答案均可通过有限采样获得,这在无固定解空间的开放式问答中不现实。为此,本文提出两阶段风险控制框架SAFER:首先,在保留校准集上,基于Clopper-Pearson精确方法,在用户指定的风险水平(即最大允许误覆盖率)下校准采样预算,若在最大采样上限内无法满足该风险水平,则放弃;否则,校准后的预算成为测试时的最低要求。随后,利用校准集中可获取正确答案的实例,应用置信风险控制方法确定统计有效的不确定性阈值,从而过滤每个测试样本候选集中的不可靠干扰项。此阶段引入额外风险水平,以控制正确答案被排除的风险。此外,我们证明SAFER兼容多种任务特定准入标准与校准-测试划分比例,凸显其鲁棒性与高数据效率。

原文摘要 · Abstract (English)

As large language models (LLMs) are increasingly deployed in risk-sensitive applications such as real-world open-ended question answering (QA), ensuring the trustworthiness of their outputs has become critical. Existing selective conformal prediction (SCP) methods provide statistical guarantees by constructing prediction sets with a constrained miscoverage rate for correct answers. However, prior works unrealistically assume that admissible answers for all instances can be obtained via finite sampling, even for open-ended QA scenarios that lack a fixed and finite solution space. To address this, we introduce a two-stage risk control framework comprising abstention-aware sampling and conformalized filtering (SAFER). Firstly, on a held-out calibration set, SAFER calibrates a sampling budget within the maximum sampling cap, using the Clopper-Pearson exact method at a user-desired risk level (i.e., the maximum allowable miscoverage rate of the sampling sets). If the risk level cannot be satisfied within the cap, we abstain; otherwise, the calibrated sampling budget becomes the minimum requirements at test time. Then, we employ calibration instances where correct answers are attainable under the calibrated budget and apply the conformal risk control method to determine a statistically valid uncertainty threshold, which filters unreliable distractors from the candidate set for each test data point. In this stage, SAFER introduces an additional risk level to guide the calculation of the threshold, thereby controlling the risk of correct answers being excluded. Furthermore, we show that SAFER is compatible with various task-specific admission criteria and calibration-test split ratios, highlighting its robustness and high data efficiency.

大模型安全风险控制开放问答

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。