arXiv:2608.10621cs.LG2026-08

用概率模型提前预判大模型输出是否危险,提升安全防护精度。

ProbGuard: Calibrated Safety Risk Estimation from LLM Output Distributions

论文配图:ProbGuard: Calibrated Safety Risk Estimation from LLM Output Distributions
图 1 · 摘自论文原文
  • 基于大模型生成过程的分布信号,构建概率化安全评估框架。
  • 在9组测试中平均降低79.6%的Brier分数与71.9%的ECE误差。
  • 仅看前10步解码结果,就能将越狱攻击成功率压至1%以下。

近期大语言模型(LLM)安全研究普遍采用护栏机制识别不安全输出。现有护栏通常将安全评估视为确定性分类任务,将离散的词元序列映射为离散的安全标签。然而,这一范式存在两大局限:首先,安全评估本质上是不确定的,尤其在生成早期阶段;其次,仅依赖离散词元序列会丢失嵌入在LLM输出分布中的丰富概率信息。为此,我们提出首个完全概率化的、架构无关的护栏 extsc{ProbGuard},利用大模型早期输出分布信号估计并校准安全概率,从而实现对不安全生成的早期终止。具体而言,给定大模型生成前缀的概率分布,我们将安全风险定义为后续生成动态中的不安全概率,并通过蒙特卡洛采样进行估计。通过对分布信号和校准后安全风险进行后训练, extsc{ProbGuard} 在全部九个模型-数据集组合设置下均达到最佳校准性能,相比最优基线,平均Brier得分和ECE分别降低79.6%和71.9%。此外, extsc{ProbGuard} 在观察到大模型仅前十个解码步骤的输出分布后,即可将六种代表性越狱攻击的成功率控制在1%以内。

原文摘要 · Abstract (English)

Recent research on Large Language Model (LLM) safety has widely adopted guardrails to identify unsafe LLM outputs. Existing guardrails typically formulate safety assessment as a deterministic classification task, mapping a discrete token sequence to a discrete safety label. However, this paradigm has two limitations: First, safety assessment is inherently an uncertain problem, particularly during the early generation state. Second, relying solely on discrete token sequences discards the rich probabilistic information embedded in the LLM output distribution. To address these limitations, we propose the first completely probabilistic architecture-agnostic guardrail \textsc{ProbGuard} to leverage the LLM early output distributional signals for estimating and calibrating the safety probability, thereby enabling early stopping of unsafe ongoing outputs. Specifically, given an LLM's generated prefix distribution, we formulate the safety risk as the unsafe probability of its continued generation dynamics and estimate this risk by Monte-Carlo sampling. Through post-training on the distributional signals and calibrated safety risk, \textsc{ProbGuard} achieves the best calibration performance across all nine model--dataset combination settings, reducing the average Brier score and ECE by 79.6\% and 71.9\%, respectively, over the best baseline. \textsc{ProbGuard} further limits the attack success rate to at most 1\% across six representative jailbreak attacks after observing the LLM early output distributions from only the first ten decoding steps.

安全防护概率建模大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。