通过日志分析揭示大模型安全失效的动态过程，超越单一成败判断。
Beyond Attack Success Rate: Temporal Logit Observability for LLM Safety Failures
- 基于解码过程中的合规拒绝差值，构建可解释的二维诊断平面
- 相同攻击成功率下，不同失败路径在平面上位置明显分离
- 适用于安全评估与早期拦截，适合模型安全研究人员使用
攻击成功率（ASR）仅以生成结束时的“是/否”标签评估越狱攻击，无法反映失败过程的差异。两个产生同样有害输出的攻击可能经历完全不同路径，但ASR无法区分。本文提出时间日志可观测性（TLO），一种无需训练的诊断方法，通过观察解码过程中的合规-拒绝边界，将每种模型-攻击组合映射到一个校准的二维平面。该平面在ASR最不敏感的区域——即因不同原因成功的攻击之间——最具信息量。在四个对齐的大语言模型和三种越狱范式上，具有几乎相同ASR的攻击在平面上分布明显不同：同一模型可通过多种时间模式失败。其几何结构与隐藏状态上的拒绝方向探测结果高度一致，仅有一个模型暴露了固定词典方法的极限。基于TLO设计的简单早停规则，可使成功越狱减少超过一半，且对正常良性查询无误报。安全评估应报告失败发生的时间与方式，而不仅是是否发生。TLO首次实现了仅从日志中观测这两点。
原文摘要 · Abstract (English)
Attack Success Rate (ASR) evaluates each jailbreak with a single yes/no label at the end of generation, telling us whether a failure happened but not how it unfolded. Two attacks that produce equally harmful outputs may have followed completely different paths, and ASR cannot tell them apart. We make those hidden paths observable from logits alone. Temporal Logit Observability (TLO) is a training-free diagnostic that watches a compliance-refusal margin during decoding and places each model-attack condition on a calibrated 2D plane. By design, this plane is most informative exactly where ASR is least informative: among attacks that succeed for genuinely different reasons. Across four aligned LLMs and three jailbreak paradigms, attacks with nearly identical ASR land at clearly different points on the plane: the same model can fail through different temporal patterns. The geometry matches refusal-direction probes from hidden states on most conditions, with one model showing the limit of our fixed-lexicon approach. A simple early-stop rule derived from TLO cuts successful jailbreaks by more than half, without false alarms on plain benign queries. Safety evaluation should report when and how a failure unfolds, not only whether it occurred. TLO makes the first two observable from logits alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。