arXiv:2607.01236cs.CLcs.AI2026-07被引 2

通过追溯证据检测大模型代理的意图偏离,显著降低错误执行风险。

Safeguarding LLM Agents from Misalignment through Provenance Analysis

论文配图:Safeguarding LLM Agents from Misalignment through Provenance Analysis
图 1 · 摘自论文原文
  • 基于可追溯证据判断工具调用是否合理,构建三阶段检测流程。
  • 在两个基准测试中,误判率从44.3%降至2.1%,干预减少近一半。
  • 适合关注大模型安全、自动化决策可信性的研究者与开发者。

随着大语言模型代理获得更多强大工具的访问权限,确保其行为与用户意图一致变得至关重要。当代理提出的动作偏离用户意图(即‘意图偏离’)时,可能造成难以挽回的损害。现有运行时防护依赖‘大模型作为裁判’模式,缺乏系统性推理框架,常导致判断不一致且难以审计。受溯源分析启发,我们提出一个概念框架,将意图偏离检测形式化为判断提议的工具调用是否有可追溯的上下文证据支持。基于此框架,我们构建了ProvenanceGuard——一个多阶段流水线,在执行前分析三种类型的意图偏离,并仅允许对齐的动作通过。我们在AgentSafetyBench和WorkBench上评估了该方法,覆盖11个基础大模型。相比‘大模型作为裁判’基线,ProvenanceGuard将Agent-SafetyBench上的误判率从44.3%降至2.1%,在WorkBench上从32.4%降至18.7%;同时将任务成功轨迹上的干预率从31.2%降至13.0%,且对对齐轨迹的非必要干预无统计显著增加。

原文摘要 · Abstract (English)

As LLM agents gain increasing access to powerful tools, ensuring that their actions align with the user's intent becomes critical. When an agent's proposed action deviates from that intent---a phenomenon called misalignment---it may cause harm that is difficult to undo. Existing runtime guardrails rely on an LLM-as-a-judge paradigm that lacks a systematic framework for reasoning about alignment, often producing inconsistent or difficult-to-audit judgments. Motivated by provenance analysis, we propose a conceptual framework that formalizes misalignment detection as determining whether a proposed tool call is supported by traceable evidence in the agent's context. Based on this framework, we build ProvenanceGuard, a multi-stage pipeline that analyzes the agent's action for three types of misalignment before its execution and only allows aligned actions. We evaluated ProvenanceGuard on AgentSafetyBench and WorkBench, across 11 backbone LLMs. Compared to the LLM-as-a-judge baseline, ProvenanceGuard reduces error rate on misaligned traces from 44.3% to 2.1% on Agent-SafetyBench and from 32.4% to 18.7% on WorkBench, while reducing interventions on task-successful traces from 31.2% to 13.0% and introducing no statistically significant increase in unnecessary interventions on aligned traces. These results demonstrate that structured, provenance-based reasoning provides an effective and practical foundation for safeguarding LLM agents from misalignment.

大模型安全意图对齐溯源分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。