arXiv:2605.08828cs.AI2026-05被引 2

测试大模型代理对环境证据的过度信任,发现其常因错误信息出错。

When Agents Overtrust Environmental Evidence: An Extensible Agentic Framework for Benchmarking Evidence-Grounding Defects in LLM Agents

  • 构建框架自动生成带过时/恶意信息的任务场景
  • 55个测试案例中多数代理出现判断失误
  • 适合关注AI安全与可靠性研究者阅读

大型语言模型代理越来越多地依赖面向环境的支撑系统(如文件、网页、API、日志),这些观察结果影响工具使用、状态追踪和动作序列,但其可靠性与权威性常不确定。环境锚定因此成为涉及上下文接纳、证据溯源、新鲜度检查、验证策略、动作控制和模型推理的系统级问题。现有代理基准主要评估任务能力或特定攻击(如提示注入、记忆污染),但未充分涵盖根本性可靠性问题:当观测信息过时、错误或恶意时,代理是否仍能保持对真实环境状态的锚定?我们提出EnvTrustBench,一个用于评估此类失效模式的可扩展代理框架。我们将证据锚定缺陷(EGD)定义为:代理在未将环境声明与当前可用证据比对的情况下,将其视为行动充分证据,导致在真实环境状态下产生任务错误的错误路径。给定任务场景,EnvTrustBench生成工作空间、环境、代理目标和验证逻辑,执行被测代理,记录其动作-观测轨迹与最终状态,并通过验证逻辑生成判断结果。使用6个LLM主干和5种常用支撑结构,我们在11个任务场景中生成55个案例,每个场景经五轮反馈引导生成迭代。结果显示,EGD在各类操作流程中持续出现,凸显环境锚定是代理可靠性的核心问题,具有重要安全意义。

原文摘要 · Abstract (English)

Large language model agents increasingly operate through environment-facing scaffolds that expose files, web pages, APIs, and logs. These observations influence tool use, state tracking, and action sequencing, yet their reliability and authority are often uncertain. Environmental grounding is therefore a systems-level problem involving context admission, evidence provenance, freshness checking, verification policy, action gating, and model reasoning. Existing agent benchmarks mainly evaluate task capability or specific attacks such as prompt injection and memory poisoning, but they under-specify a fundamental reliability question: whether agents remain grounded in the true environment state when observations are stale, incorrect, or malicious. We introduce EnvTrustBench, an agentic framework for benchmarking this failure mode. We define an evidence-grounding defect (EGD) as a behavioral failure in which an agent treats an environment-facing claim as sufficient evidence for action without resolving it against available current evidence, leading to a task-incorrect false path under the true environment state. Given a task scenario, EnvTrustBench generates the workspace, environment, agent-facing objective, and validation oracle, executes the evaluated agent, records its action-observation trajectory and final state, and applies the oracle to produce a verdict. Using 6 LLM backbones and 5 widely used scaffolds, we evaluate 55 generated cases across 11 task scenarios, with each scenario expanded through five feedback-guided generation iterations. Results show that EGDs consistently emerge across operational workflows, highlighting environmental grounding as a core agent reliability problem with important security implications.

Agent安全大模型可靠性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。