arXiv:2608.12654cs.AIcs.CL2026-08

测试大模型在工作场景中是否该执行操作的判断能力,发现模型常误拦合规操作。

SteerBench-Work: A Benchmark for Agent Steering at Action Boundaries

论文配图:SteerBench-Work: A Benchmark for Agent Steering at Action Boundaries
图 1 · 摘自论文原文
  • 以真实事件为背景设计双向测试题,评估模型在操作边界上的决策能力
  • 28.1%合规操作被错误阻止,仅1.0%危险操作被错误放行,偏差严重
  • 高能力模型反而更爱拦截,证据反转题表现差,适合安全与合规研究者

长期运行的大语言模型代理通过工具执行任务,单步可发送邮件、合并代码或转账。决策选择即在操作边界前决定是否继续或暂停等待人工或政策审查。我们提出 SteerBench-Work,一个基于公开事件、双向设计的工作场景代理决策基准,涵盖开发运维、客户服务、金融、法律、医疗、人力资源和安全等领域。发布版本 v2026-05 包含 106 个真实事件锚定的场景,配对反向证据镜像与校准控制,标注中“继续”与“暂停”几乎均等。模型需根据拟执行动作及可用证据做出门控决策,评分依据是否正确跨过或守住边界。30 种模型条件下,错误几乎全向一边:28.1% 的合规、有证据支持的操作被错误拦截,1.0% 的不安全操作被错误允许。最难的是风险已解除的提交,即使已有签名或结构化证据清除风险触发,模型仍表现不佳;在著名事件的证据反转镜像中准确率仅为 63.8%,低于原事件的 98.5%。通用能力不等于决策校准:高能力模型常过度拒绝提交边界,更多推理可修复弱门控,但无法改善已校准的门控。

原文摘要 · Abstract (English)

Long-running LLM agents act through tools, and a single step can send an email, merge a pull request, or wire a payment. The steering decision is the pre-commit choice at that boundary: proceed, or hold for human or policy review. We introduce SteerBench-Work, an incident-anchored, bidirectional benchmark for that decision in workplace agents across developer operations, customer service, finance, legal, medical, HR, and security. Release v2026-05 contains 106 scenarios anchored in public incidents, paired evidence-reversed mirrors, and calibration controls, with labels split nearly evenly between proceed and hold so the two error directions get near-identical numbers of chances. A model sees the proposed action and the available evidence, returns a gate decision, and is scored on whether it crosses or holds the boundary correctly. Across 30 model conditions the failures run almost entirely in one direction: models wrongly hold authorized, evidence-cleared work on 28.1% of opportunities and wrongly allow unsafe work on 1.0%. The hardest cases are risk-resolved commits, where signed or structured evidence has already cleared a real risk trigger, and models score markedly worse on evidence-reversed mirrors of famous incidents (63.8%) than on the incidents themselves (98.5%). General capability is not the same as steering calibration: higher-capability models often over-refuse at the commit boundary, and more reasoning can repair a weak gate while leaving a calibrated one flat. The public leaderboard is at steerbench.com.

代理决策安全校准基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。