提出可量化的人类监督机制，揭示过度审查反而降低安全性的反直觉现象。
Oversight Has a Capacity: Calibrating Agent Guards to a Subjective, Fatiguing Human
- 将监督器建模为带代价的择优分类，评估其在疲劳下的真实效能
- 实验显示人类对风险判断一致性仅0.52（中度），无绝对正确标签
- 发现监督过载会导致安全性能倒U型下降，适合关注系统可靠性的研究者
随着大模型代理开始执行不可逆操作（如命令行指令、文件修改、部署），标准安全模式是人工审批：高风险动作暂停并等待人类确认。我们指出，审批环节本身简单，真正的难点在于判断哪些动作应被阻止。当前领域存在两个错误假设：一是存在绝对意义上的‘风险’定义；二是人类审查者是完美且无限可用的。我们在一个包含125个对抗加权动作的手动标注数据集上发现：(i) 审查者对风险的判断一致性仅为0.52（Fleiss' kappa），表明不存在单一正确标签；(ii) 将监督器视为异构成本下的选择性分类，可测量其运行边界，在困难样本上无法安全自动决策；(iii) 当将审查者建模为内生变量（随升级负荷增加而疲劳），实际安全性呈现倒U型曲线——过多人工干预反而降低系统安全，最优监督策略应低于全量升级，这一设置也能抵御恶意攻击者利用疲劳绕过审查的洪水式攻击。因此，代理监督不仅是分类问题，更是资源分配问题：人类注意力有限，监督策略会消耗它。我们不主张新方法，而是整合已有成果（如疲劳感知学习延迟、成本敏感延迟、轨迹级防护、审查疲劳/洪水攻击）并开源实现，将‘我的监督器好不好？’从猜测变为可量化的曲线。倒U型关系与洪水攻击是建模结果，旨在推动后续人类实验。
原文摘要 · Abstract (English)
As LLM agents begin to take real, irreversible actions (shell commands, file edits, deploys), the standard safety pattern is a human-in-the-loop approval gate: risky actions pause and wait for a person. We argue the gate is the easy part; the hard part is the judgment - which actions to stop - which the field evaluates against two false assumptions: that there is a ground-truth notion of "risky," and that the human reviewer is a perfect, infinitely-available oracle. On a hand-labeled set of 125 adversarially-weighted agent actions we show that (i) reviewers only moderately agree on what is risky (Fleiss' kappa = 0.52), so there is no single correct label; (ii) framing the guard as selective classification under asymmetric cost makes its operating limits measurable, and on hard inputs the guard cannot safely auto-decide; and (iii) when the reviewer is modeled as endogenous (fatiguing as escalation load grows), realized safety becomes an inverted-U in the escalation rate: more human oversight can make a system less safe, and the safety-optimal guard escalates below full escalation - a setting a load-aware policy also uses to resist a flooding attack that slips a malicious action past a fatigued reviewer. Agent oversight, framed this way, is not only a classification problem but a resource-allocation one: human attention is finite, and the guard's escalation policy spends it. We claim none of these mechanisms as novel - fatigue-aware learning-to-defer (FALCON), cost-sensitive deferral under workload constraints (DeCCaF), trajectory-level guarding, and reviewer-fatigue/flooding attacks are all prior art we cite. Our contribution is an open-source agent-oversight system that operationalizes and measures them in the LLM-agent action-gating setting, turning "is my guard good?" from a guess into a curve. The inverted-U and the flooding attack are modeling results that motivate a human study.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。