让AI助手按上下文决定执行、询问或拒绝动作,更安全灵活。
SAFETY SENTRY: Context-Aware Human Intervention via EXECUTE-ASK-REFUSE Routing

- 三路路由机制:执行、询问、拒绝,精准判断每一步行为
- 单次解码完成判断,模型轻量且可适配不同风险偏好
- 相比主流方案提升安全召回率,同时控制误判率
LLM智能体通过工具调用与现实环境交互,一次误判可能造成不可逆损害。传统防护依赖守卫模型对动作进行安全/不安全二元标记,但该方式混淆了动作本身危害性与用户上下文适宜性,并在动作类别层面判断,导致频繁干扰,削弱自主性并使用户习惯忽略关键警告。本文提出将问题重构为针对每个动作实例的三类路由决策:{执行, 询问, 拒绝},并构建轻量级守卫模型Safety Sentry,其推理仅需一次解码调用。通过单一解码时刻阈值,同一检查点可适应不同部署场景的风险容忍度,无需重新训练。Safety Sentry在整体准确率和安全相关召回率上优于多款开源及前沿闭源基线模型,同时有效控制两类方向性错误率。
原文摘要 · Abstract (English)
LLM agents act on real-world environments through tool calls, and a single misjudged action can cause irreversible harm. The standard safeguard is a guard model that labels each proposed action as safe or unsafe, but this binary view conflates two distinct decisions: whether the action is harmful in itself, and whether it is appropriate given the user's context. It also operates at the granularity of action categories rather than individual instances, producing routine interruptions that erode autonomy and train users to wave through the most consequential alerts. We reframe the problem as a per-instance three-way routing decision over {EXECUTE, ASK, REFUSE} and instantiate it with Safety Sentry, a lightweight guard model whose inference reduces to a single decoding call. A single decoding-time threshold lets one fixed checkpoint be re-positioned across deployments of differing risk tolerance without retraining. Safety Sentry outperforms a broad set of open-weight and frontier closed-source baselines on overall accuracy and safety-related recall, while controlling both directional error rates simultaneously.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。