arXiv:2609.04629cs.AIcs.LG2026-09

提出结构保持的工具代理准入机制,解决大模型决策中违反约束却仍能通过的问题。

SiLR: Structure-Preserving Admission and Process Reward for LLM Tool Agents

论文配图:SiLR: Structure-Preserving Admission and Process Reward for LLM Tool Agents
图 1 · 摘自论文原文
  • 采用分支级违规状态的乘积序判断准入,避免局部优化陷阱
  • 在24个场景中恢复成功率提升至100%,远超传统方法
  • 适合需要严格安全约束的复杂决策系统,如自动驾驶与能源管理

LLM工具代理的运行时准入通常被视为过滤器。在ReAct循环中,被拒绝的提议会立即触发新提议,因此准入机制实质上是对提议流的搜索操作,其准入准则决定了可达轨迹。本文研究了违规后恢复的准入策略,发现标量投影存在陷阱:聚合得分门禁会接纳局部改进的提议,并将轨迹锁定在平台区。相比之下,SiLR对每个提议进行影子执行,基于分支级违规状态(过载分支支持与各分支严重度)的乘积序进行准入判定。证明任何标量代理均无法正确表示该顺序,失败源于表征缺陷而非阈值调整。在挖掘的Gym-ANM场景中,SiLR在21个多动作任务中全部恢复成功,而终端门禁为0/21,最优标量门禁仅9/21,整体表现显著。该终端与结构化门禁的差异在三种模型族和CityLearn中均成立。由于准入依赖确定性模拟,LLM被排除在信任边界外:一种幅度重分配攻击可击败标量与仅支持基线,唯有完整分支谓词能有效防御。当两种约束同时激活时,所有测试标量投影均允许物理不安全动作;仅支持基线容忍度最高(42,410条中的63.2%),乘积序则为0。在最困难的双约束轨迹中,标量门禁仅通过不安全类别实现恢复。作为GRPO过程奖励复用,其性能优于计数投影,在所有挖掘场景中表现最佳,且无门禁策略优于未训练基础模型(0.844 vs. 0.778)。标量投影在设计点上均丢失违规几何结构;唯有完整乘积序具备结构性充分性。

原文摘要 · Abstract (English)

A runtime gate for an LLM tool agent is usually cast as a filter. In a ReAct loop a rejected proposal is followed by another at the same state, so the gate is a search operator over the proposal stream whose admission criterion shapes which trajectories are reachable. We study post-violation recovery admission, where progress must be admitted while the system is still in violation, and identify the scalar projection trap: an aggregate-score gate accepts a locally improving proposal and commits the trajectory to a plateau. SiLR instead shadow-executes each proposal and admits it under a product order over the branch-level violation state (overloaded-branch support and per-branch severity). We prove that no scalar surrogate is sound for this order, so the failure is representational, not a matter of threshold tuning. On mined Gym-ANM scenarios, SiLR recovers 21/21 multi-action episodes against 0/21 for terminal and 9/21 for the best scalar gate, significant across the full 24-scenario benchmark. The terminal-versus-structured dichotomy holds across three model families and in CityLearn. Because admission rests on deterministic simulation, the LLM lies outside the trust boundary: a magnitude-redistribution attack that defeats both scalar and support-only baselines is contained only by the full per-branch predicate. With two constraint families active, every tested scalar projection admits physically unsafe actions; support-only admits the largest fraction (63.2% of 42,410; product order 0). In the hardest dual-family traces, scalar gates recover only through that unsafe class. Reused as a GRPO process reward, it outperforms its count projection in every mined scenario and is the only tested reward whose ungated policy exceeds the untrained base (0.844 vs. 0.778). Scalar projection loses the violation geometry at both design points; only the full product order is structurally sufficient.

大模型决策安全约束工具代理结构保持

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。