让大模型技能执行更可靠,通过实时监控和动态优化确保任务成功。
SkillSentry: Reliable Skill Execution for LLM Agents via Runtime Assurance

- 用领域语言构建运行时指导,结合文档与历史经验初始化。
- 在15个技能上平均提升任务成功率24.1%,重复执行更稳定。
- 适合追求高可靠性、需持续执行复杂任务的大模型应用。
大模型代理通过多步推理和工具使用来执行复杂任务,依赖技能提供可复用的程序知识。然而,即使代理在引导下展示过完成任务的能力,仍可能因偏离技能流程或步骤执行错误而在相似任务或重复运行中失败,导致执行不稳定,限制实际可靠性。为此,我们提出SkillSentry——一种面向技能的运行时保障框架,基于新的领域特定语言(DSL)表示技能执行的运行时指导。SkillSentry通过整合从技能文档中提取的规范与历史成功和失败轨迹中挖掘的执行经验,初始化运行时指导;随后嵌入代理执行循环,实时监控并引导技能执行,并利用新收集的轨迹迭代优化指导。我们在两个大模型代理上评估了15个技能,每个代理搭配两个主干模型:Claude Code(Claude-Haiku-4.5 和 Claude-Opus-4.6),以及Codex(GPT-5.2 和 GPT-5.4)。结果表明,SkillSentry在所有技能上平均将任务成功率提升24.1%,同时显著降低重复运行间的波动性。
原文摘要 · Abstract (English)
LLM agents are increasingly equipped with skills to perform complex tasks through multi-step reasoning and tool use. Although skills provide reusable procedural knowledge, agents may still execute them unreliably. Even when an agent has demonstrated the capability to complete tasks under the guidance of a skill, it may fail to do so consistently across similar tasks or repeated runs due to deviations from the skill procedure or incorrect execution of individual steps. Such instability limits the practical reliability of LLM agents. To address this problem, we propose SkillSentry, a skill-oriented runtime assurance framework built upon a new domain-specific language (DSL) for representing runtime guidance for skill execution. SkillSentry initializes the runtime guidance by combining a skill specification extracted from the corresponding skill document with execution experience mined from historical successful and failed traces. It then wraps around the agent execution loop to monitor and guide skill execution under the current guidance, while iteratively refining the guidance using newly collected traces. We evaluate SkillSentry on 15 skills across two LLM agents, each paired with two backbone models, i.e., Claude Code with Claude-Haiku-4.5 and Claude-Opus-4.6, and Codex with GPT-5.2 and GPT-5.4. Our results show that SkillSentry improves the task success rate of LLM agents by 24.1% across skills, on average, while exhibiting lower variability across repeated runs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。