通过双向图对齐检测智能体技能声明与实现不一致，提升安全性和可靠性。
SkillConsist: Detecting Inconsistencies in Agent Skills via Bidirectional Graph Alignment
- 将声明与实现内容分离为行为图，用双向图对齐匹配对应关系。
- 在633个技能的基准上达到87.93% F1，比最优基线提升20.43个百分点。
- 适合关注LLM智能体安全、技能可靠性的研究人员和开发者。
智能体技能提供可复用的能力，其不一致性可能暴露未公开的危险行为或导致错误的技能选择。现有方法基于预定义类别或声明范围评估行为或安全属性图。最近的PL-HCL利用基于大模型的方法学习元数据、指令和资源间的一致性，但声明与实现的行为常混合于文本和代码中，且简洁声明可能对应多个关联实现步骤。本文提出SkillConsist以解决上述挑战：利用大模型将声明与实现内容分离为行为记录，静态分析补充实现记录，构建声明与实现行为图；从任一端的行为记录出发，双向图对齐在另一图中搜索候选子图，并沿行为关系扩展直至完整表达源端行为；图差异分析识别对齐子图间的冲突并输出检测结果。我们基于ClawHub中下载量最高的500个公共技能及133个技能注入包构建了包含633个技能的基准，其中319个存在不一致，314个一致，共442处定位标注。在该基准上，SkillConsist在包级检测中取得86.85%精度、89.03%召回率和87.93% F1，F1优于最佳基线20.43个百分点；在定位任务中达67.60%精度、58.14%召回率和62.52% F1。
原文摘要 · Abstract (English)
Agent Skills provide reusable capabilities to LLM agents. Agent Skill inconsistencies can expose undisclosed dangerous behavior or cause wrong Skill selection. Recent Agent Skill research has increasingly examined Agent Skill consistency detection. Existing methods evaluate behaviors or security-property graphs against predefined categories or declared scopes. More recently, PL-HCL uses an LLM-based model to learn consistency across metadata, instructions, and resources. However, declaration and implementation behavior can be mixed across text and code, and a concise declaration can correspond to multiple connected implementation steps. We present SkillConsist to address both challenges. An LLM separates declaration and implementation content into behavior records on the implementation and declaration sides, while static analysis supplements implementation records. These records form declaration and implementation behavior graphs, respectively. Starting from a behavior record on either side, bidirectional graph alignment searches the other graph for a candidate subgraph and expands it along behavior relations until it completely expresses the source-side behavior. Graph differencing identifies conflicts between aligned subgraphs and outputs the detection results. We construct a 633-Skill benchmark from ClawHub's 500 most-downloaded public Skills and 133 Skill-Inject packages. The benchmark contains 319 inconsistent and 314 consistent Skills and 442 localized inconsistency annotations. On this benchmark, SkillConsist achieves 86.85% precision, 89.03% recall, and 87.93% F1 for package-level detection, improving F1 over the best baseline by 20.43 percentage points. For localization, it achieves 67.60% precision, 58.14% recall, and 62.52% F1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。