878个安全技能中仅2.3%有完整说明，影响用户理解
Toward User Comprehension Supports for LLM Agent Skill Specifications
- 用规则编码分析技能说明中的四大理解线索
- 仅19%技能包含示例，2.3%完整覆盖四类线索
- 缺少示例使用户需查代码才能理解参数和输出
用户通常通过SKILL markdown格式的说明来理解与选择智能体技能。为保护用户，现有审计多关注恶意或不安全技能，但本文研究说明是否帮助用户形成对技能输入、输出及覆盖范围的合理预期。针对878个网络安全技能，我们采用规则编码方法，测量了四个理解锚点（操作基础、输出契约、边界披露、示例能力展示）的文本线索。结果显示，操作基础线索普遍存在，但仅有19.0%的说明包含任务示例、样本或预期结果，仅2.3%同时具备全部四类线索。通过对6个DNS/C2遥测技能的小规模分析发现，含示例的技能更易进行初步本地验证，而无示例技能通常需检查辅助代码才能恢复命令参数或输出字段。因此，我们认为智能体技能评估应将说明视为面向用户的能力建议，而非仅是可执行指令容器。
原文摘要 · Abstract (English)
Users often interpret and select agent skills through their SKILL markdown specifications. To protect users, existing audits mainly focus on malicious or unsafe skills. We study the complementary question of whether specifications help users form bounded expectations about what a skill consumes, produces, and covers. Across 878 cybersecurity skills, we used rule-based coding to measure textual cues for four comprehension anchors, namely operational basis, output contract, boundary disclosure, and example capability demonstration. Cues for operational basis were common, but only 19.0% of specifications exhibited cues for an example task, sample, or expected outcome, and only 2.3% exhibited cues for all four anchors. We further examined a small DNS/C2 telemetry subset (n$=$6) to illustrate why missing examples may matter. Examples appeared to make first local checks easier to construct, while no-example skills typically required helper code inspection to recover command arguments or output fields. We argue that agent-skill evaluation should treat specifications as user-facing capability disclosures, not merely as containers for executable instructions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。