研究技能组织方式如何影响智能体运行行为,发现结构设计能显著改变知识调用模式。
SkillJuror: Measuring How Agent Skill Organization Changes Runtime Behavior

- 采用渐进式披露机制,按需加载支持资源,替代扁平化直接调用
- 轨迹中触及的资源数从1.18升至3.85,有效调用事件从1.33增至3.92
- 在82项任务中提升4.1%通过率,适合依赖资源指导的任务
智能体技能在推理时为大语言模型提供程序性知识,但现有基准很少区分技能内容与组织方式。本文通过渐进式披露(Progressive Disclosure)研究这一差异:由简洁根文件按需指向支持资源,对比标准化扁平基线。提出SkillJuror框架,通过语义控制变体、匹配多轮评估与轨迹证据,在固定任务知识前提下评估技能编写范式。在82项任务的SkillsBench测试中,渐进式披露使每条轨迹触及的技能资源数从1.18增至3.85,有效调用事件从1.33增至3.92。相比基线,在410组匹配试验中额外获得17次验证通过(+4.1%)。该优势具任务依赖性:当支持资源用于实现、检查或修复时效果明显;但在依赖精确输出格式、数值阈值或长流程生成任务中表现较弱。结果表明,技能组织不仅是呈现方式,更会影响智能体搜索与应用知识的方式,而最终成效取决于资源是否可操作。代码已开源。
原文摘要 · Abstract (English)
Agent Skills augment large language model (LLM) agents with procedural knowledge at inference time, but current benchmarks rarely distinguish what a Skill says from how it is organized. We study this distinction through Progressive Disclosure, where a concise root file points agents to supporting resources on demand, and compare it with a normalized flat baseline. We present SkillJuror, a framework for evaluating Skill writing paradigms through semantically controlled variants, matched multi-trial evaluations, and trajectory evidence while holding task knowledge fixed. In an 82-task SkillsBench study, Progressive Disclosure changes runtime behavior before aggregate outcomes: distinct Skill resources touched per trajectory rise from 1.18 to 3.85, and effective uptake events rise from 1.33 to 3.92. It also yields 17 additional verifier-passing trials out of 410 matched trials (+4.1%) over the normalized flat baseline. The benefit is task-dependent. Progressive Disclosure helps when supporting resources guide implementation, checking, or repair, but is weaker when success hinges on exact output conventions, numerical thresholds, or long artifact-generation pipelines. These results show that Skill organization is not mere presentation: it can change how agents search and apply procedural knowledge, while outcome gains depend on whether the exposed resources are actionable for the task. Code is available at https://github.com/zhiyuchen-ai/skill-juror.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。