构建可扩展的智能体技能管理体系,提升复杂任务执行效率。
Organizing, Orchestrating, and Benchmarking Agent Skills at Ecosystem Scale
- 用能力树组织技能,支持高效检索与发现。
- 通过有向无环图编排多技能,显著优于直接调用。
- 在200至20万技能规模下验证了结构化组合的有效性。
随着Claude智能体技能的快速增多,如何有效利用、管理与扩展技能生态成为关键问题。本文提出AgentSkillOS,首个面向技能选择、编排与生态级管理的系统性框架。该框架包含两个阶段:(i) 管理技能,通过节点级递归分类将技能组织为能力树,实现高效发现;(ii) 解决任务,基于有向无环图(DAG)管道检索、编排并执行多个技能。为评估智能体调用技能的能力,我们构建了一个涵盖5个类别(数据计算、文档生成、运动视频、视觉设计、网络交互)共30个富产物任务的基准测试集。使用基于大语言模型的成对评估方法衡量输出质量,并通过Bradley-Terry模型聚合得分。在200到20万技能规模的实验中,树形检索能有效逼近理想技能选择,而DAG编排显著优于原生扁平调用,即使使用相同的技能集合。结果表明,结构化组合是释放技能潜力的关键。代码已开源:https://github.com/ynulihao/AgentSkillOS。
原文摘要 · Abstract (English)
The rapid proliferation of Claude agent skills has raised the central question of how to effectively leverage, manage, and scale the agent skill ecosystem. In this paper, we propose AgentSkillOS, the first principled framework for skill selection, orchestration, and ecosystem-level management. AgentSkillOS comprises two stages: (i) Manage Skills, which organizes skills into a capability tree via node-level recursive categorization for efficient discovery; and (ii) Solve Tasks, which retrieves, orchestrates, and executes multiple skills through DAG-based pipelines. To evaluate the agent's ability to invoke skills, we construct a benchmark of 30 artifact-rich tasks across five categories: data computation, document creation, motion video, visual design, and web interaction. We assess the quality of task outputs using LLM-based pairwise evaluation, and the results are aggregated via a Bradley-Terry model to produce unified quality scores. Experiments across three skill ecosystem scales (200 to 200K skills) show that tree-based retrieval effectively approximates oracle skill selection, and that DAG-based orchestration substantially outperforms native flat invocation even when given the identical skill set. Our findings confirm that structured composition is the key to unlocking skill potential. Our GitHub repository is available at:https://github.com/ynulihao/AgentSkillOS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。