arXiv:2605.18693cs.AI2026-05被引 14

评测大模型生成可复用技能的能力,解决当前评估缺失关键环节的问题。

SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents

论文配图:SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents
图 1 · 摘自论文原文
  • 统一协议下评估技能生成管道,分任务相关与无关两种模式。
  • 在代码库和长文档中生成技能,成功率差异显著,最高仅达47%。
  • 适合研究智能体技能自动生成的学者,尤其关注可复用性与鲁棒性。

随着大模型智能体依赖可复用技能,核心挑战已从能否使用已有技能,转向能否从文档与代码库中生成正确、可执行且可复用的技能。现有基准主要评估给定技能的有效性或智能体在原始上下文中的任务求解能力,但未将技能生成本身作为独立研究目标。本文提出SkillGenBench,一个在统一可控协议下评估技能生成流程的基准。在该基准中,生成器接收原始语料并输出标准化技能产物,再通过固定执行环境与统一评估流程进行测试。涵盖两类生成范式:任务条件生成(任务揭示后生成特定技能)与任务无关生成(预先构建可复用技能库)。同时覆盖两类过程来源:基于代码仓库(代码、配置、脚本分散)与基于长篇文档(需从文本中提炼程序与约束)。提供标准化任务说明、固定环境与以确定性执行检查为核心的评估协议,辅以诊断信号。跨多种生成方法与模型的实验显示性能差异显著,揭示可复用技能提炼难度高,并暴露出从代码库与长文档生成技能时的不同失败模式。SkillGenBench为智能体系统中技能生成这一独立研究问题提供了可复现的测试平台。

原文摘要 · Abstract (English)

As LLM agents are increasingly built around reusable skills, a central challenge is no longer only whether agents can use provided skills, but whether they can generate correct, reusable, and executable skills from repositories and documents. Existing benchmarks primarily evaluate the efficacy of given skills or the ability of agents to solve downstream tasks from raw context, but they do not isolate skill generation itself as the object of study. We introduce SkillGenBench, a benchmark for evaluating skill generation pipelines under a unified and controlled protocol. In SkillGenBench, a generator receives raw corpora and produces standardized skill artifacts, which are then executed under fixed harnesses and assessed with unified evaluation procedures. The benchmark covers two generation regimes: task-conditioned generation, where a task-specific skill is synthesized after the task is revealed, and task-agnostic generation, where a reusable skill library must be distilled before downstream tasks are known. It also spans two complementary procedural sources: repository-grounded instances, where procedures are distributed across code, configuration, and scripts, and document-grounded instances, where procedures and constraints must be distilled from long-form text. We provide standardized task specifications, pinned environments, and evaluation protocols centered on deterministic execution-based checks, supplemented by auxiliary signals for diagnosis. Experiments across a range of skill-generation methods and backbones show substantial performance variation, highlight the difficulty of reusable skill distillation, and reveal distinct failure modes in skill generation from software repositories versus long-form documents. SkillGenBench establishes a reproducible testbed for studying skill generation as an independent research problem in agent systems.

智能体技能生成基准评测大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。