构建大规模技能检索基准，提升大模型智能体的精准技能匹配能力。
SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agents
- 构建含1.6万项技能的结构化检索库，支持训练与评估分离。
- 微调后模型在10个召回率上提升12.9点，显著优于现有方法。
- 适合研究智能体系统、信息检索及大模型应用的开发者使用。
随着大模型智能体拥有大量可复用技能，如何为用户请求精准选择合适技能已成为关键系统挑战。在小型技能库中可通过名称直接调用，但随着技能生态规模扩大且受限于上下文和延迟预算，该方式失效。尽管重要，技能检索仍缺乏充分研究，现有基准有限，对真实技能库中的检索行为理解不足。为此，我们提出SkillRet，一个大规模技能检索基准。该数据集包含16,129个公开智能体技能，采用结构化语义标签与两级分类体系（6大类、18小类），提供63,259条训练样本和4,392个评估查询，且训练与测试技能池互不重叠，支持基准评测与检索训练。实验表明，现有检索模型在真实大规模场景下表现仍不理想；在SkillRet上进行任务特定微调后，模型在NDCG@10指标上比最强先验检索器提升12.9点，比最强现成检索器提升16.2点。分析显示，提升源于微调模型能更好聚焦长而嘈杂查询中的微弱相关信号。本工作确立了SkillRet作为大模型智能体检索研究的重要基准。我们已公开数据集（https://huggingface.co/datasets/ThakiCloud/SKILLRET）、代码（https://github.com/ThakiCloud/SKILLRET）及模型权重（0.6B：https://huggingface.co/ThakiCloud/SKILLRET-Embedding-0.6B；8B：https://huggingface.co/ThakiCloud/SKILLRET-Embedding-8B）。
原文摘要 · Abstract (English)
As LLM agents are increasingly deployed with large libraries of reusable skills, selecting the right skill for a user request has become a critical systems challenge. In small libraries, users may invoke skills explicitly by name, but this assumption breaks down as skill ecosystems grow under tight context and latency budgets. Despite its practical importance, skill retrieval remains underexplored, with limited benchmarks and little understanding of retrieval behavior on realistic skill libraries. To address this gap, we introduce SkillRet, a large-scale benchmark for skill retrieval in LLM agents. SkillRet contains 16,129 public agent skills, organized with structured semantic tags and a two-level taxonomy spanning 6 major categories and 18 sub-categories. It provides 63,259 training samples and 4,392 evaluation queries with disjoint skill pools, enabling both benchmarking and retrieval-oriented training. Across a diverse set of retrievers, we find that skill retrieval remains far from solved: off-the-shelf models struggle on realistic large-scale skill libraries, and prior skill-retrieval models still leave substantial headroom. Task-specific fine-tuning on SkillRet improves NDCG@10 by 12.9 points over the strongest prior retriever and by 16.2 points over the strongest off-the-shelf retriever. Our analysis further suggests that these gains arise because fine-tuned models better focus on the small skill-relevant signals within long and noisy queries. These results establish SkillRet as a strong benchmark and foundation for future research on retrieval in large-scale agent systems. We publicly release the benchmark (https://huggingface.co/datasets/ThakiCloud/SKILLRET), code (https://github.com/ThakiCloud/SKILLRET), and model checkpoints (0.6B: https://huggingface.co/ThakiCloud/SKILLRET-Embedding-0.6B; 8B: https://huggingface.co/ThakiCloud/SKILLRET-Embedding-8B).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。