用公开技能包做预训练，提升智能体模型表现。
SPT: Skills as Pre-Training Data for Agentic Language Models
- 将公开多文件技能包作为预训练数据，通过因果语言建模学习工具语义与流程。
- 在多个模型规模和训练方案下，智能体任务性能持续优于通用或轨迹数据训练。
- 适合需要高效调用工具的智能体系统开发者，可降低数据构建成本。
智能体语言模型通常在后训练阶段使用工具调用日志和智能体轨迹进行训练，这类数据提供直接行为监督，但生成需依赖任务环境、执行与验证，覆盖广泛工具和任务代价高昂。公开技能包提供了另一种训练数据来源：它们编码了可复用的工具语义与工作流，但通常仅在推理时作为上下文使用。本文提出技能预训练（SPT），一种中期训练方法，将因果语言建模应用于SkillCorpus——一个包含公开多文件技能包的数据集，可选地与通用数据混合。为保留包内文件间的关系，引入参考插入（Reference Insert）策略，将支持文件放置在主指令中提及的位置附近。在多个模型规模和后训练方案下的实验表明，SPT在智能体性能上始终优于在通用或轨迹数据上的中期训练，同时基本保持通用性能。数据混合实验显示，将技能数据与通用退火语料结合有额外收益。结果表明，技能包是预训练智能体语言模型的宝贵数据源。
原文摘要 · Abstract (English)
Agentic (tool-using) language models are mainly trained on tool-call traces and agent trajectories during post-training. These data provide direct behavioral supervision, but producing them requires task environments, execution, and verification, making broad tool and task coverage expensive. Publicly available skills offer another source of training data: they encode reusable tool semantics and workflows but are typically used only as inference-time context. We introduce Skill Pre-Training (SPT), a mid-training method that applies causal language modeling to SkillCorpus, a collection of public multi-file skill packages, optionally mixed with general data. To preserve relations among files within each package, we also introduce Reference Insert, a reference-aware assembly strategy that places supporting files near their mentions in the primary instruction. Experiments across multiple model scales and post-training recipes show that SPT consistently improves agentic performance over mid-training on general or trajectory data, while largely preserving general performance. Data mixture experiments show additional benefits from combining skill data with general annealing corpora. These results indicate that skill packages are a valuable data source for pre-training agentic language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。