13亿参数小模型实现高效推理，适配边缘设备部署。
Xmodel-2.5: 1.3B Data-Efficient Reasoning SLM
- 用最大更新参数化训练，超参可直接迁移至大模型。
- 切换优化器使13项推理任务平均提升4.58%。
- 适合资源受限场景的智能代理核心，开源可用。
大型语言模型具备强大的推理与工具使用能力，但计算开销大，难以在边缘或成本敏感场景部署。我们提出 extbf{Xmodel-2.5}，一个13亿参数的小型语言模型，作为即插即用的代理核心。通过最大更新参数化（μP）训练，使得在2000万参数代理上调整的超参数可直接迁移至完整模型，即使在参数共享的 tie-word-embedding 架构下也有效。采用1.4万亿token的温启动-稳定-衰减训练课程，并发现：在衰减阶段从AdamW切换到Muon优化器，在保持其他超参不变的情况下，使13项推理任务平均性能提升4.58%，验证了早期使用AdamW保持稳定、后期使用Muon增强精度的有效性。采用FP8混合精度训练兼顾准确率与吞吐量。所有检查点、训练配方和评估代码均以Apache-2.0许可证开源。训练代码与评估框架见GitHub仓库。
原文摘要 · Abstract (English)
Large language models deliver strong reasoning and tool-use skills, yet their computational demands make them impractical for edge or cost-sensitive deployments. We present \textbf{Xmodel-2.5}, a 1.3-billion-parameter small language model designed as a \emph{drop-in agent core}. Training with maximal-update parameterization ($μ$P) allows hyper-parameters tuned on a 20M-parameter proxy to transfer directly to the full model, even under the parameter-tied \emph{tie-word-embedding} architecture. A 1.4T-token Warmup--Stable--Decay curriculum is used, and we further show that \textbf{switching from AdamW to Muon during the decay phase} improves the 13-task reasoning average by 4.58\,\% while keeping every other hyper-parameter fixed, verifying that early AdamW stability can be paired with late Muon sharpening for better downstream performance. FP8-mixed-precision training balances accuracy and throughput. All checkpoints, recipes, and evaluation code are released under the Apache-2.0 license.\footnote{https://huggingface.co/XiaoduoAILab/Xmodel-2.5 and https://huggingface.co/XiaoduoAILab/Xmodel-2.5-history (training checkpoints).} Training code and evaluation harness: https://github.com/XiaoduoAILab/Xmodel-2.5.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。