将Orion-14B模型转换为Llama架构，性能接近原模型且支持长文本编程。
Llamion Technical Report
- 用三步法完成跨架构转换：参数映射、层归一化优化初始化、冻结教师蒸馏。
- 仅用1230万条数据训练四天，在多个评测中复现Orion性能，长文本能力完整保留。
- 适合需要开源、兼容Hugging Face的开发者快速部署高性能中文语言模型。
我们发布Llamion，一个由Orion-14B通过标准化转换得到的140亿参数开源语言模型家族。转换采用高效知识保留方案KEPT：(i) 对未变化模块使用标准参数映射（NPM）；(ii) 提出无需训练的层归一化到RMSNorm初始化方法OPM，证明在权重衰减诱导的近零均值激活下最优；(iii) 采用等尺寸冻结教师蒸馏（XKD），使转换后模型输出与源模型在合理输入分布下对齐。Llamion在H6、MT-Bench和KoMMLU上复现Orion表现，仅需约1230万条训练样本，在单张A100上耗时四天；其中Llamion-Base在KoMMLU上达66.87%，超过开放榜次优项超7.0个百分点。原训练语料中缺失的能力（如Python编程、20万词上下文处理）在转换后完全保留。我们发布三个检查点（Base、Chat、LongChat），可在Hugging Face Transformers库中无需trust_remote_code加载。
原文摘要 · Abstract (English)
We release Llamion, a family of 14B-parameter open-weight language models obtained by transforming Orion-14B into the standardized Llama-family architecture. The transformation is performed by Efficient Knowledge Preservation for Transformation (KEPT), a recipe that combines (i) Normal Parameter Mapping (NPM) for unchanged modules, (ii) Optimized Parameter Mapping (OPM), a training-free LayerNorm-to-RMSNorm initialization we prove optimal under the near-zero-mean activation regime induced by weight decay, and (iii) Cross-architecture Knowledge Distillation (XKD), an equal-size frozen-teacher distillation that aligns the converted model's outputs with the source model's on any reasonable input distribution. Llamion recovers Orion's behaviour on H6, MT-Bench, and KoMMLU with only ~123M tokens on a single A100 in four days; Llamion-Base reaches 66.87% on KoMMLU, exceeding the next-best entry of the Open Ko LLM Leaderboard by >7.0 absolute points at submission time. Capabilities entirely absent from the transfer corpus (Python programming and 200K-token context handling) survive the architectural transition intact. We release three checkpoints (Base, Chat, LongChat) that load with trust_remote_code=False in the Hugging Face Transformers library.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。