1200亿参数稀疏专家模型在单机8卡上端到端训练，靠可逆计算与状态保持实现扩展。
Reversible Foundations: Training a 120B Sparse MoE through State-Preserving Scaling
- 用可逆循环栈避免存储激活值，内存不随模型增大而上升。
- 120B模型仅需2.26亿适配器参数存优化器状态，比原方案少45倍。
- 全程单节点训练，支持多语言和代码能力，适合资源有限但想练大模型的团队。
本文报告在单个八卡节点上端到端训练一个百亿级稀疏专家混合模型的过程。LightningLM 0.1V 从一个小型稠密模型出发，经由50亿和90亿参数的专家混合模型逐步扩展至1200亿参数模型，包含460个路由专家，使用Top-12路由策略。每个更大模型均基于前一模型权重生长，活跃参数从初始17.8亿单调增长至120B时的59.3亿（约占总参数1186.7亿的5%）。整个演进过程在单节点完成，后期阶段上下文长度达8192，最终训练损失降至1.78。该研究聚焦三大原则：可逆性——通过可逆循环栈在反向传播中重建激活值，使激活内存恒定；状态保持扩展——每次扩展（稠密转稀疏、浅层转深层、少专家转多专家）均有可复现的规范与错误后果；单节点经济性——采用量化基础专家权重与训练低秩适配器（TQP），将优化器状态压缩至22.6亿参数，相比原方案减少约45倍。成果为完整可复现的模型家族，含分域保留损失验证，证明多语言（印地语系）、代码等能力被显式学习。模型、分词器与训练代码均已开源。
原文摘要 · Abstract (English)
This paper reports on training a hundred-billion-parameter sparse mixture of experts on a single eight-GPU node, end to end. LightningLM 0.1V is a recurrence-backbone language model family grown in four stages from a small dense seed, through a 5B and a 9B mixture of experts, to a 120B model with 460 routed experts under top-12 routing. Each larger model is grown from the trained weights of the smaller one; active parameters rise monotonically from 1.78B at the dense seed to 5.93B at 120B (about 5% of the 118.67B stored). The full lineage runs on single nodes, the larger stages at 8K context, reaching a released training loss of 1.78 at 120B scale. This is a systems and experience report. It is organized around three disciplines. Reversibility: a reversible recurrence stack reconstructs activations in the backward pass instead of storing them, holding activation memory flat as the model grows. State-preserving growth: each expansion (dense to MoE, shallow to deep, few experts to many) is given as a reproducible principle paired with the failure that results from getting it wrong; several failures are silent. Single-node economics: the 120B trains through TQP, a strategy of quantized base expert weights and trained low-rank adapters that carries optimizer state on 2.26B adapter parameters rather than 100B+ resident in routed experts, cutting expert-path optimizer state by a factor of ~45. What is new is the integration of known primitives, not any primitive in isolation: one grown lineage running end to end on a single node, documented at practitioner level, with per-domain held-out loss as evidence that targeted capabilities (multilingual Indic competence, code) were learned by construction. Model family, tokenizer, and training code are released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。