arXiv:2507.07129cs.LGcs.CL2025-07被引 1

在参数总量不变下,通过模块化叠加新层实现模型持续扩展。

Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate

  • 仅训练新增层与语言模型头,保持活跃参数量稳定增长。
  • 16层模型在固定接口下达到28.92% MMLU得分,验证长期可扩展性。
  • 适合资源受限场景下的渐进式模型演进,如边缘部署或低频更新。

我们研究了一种约束训练范式:解码器仅变压器在令牌接口固定、先前训练的密集块不重新开启、活跃可训练参数集随深度增长保持近似恒定的条件下进行训练。从浅层模型开始,逐步堆叠新模块,仅训练最新模块和语言模型头;可选的LoRA阶段在相同活跃参数预算下提供有限全局调整。在通用协议的9层冻结字符子结构研究中,构造的冻结字符模型使用105.0M活跃可训练参数,相比接口匹配的单体冻结基线(180.5M)和全可训练单体基线(247.6M)显著更少。进一步考虑极端固定接口:每个令牌仅由冻结的16维二进制令牌ID编码确定性提升至d_model,导致令牌嵌入矩阵秩最多为16。即使在此设置下,持续增长仍具可行性。在68.9B令牌的FineWeb-Edu + Cosmopedia数据集上,16层269.7M参数模型经交错式LoRA阶段训练后,在合并最后阶段的LoRA适配器到269.7M基模型后达到28.92% MMLU得分。由于各阶段数据混合比例变化,该结果被视为可行性演示而非清晰因果对比。整体证据支持一个狭窄主张:在受限活跃参数预算下,可在冻结最小接口上实现有效持续学习,但与密集单体训练相比在最终困惑度上存在明显权衡。

原文摘要 · Abstract (English)

We study a constrained training regime for decoder-only Transformers in which the token interface is fixed, previously trained dense blocks are not reopened, and the active trainable parameter set is kept approximately constant as depth grows. Starting from a shallow model, we stack new blocks and train only the newest blocks and the LM head; optional LoRA phases provide limited global readjustment under the same active-parameter budget. The paper asks a feasibility/tradeoff question, not whether this regime matches tuned monolithic pretraining. In a common-protocol 9-layer study on a frozen Unicode substrate, the constructive frozen-Unicode model uses 105.0M active trainable parameters, compared with 180.5M for the interface-matched monolithic frozen baseline and 247.6M for the fully trainable monolithic baseline. We then consider an extreme fixed interface: each token is represented only by a frozen 16-dim binary token-ID code, deterministically lifted to d_model, so the resulting token embedding matrix has rank at most 16. Even in this setting, continued growth remains viable. In a 68.9B-token run on FineWeb-Edu + Cosmopedia, a 16-layer 269.7M model trained above this fixed interface reaches 28.92\% MMLU after an interleaved LoRA stage. Reported final metrics are measured after merging the last-stage LoRA adapters into the 269.7M base model. Because the data mixture changes across stages in this long-horizon run, we interpret it as a viability demonstration rather than a clean causal comparison. Overall, the evidence supports a narrow claim: useful continued learning can proceed above a frozen minimal interface under a bounded active trainable-parameter budget, with a clear tradeoff against dense monolithic training in final perplexity.

模型扩展参数效率持续学习冻结接口

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。