提出双路径结构，同时提升大模型算力与参数容量。
A Dual-Path Architecture for Scaling Compute and Capacity in LLMs
- 双路径设计：共享深层子层重复计算，宽广子层单次处理，灵活调节算力与参数。
- 在相同浮点运算量下，模型性能超越基线，参数更少。
- 可解释的路由门控机制，显示词性影响计算分配策略。
循环变压器通过多次复用同一模块实现算力扩展，但固定浮点运算量下其参数容量低于标准Transformer。本文提出一种新型双路径模块，可独立调节算力（序列操作次数）和容量（单步可用参数）。该模块在单层内并行设置两个路径：一个深度子层重复K次使用共享参数，另一个宽广子层仅执行一次、包含更大前馈网络。每个令牌通过独立门控机制决定路径选择，支持细粒度路由分析。实验表明，在两种浮点预算下，该模型在语言建模和下游任务中均优于同浮点预算的基线模型，且参数量更低。学习到的门控机制具有可解释性，显示功能词与词汇内容倾向于走宽路径，标点、符号与数值运算符则倾向走深路径。
原文摘要 · Abstract (English)
Looped transformers apply a shared block multiple times and have emerged as a parameter-efficient route to scaling compute in language models. However, at fixed FLOPs a looped model has strictly less capacity than a baseline transformer. We propose a novel dual-path block that can flexibly scale compute, the number of sequential operations applied to a hidden state, and capacity, the parameters available at a single step. For this we expose both axes as parallel pathways within a single layer: a deep sublayer re-applied K times with shared parameters, and a wide sublayer with an enlarged feed-forward network applied once. Independent per-token gates combine both axes and allow detailed per-token routing analyses. We show that across two FLOP budgets, our dual-path model surpasses iso-FLOP matched models on language modeling and downstream evaluations, while using fewer parameters than the baseline at matched FLOPs. The learned gates are directly interpretable and show systematic per-token allocation with function words and lexical content trend wide, while punctuation, symbols, and arithmetic tokens trend deep.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。