对比七种线性化架构,发现模型结构比训练规模更决定性能上限。
What Matters in Linearizing Language Models? A Comparative Study of Architecture, Scale, and Task Adaptation
- 用统一框架比较xLSTM、GLA等七种线性化架构的性能表现
- 140M到1.7B参数下性能排名稳定,纠错更新规则提升最快
- 门控型模型适合长文本检索,加法型模型会因状态饱和失效
线性化是构建高效语言模型的新策略,通过将Transformer中的注意力机制替换为计算高效的次二次项混合器(token mixers)实现。随着各类混合器不断涌现,尚不清楚哪些归纳偏置最适合继承Transformer的能力,且线性化在参数与序列长度扩展下的影响也不明确。为此,本文提出统一评估框架,对比七种代表性架构(包括xLSTM、GLA、Gated DeltaNet)。结果表明:从140M到1.7B参数,性能层级保持稳定,纠错更新规则展现出更优的缩放指数;性能差距在早期即形成,并持续至100亿令牌训练后的渐近成熟阶段,说明状态分辨能力是比蒸馏预算更根本的瓶颈。尽管多数模型能适应指令微调,但仅门控型δ规则模型能维持长上下文检索所需的精度,加法型模型则因不可逆的状态饱和导致性能下降。研究结论表明,成功的线性化仍受限于架构的归纳偏置,单纯增加训练算力无法克服这一根本限制。
原文摘要 · Abstract (English)
Linearization has emerged as a strategy for developing efficient language models (LMs). Starting from an existing Transformer-based LM, linearization replaces the attention component with computationally efficient subquadratic \textit{token mixers}. However, as an increasing number of mixers are proposed, it remains unclear which inductive biases are best suited to inherit the original Transformer's capabilities. Furthermore, it is unknown how linearization is affected by parameter and token budget scaling. To address these questions, we propose a unified setup to compare seven representative architectures, including xLSTM, GLA, and Gated DeltaNet. Our findings reveal that performance hierarchies remain stable from 140M to 1.7B parameters, with error-correcting update rules demonstrating superior scaling exponents. We show that performance gaps are established early and persist through asymptotic maturity at 10B tokens, suggesting that state resolution is a more fundamental bottleneck than the distillation budget. Finally, while most models adapt to instruction tuning, only gated delta-rule formulations maintain the precision necessary for long-context retrieval, whereas additive models suffer from irreversible state saturation. These results suggest that for successful linearization, architectural inductive biases remain the primary constraint that cannot be overcome by simply scaling training compute.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。