xLSTM在保持线性复杂度的同时，性能超越Transformer。
xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexity
- 通过计算最优和过拟合两种场景，对比xLSTM与Transformer的缩放规律。
- 相同算力下，xLSTM交叉熵损失更低，性能更优。
- 适合关注长序列建模效率与推理成本的研究者和工程师。
缩放定律在大语言模型的成功中起着核心作用，可帮助在训练前预测模型性能与算力预算的关系。尽管变压器（Transformers）是主流架构，但xLSTM等新范式在上下文长度上具有线性复杂度，且在十亿参数量级下仍具竞争力。本文系统比较了Transformer与xLSTM在算力最优与过训练场景下的缩放行为，涵盖80M-7B参数规模及20亿至2万亿训练标记。研究发现，在典型的大语言模型训练与推理场景中，xLSTM表现更优；尤其在相同算力下，xLSTM始终以更低的交叉熵损失帕累托主导（Pareto-dominate）Transformer模型。
原文摘要 · Abstract (English)
Scaling laws play a central role in the success of Large Language Models (LLMs), enabling the prediction of model performance relative to compute budgets prior to training. While Transformers have been the dominant architecture, recent alternatives such as xLSTM offer linear complexity with respect to context length while remaining competitive in the billion-parameter regime. We conduct a comparative investigation on the scaling behavior of Transformers and xLSTM along the following lines, providing insights to guide future model design and deployment. First, we study the scaling behavior for xLSTM in compute-optimal and over-training regimes using both IsoFLOP and parametric fit approaches on a wide range of model sizes (80M-7B) and number of training tokens (2B-2T). Second, we examine the dependence of optimal model sizes on context length, a pivotal aspect that was largely ignored in previous work. Finally, we analyze inference-time scaling characteristics. Our findings reveal that in typical LLM training and inference scenarios, xLSTM scales favorably compared to Transformers. Notably, xLSTM models consistently Pareto-dominate Transformer models, delivering lower cross-entropy loss for the same compute budget.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。