用线性注意力和轻量增强实现高效语言建模，适合资源受限场景。
Sample-Efficient Language Modeling with Linear Attention and Lightweight Enhancements
- 用mLSTM替代自注意力，结合滑动窗口与动态调制提升效率
- 在STRICT和STRICT-SMALL任务上零样本性能持续提升
- 新优化器Muon稳定训练，降低困惑度，适合小数据场景
我们研究了在BabyLM 2025共享任务约束下，实现样本高效语言建模的架构与优化技术。我们的模型BLaLM将自注意力替换为线性时间复杂度的mLSTM token mixer，并探索了轻量级增强方法，包括短卷积、带动态调制的滑动窗口注意力以及Hedgehog特征映射。为支持低资源训练，我们构建了一个高质量语料库，强调可读性和教学结构。在STRICT和STRICT-SMALL两个赛道的实验表明：（1）线性注意力结合滑动窗口注意力能持续提升零样本性能；（2）Muon优化器稳定收敛，相比AdamW降低困惑度。这些结果展示了无需依赖规模即可实现高效语言建模的有效策略。
原文摘要 · Abstract (English)
We study architectural and optimization techniques for sample-efficient language modeling under the constraints of the BabyLM 2025 shared task. Our model, BLaLM, replaces self-attention with a linear-time mLSTM token mixer and explores lightweight enhancements, including short convolutions, sliding window attention with dynamic modulation, and Hedgehog feature maps. To support training in low-resource settings, we curate a high-quality corpus emphasizing readability and pedagogical structure. Experiments across both STRICT and STRICT-SMALL tracks show that (1) linear attention combined with sliding window attention consistently improves zero-shot performance, and (2) the Muon optimizer stabilizes convergence and reduces perplexity over AdamW. These results highlight effective strategies for efficient language modeling without relying on scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。