在数据有限时，用递归结构提升模型效率，避免过拟合。
Squeezing More from Limited Data with Recursive Transformers
- 用共享模块递归堆叠，降低参数消耗，提升计算效率。
- 在1000万词数据下，性能超越标准Transformer，且不逊于顶级竞赛模型。
- 适合资源受限但需高效训练的场景，如小语种或垂直领域任务。
在固定数据预算下（1000万至1亿词），增加参数量仅在一定规模内有益；超过最优值后模型会过拟合，泛化能力下降。我们通过两个语料库和多项下游任务验证了这一现象，发现最优模型规模依赖于数据量与目标任务。传统Transformer在此场景下表现不佳，因其嵌入层占大量参数，且每标记计算与表达能力强耦合。为此，我们提出递归Transformer，通过深度复用共享模块来扩展计算量，并采用因子化嵌入减少词汇映射参数。训练三个递归模型后发现，在1000万和1亿词数据下均优于标准Transformer，且与BabyLM Challenge 2025优胜者相当。
原文摘要 · Abstract (English)
Pre-training under limited data requires a different view of scaling than web-scale language modeling. With a fixed data budget but relatively abundant compute, increasing parameter count helps only up to an optimal scale; beyond that point, models overfit and generalization worsens. We study this behavior across 10M-100M word pre-training budgets, two corpora, and multiple downstream evaluations, and find that optimal size depends strongly on both the data budget and the downstream target. We argue that standard Transformers scale down poorly to this setting, because embeddings consume a large fraction of the parameter budget and per-token computation is tied to representational capacity. To address this coupling, we study recursive Transformers, reusing a shared block across depth to scale compute, together with factorized embeddings to reduce vocabulary-map parameters. We train three recursive models and find that they outperform standard Transformers at 10M and 100M words, while remaining competitive with BabyLM Challenge 2025 winners.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。