探究适合小规模语言模型的最优序列长度。
What is the Best Sequence Length for BABYLM?
- 在固定算力下对比不同序列长度对模型性能的影响。
- 长序列更利于形态类比推理，短序列足够语法泛化任务。
- 模型架构与任务类型共同决定最佳序列长度。
Transformer语言模型通常使用固定长度的上下文窗口，其长度随大规模预训练数据集的增长而增加。然而，在BabyLM挑战中，许多过往提交都采用了较短的序列长度。本文研究序列长度对BabyLM预训练的影响，回答一个简单问题：训练小型语言模型时应使用多长的序列？在1亿词训练数据和固定算力预算下，我们对比了1.25亿参数的Mamba和OPT模型，发现虽然更长的序列通常表现更好，但最优长度取决于任务和模型架构。短序列足以完成语法泛化任务，而长上下文则更有利于形态类比推理任务。
原文摘要 · Abstract (English)
Transformer language models typically operate with a fixed-length context window, which has grown in step with large-scale pretraining datasets. In the BabyLM Challenge, however, many past submissions have defaulted to using much shorter sequence lengths. We examine the impact of sequence length on BabyLM pretraining, to answer the simple question: what sequence length should we be using when training Baby LMs? Using 100M-word training data and fixed compute budgets, we compare 125M-parameter Mamba and OPT models, finding that although longer is often better, the optimal length depends on both task and architecture. Shorter sequences are sufficient for grammatical generalization tasks whereas longer contexts benefit morphological analogical reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。