小模型在算力受限下，注意力结构更高效，但深度和上下文过大会拖垮性能。
Architectural Trade-offs in Small Language Models Under Compute Constraints
- 从线性模型逐步引入非线性、自注意力与多层架构，系统对比性能与效率。
- 注意力模型在每浮点运算上表现优于MLP，小规模下仍具优势。
- 大模型成功的位置编码（如RoPE）在小模型中未必有效，需重新设计。
我们在严格算力约束下对小型语言模型进行系统性实证研究，分析架构选择与训练预算如何共同决定性能。从线性预测器出发，逐步引入非线性、自注意力机制及多层Transformer架构，在字符级的Tiny Shakespeare和词级的Penn Treebank（PTB）、WikiText-2数据集上评估。通过测试负对数似然（NLL）、参数量和近似训练浮点运算次数（FLOPs）刻画准确率-效率权衡。结果表明，即使在小规模下，基于注意力的模型在每FLOP上的效率也优于MLP；而增加深度或上下文长度若缺乏充分优化，反而会降低性能。我们进一步考察旋转位置编码（RoPE），发现大模型中有效的架构技术在小模型中并不一定适用。
原文摘要 · Abstract (English)
We present a systematic empirical study of small language models under strict compute constraints, analyzing how architectural choices and training budget interact to determine performance. Starting from a linear next-token predictor, we progressively introduce nonlinearities, self-attention, and multi-layer transformer architectures, evaluating each on character-level modeling of Tiny Shakespeare and word-level modeling of Penn Treebank (PTB) and WikiText-2. We compare models using test negative log-likelihood (NLL), parameter count, and approximate training FLOPs to characterize accuracy-efficiency trade-offs. Our results show that attention-based models dominate MLPs in per-FLOP efficiency even at small scale, while increasing depth or context without sufficient optimization can degrade performance. We further examine rotary positional embeddings (RoPE), finding that architectural techniques successful in large language models do not necessarily transfer to small-model regimes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。