小模型也能用数据规模定律，少用数据省成本
Is More Data Worth the Cost? Dataset Scaling Laws in a Tiny Attention-Only Decoder
- 用极简注意力解码器隔离数据量影响
- 用30%数据就达到90%的准确率
- 适合资源有限的研究者参考
训练Transformer语言模型成本高昂，性能通常随数据量和算力增加而提升。尽管大规模下的缩放定律已被研究，但在可控的小规模设置中其影响仍不明确。本文采用简化版仅注意力解码器架构，通过逐步训练功率为二的子集，观察到性能平滑提升并呈现显著递减收益，符合缩放定律特征。仅使用约30%的训练数据即可达到接近全数据量90%的验证阶段词级别准确率。结果为在受控、组件隔离环境下提供了数据缩放的实际洞见，并为计算与数据受限场景（如小型研究实验室和探索性模型开发）中的数据与算力权衡提供实用指导。
原文摘要 · Abstract (English)
Training Transformer language models is expensive, as performance typically improves with increasing dataset size and computational budget. Although scaling laws describe this trend at large scale, their implications in controlled, smaller-scale settings remain less explored. In this work, we isolate dataset-size effects using a strongly reduced attention-only decoder architecture. By training on progressively larger power-of-two subsets, we observe smooth performance improvements accompanied by clear diminishing returns, consistent with scaling-law behavior. Using only about 30% of the training data is sufficient to reach approximately 90% of the full-data validation token-level accuracy. These results provide actionable insights into dataset scaling in a controlled, component-isolated setting and offer practical guidance for balancing dataset size and computational cost in compute- and data-restricted environments, such as small research labs and exploratory model development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。