预训练数据决定大模型损失缩放规律，影响下游性能表现。
LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws
- 预训练数据是决定损失缩放趋势的关键因素
- 模型规模、架构等对缩放规律影响有限
- 选对数据集比调参更关键，适合模型开发者参考
缩放定律指导大语言模型（LLM）的开发，提供模型规模、训练样本量和算力之间的最优平衡估算。近年来，损失到损失的缩放定律通过关联预训练数据与下游任务的损失，成为理解并提升LLM性能与泛化能力的强大工具。本文研究发现，预训练数据决定了损失缩放的趋势。相比之下，模型规模、优化超参数、分词器设计，甚至如Llama（Transformer）与Mamba（状态空间模型）之间的显著架构差异，通常对缩放规律影响较小。因此，从业者应精心选择合适的预训练数据以实现最优下游性能，而模型架构及其他设置可自由调整以提升训练效率。
原文摘要 · Abstract (English)
Scaling laws guide the development of large language models (LLMs) by offering estimates for the optimal balance of model size, tokens, and compute. More recently, loss-to-loss scaling laws that relate losses across pretraining datasets and downstream tasks have emerged as a powerful tool for understanding and improving LLM performance and generalization. In this work, we investigate which factors most strongly influence loss-to-loss scaling. Our experiments reveal that the pretraining data determines the scaling trend. In contrast, model size, optimization hyperparameters, tokenizer and even significant architectural differences, such as between transformer-based models like Llama and state-space models like Mamba, generally have limited impact. Consequently, practitioners should carefully curate suitable pretraining datasets for optimal downstream performance, while architectures and other settings can be freely optimized for training efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。