不同技能的模型应采用不同的扩展策略，否则会严重浪费算力。
Compute Optimal Scaling of Skills: Knowledge vs Reasoning
- 按任务类型区分缩放规律，而非统一使用通用公式
- 知识型和推理型任务的最优参数量差异可达50%
- 验证集的技能构成会显著影响算力分配决策
缩放定律是大模型开发中的关键环节，常用于预测训练决策，如在参数量与数据集大小间实现算力最优权衡。本文探讨算力最优缩放是否具有技能依赖性。研究发现，基于知识的任务（如知识问答）与基于推理的任务（如代码生成）存在显著不同的缩放行为。通过系统分析多种预训练数据混合比例，即使校正数据混合差异，两类任务的缩放特性仍本质不同。进一步分析表明，若验证集技能构成不匹配，可能导致算力最优参数量估计偏差近50%。
原文摘要 · Abstract (English)
Scaling laws are a critical component of the LLM development pipeline, most famously as a way to forecast training decisions such as 'compute-optimally' trading-off parameter count and dataset size, alongside a more recent growing list of other crucial decisions. In this work, we ask whether compute-optimal scaling behaviour can be skill-dependent. In particular, we examine knowledge and reasoning-based skills such as knowledge-based QA and code generation, and we answer this question in the affirmative: scaling laws are skill-dependent. Next, to understand whether skill-dependent scaling is an artefact of the pretraining datamix, we conduct an extensive ablation of different datamixes and find that, also when correcting for datamix differences, knowledge and code exhibit fundamental differences in scaling behaviour. We conclude with an analysis of how our findings relate to standard compute-optimal scaling using a validation set, and find that a misspecified validation set can impact compute-optimal parameter count by nearly 50%, depending on its skill composition.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。