提出新型缩放定律,让一个模型能生成多个子模型,部署更灵活。
Theoretical Foundations of Scaling Law in Familial Models
- 引入粒度G作为新变量,统一描述模型规模、数据量与子模型数量的关系。
- 发现粒度惩罚呈幂律关系,指数极小,说明多子模型部署几乎不损性能。
- 适合关注边缘计算与高效部署的AI研究者和工程师。
神经网络缩放定律已成为优化大语言模型训练的基础,但传统方法通常假设单一密集模型输出。这一限制忽视了'家族模型'(Familial models)这一变革性范式,该范式对实现异构设备-边缘-云层级的普适智能至关重要。家族模型通过早期退出与中继推理机制,从单一共享骨干网络中生成G个可部署子模型。本文从理论与实证两方面将缩放定律扩展至'一次运行、多个模型'的范式,引入粒度(G)作为与模型规模(N)和训练数据量(D)并列的基本缩放变量。为精确量化该关系,我们提出统一函数形式L(N, D, G),并通过大规模实证实验参数化。具体地,采用严格的IsoFLOP实验设计,严格分离架构影响与计算规模的影响。在固定计算预算下,系统性扫描模型规模(N)与粒度(G),动态调整训练数据量(D)。该方法有效解耦粒度的边际成本与规模收益,确保缩放定律的高保真参数化。结果表明,粒度惩罚遵循乘法幂律,且指数极小。理论上,这连接了固定算力训练与动态架构;实践中,验证了'训练一次,部署多次'的可行性,证明部署灵活性可在不牺牲密集基线计算最优性的前提下实现。
原文摘要 · Abstract (English)
Neural scaling laws have become foundational for optimizing large language model (LLM) training, yet they typically assume a single dense model output. This limitation effectively overlooks "Familial models, a transformative paradigm essential for realizing ubiquitous intelligence across heterogeneous device-edge-cloud hierarchies. Transcending static architectures, familial models integrate early exits with relay-style inference to spawn G deployable sub-models from a single shared backbone. In this work, we theoretically and empirically extend the scaling law to capture this "one-run, many-models" paradigm by introducing Granularity (G) as a fundamental scaling variable alongside model size (N) and training tokens (D). To rigorously quantify this relationship, we propose a unified functional form L(N, D, G) and parameterize it using large-scale empirical runs. Specifically, we employ a rigorous IsoFLOP experimental design to strictly isolate architectural impact from computational scale. Across fixed budgets, we systematically sweep model sizes (N) and granularities (G) while dynamically adjusting tokens (D). This approach effectively decouples the marginal cost of granularity from the benefits of scale, ensuring high-fidelity parameterization of our unified scaling law. Our results reveal that the granularity penalty follows a multiplicative power law with an extremely small exponent. Theoretically, this bridges fixed-compute training with dynamic architectures. Practically, it validates the "train once, deploy many" paradigm, demonstrating that deployment flexibility is achievable without compromising the compute-optimality of dense baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。