小模型性能随规模变化规律首次被揭示，发现越小越准但错得不同。
Scaling Laws in the Tiny Regime: How Small Models Change Their Mistakes
- 在22K至19.8M参数范围内测试90个模型，分析误差率与规模的幂律关系。
- 小模型误差下降更快（指数α=0.106~0.156），但大到一定程度后性能饱和。
- 小模型误判集中在简单样本，且校准性更好，不适合用大模型经验评估。
神经网络缩放定律描述了模型性能随规模提升的幂律关系，但现有研究主要针对超1亿参数的大模型。本文首次系统考察小于2000万参数的微型模型（即边缘计算场景）：在CIFAR-100上训练90个模型（22K–19.8M参数），采用Plain ConvNet和MobileNetV2两种架构，固定深度与训练条件，仅调整宽度。结果表明，两类模型误差率均近似遵循幂律关系，对应指数分别为α=0.156±0.002（ScaleCNN）与α=0.106±0.001（MobileNetV2），比大型语言模型（α≈0.076）更陡峭。然而幂律并非全局成立：局部指数随规模衰减，且MobileNetV2在19.8M处出现饱和（局部指数α_local=0.006）。错误模式也显著变化：最小与最大规模模型的错误样本重叠率仅0.35（25组种子，±0.004），说明压缩改变了误判对象而非数量。小模型将能力集中于简单类别（吉尼系数0.26，22K参数时；4.7M时为0.09），放弃最难类别（底5类准确率：22K时10% → 4.7M时53%）。令人意外的是，最小模型校准最优（ECE=0.013，中等规模峰值达0.110）。因此，边缘部署时仅依赖聚合精度是误导性的，验证必须在目标模型尺寸进行。
原文摘要 · Abstract (English)
Neural scaling laws describe how model performance improves as a power law with size, but existing work focuses on models above 100M parameters. The sub-20M regime -- where TinyML and edge AI operate -- remains unexamined. We train 90 models (22K--19.8M parameters) across two architectures (plain ConvNet, MobileNetV2) on CIFAR-100, varying width while holding depth and training fixed. Both follow approximate power laws in error rate: $α= 0.156 \pm 0.002$ (ScaleCNN) and $α= 0.106 \pm 0.001$ (MobileNetV2) across five seeds. Since prior work fit cross-entropy loss rather than error rate, direct exponent comparison is approximate; with that caveat, these are 1.4--2x steeper than $α\approx 0.076$ for large language models. The power law does not hold uniformly: local exponents decay with scale, and MobileNetV2 saturates at 19.8M parameters ($α_{\mathrm{local}} = 0.006$). Error structure also changes. Jaccard overlap between error sets of the smallest and largest ScaleCNN is only 0.35 (25 seed pairs, $\pm 0.004$) -- compression changes which inputs are misclassified, not merely how many. Small models concentrate capacity on easy classes (Gini: 0.26 at 22K vs. 0.09 at 4.7M) while abandoning the hardest (bottom-5 accuracy: 10% vs. 53%). Counter to expectation, the smallest models are best calibrated (ECE = 0.013 vs. peak 0.110 at mid-size). Aggregate accuracy is therefore misleading for edge deployment; validation must happen at the target model size.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。