统一神经网络训练与压缩的谱演化规律，揭示其内在机制。
A Generalized Spectral Framework to Expain Neural Scaling and Compression Dynamics
- 提出广义谱框架，用多项式函数描述谱演化过程。
- 发现学习与压缩间存在不变关系，适用于不同模型规模场景。
- 为理解模型缩放和压缩动力学提供统一理论视角。
经验缩放定律描述了测试损失及其他性能指标如何随模型规模、数据集规模和计算量变化。尽管这些定律在特定范围内一致，但在相关设置（如模型压缩）中却报告了看似不同的缩放行为。受神经表示谱分析近期进展的启发，本文提出一个广义谱框架，将学习动态与压缩现象统一于同一函数假设下。我们将谱演化函数从线性核形式 $g(λt)=λt$ 推广为渐近多项式形式 $g(λ,t;β)$，由有效谱-时间弹性 $ρ(β)$ 描述。该框架涵盖懒惰学习与特征学习理论作为特例，并推导出学习与压缩之间的不变关系。
原文摘要 · Abstract (English)
Empirical scaling laws describe how test loss and other performance metrics depend on model size, dataset size, and compute. While such laws are consistent within specific regimes, apparently distinct scaling behaviors have been reported for related settings such as model compression. Motivated by recent progress in spectral analyses of neural representations, this paper develops a \emph{generalized spectral framework} that unifies learning dynamics and compression phenomena under a common functional ansatz. We generalize the spectral evolution function from the linear kernel form $g(λt)=λt$ to an asymptotically polynomial function $g(λ,t;β)$, characterized by an effective spectral--temporal elasticity $ρ(β)$. This framework recovers existing lazy and feature-learning theories as special cases and yields an invariant relation between learning and compression
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。