arXiv:2604.21691stat.MLcs.LG2026-04被引 20

深度学习正走向可解释的科学理论,揭示训练与性能的通用规律。

There Will Be a Scientific Theory of Deep Learning

论文配图:There Will Be a Scientific Theory of Deep Learning
图 1 · 摘自论文原文
  • 从理想化模型出发,提炼学习动态的宏观规律
  • 发现跨系统共有的普适行为,支持可验证预测
  • 适合对理论框架或可解释性感兴趣的科研者

本文主张深度学习的科学理论正在形成。该理论旨在刻画训练过程、隐层表征、最终权重及模型性能的重要性质与统计特征。我们整合当前研究进展,识别出五个关键方向:(a) 可解的理想化设定,为真实系统提供学习动态直觉;(b) 可处理的极限情形,揭示基础学习现象;(c) 揭示重要宏观可观测量的简单数学规律;(d) 超参数理论,将其与训练过程解耦,简化系统;(e) 跨系统共享的普适行为,明确需解释的现象。这些工作共同具备三类特征:关注训练动态、聚焦粗粒度统计、强调可证伪的定量预测。我们提出将该理论视为学习过程的‘力学’,命名为学习力学(learning mechanics)。它与统计与信息论视角并行互补,并与机制可解释性形成协同关系。我们回应了关于理论不可能或不重要的常见质疑,并展望关键开放问题与入门建议。更多信息见 learningmechanics.pub。

原文摘要 · Abstract (English)

In this paper, we make the case that a scientific theory of deep learning is emerging. By this we mean a theory which characterizes important properties and statistics of the training process, hidden representations, final weights, and performance of neural networks. We pull together major strands of ongoing research in deep learning theory and identify five growing bodies of work that point toward such a theory: (a) solvable idealized settings that provide intuition for learning dynamics in realistic systems; (b) tractable limits that reveal insights into fundamental learning phenomena; (c) simple mathematical laws that capture important macroscopic observables; (d) theories of hyperparameters that disentangle them from the rest of the training process, leaving simpler systems behind; and (e) universal behaviors shared across systems and settings which clarify which phenomena call for explanation. Taken together, these bodies of work share certain broad traits: they are concerned with the dynamics of the training process; they primarily seek to describe coarse aggregate statistics; and they emphasize falsifiable quantitative predictions. We argue that the emerging theory is best thought of as a mechanics of the learning process, and suggest the name learning mechanics. We discuss the relationship between this mechanics perspective and other approaches for building a theory of deep learning, including the statistical and information-theoretic perspectives. In particular, we anticipate a symbiotic relationship between learning mechanics and mechanistic interpretability. We also review and address common arguments that fundamental theory will not be possible or is not important. We conclude with a portrait of important open directions in learning mechanics and advice for beginners. We host further introductory materials, perspectives, and open questions at learningmechanics.pub.

深度学习理论学习力学可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。