用物理粒子模拟优化神经网络参数，防止参数坍缩。
KO: Kinetics-inspired Neural Optimizer with PDE Simulation Approaches
- 将参数更新建模为粒子碰撞的玻尔兹曼方程，模拟热扩散。
- 在多个数据集上超越Adam、SGD等基线，准确率提升显著。
- 适合追求稳定训练和高泛化能力的研究者使用。
神经网络优化算法的设计仍面临挑战，现有方法多依赖梯度法的启发式改进。本文提出一种受动力学理论启发的新型优化器KO（Kinetics-inspired Optimizer），将网络参数的训练动态重视为由动力学原理支配的粒子系统演化过程。通过数值求解描述随机粒子碰撞的玻尔兹曼输运方程（BTE），模拟参数更新。该物理驱动方法在优化过程中自然促进参数多样性，有效缓解参数凝结现象（即参数塌陷至低维子空间），其机制类似于物理系统的热扩散。我们从数学和物理解释两方面分析该性质。在图像分类（CIFAR-10/100、ImageNet）和文本分类（IMDB、Snips）任务上的大量实验表明，KO持续优于基准优化器（如Adam、SGD），在计算开销相当的情况下实现更高的准确率。
原文摘要 · Abstract (English)
The design of optimization algorithms for neural networks remains a critical challenge, with most existing methods relying on heuristic adaptations of gradient-based approaches. This paper introduces KO (Kinetics-inspired Optimizer), a novel neural optimizer inspired by kinetic theory and partial differential equation (PDE) simulations. We reimagine the training dynamics of network parameters as the evolution of a particle system governed by kinetic principles, where parameter updates are simulated via a numerical scheme for the Boltzmann transport equation (BTE) that models stochastic particle collisions. This physics-driven approach inherently promotes parameter diversity during optimization, mitigating the phenomenon of parameter condensation, i.e. collapse of network parameters into low-dimensional subspaces, through mechanisms analogous to thermal diffusion in physical systems. We analyze this property, establishing both a mathematical proof and a physical interpretation. Extensive experiments on image classification (CIFAR-10/100, ImageNet) and text classification (IMDB, Snips) tasks demonstrate that KO consistently outperforms baseline optimizers (e.g., Adam, SGD), achieving accuracy improvements while computation cost remains comparable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。