用物理动力学采样神经网络损失极小值，更准更可控。
Don't Stop Me Yet: Sampling Loss Minima via Dissipative Riemannian Mechanics
- 基于动能、引力和阻尼的物理系统采样
- 能精确采样损失极小值集，不偏移
- 适合需要可靠不确定性的贝叶斯推断场景
现代神经网络损失函数的极小值通常不是孤立点，而是训练数据上具有重参数化不变性的连通区域。解析刻画这些解是难题，但采样方法可行。现有方法要么在低损失区域扩散，无法精确采样重参数化不变解；要么过于局部，限制对其他极小谷的探索。本文提出基于动能、引力和耗散摩擦项的动态系统来采样此类不变解。所提采样器DiMS能保证精确采样极小值集，且依赖物理启发的超参数，可调控探索能力。以贝叶斯推理中的不确定性量化为动机，实验显示性能优于已有方法。
原文摘要 · Abstract (English)
The minima of modern neural network loss functions are typically not isolated, rather they form connected components of reparameterization invariant solutions on the training data. Analytically characterizing these solutions is a hard problem, but sampling approaches are feasible. By construction, existing methods either spread over low-loss regions, and thus do not sample reparameterization invariant solutions exactly, or are inherently local, which limits exploration of other minima valleys. We propose sampling such reparameterization invariant models using a dynamical system based on kinetic energy, subject to a gravitational pull and a friction term that dissipates energy from the system. Our proposed sampler, DiMS, is guaranteed to sample exactly from the minimum level sets and depends on physically motivated hyperparameters which allows control over the exploration capabilities of the sampler. We consider uncertainty quantification in Bayesian inference as the motivating problem and observe improved performance compared to previously proposed approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。