机器学习中的优化困境源于一种新型不确定性原理。
A new Uncertainty Principle in Machine Learning
- 用双层表达式实现任意多项式的赫维赛德化，简化求解思路。
- 越尖锐的极小值点，对应越平缓的峡谷，导致优化难以收敛。
- 为科学建模提供新视角，适合研究优化机制的学者。
机器学习中的许多科学问题可归结为在特定变量中寻找多项式解。任意多项式的赫维赛德化可通过统一的两层表达式实现。然而，赫维赛德与Sigmoid展开的固有退化性会将梯度下降过程困在谷底，接近起始点却远离真实最小值。这一问题无法避免，可表述为一种独特的不确定性原理——极小值越尖锐，峡谷越平滑。这与傅里叶展开中的经典不确定性原理直接类比。主流机器学习软件通过随机初始化多组起点并择优来应对，属于经验性方法。值得注意的是，机器学习应用于科学时所遇现象本质上属于物理学范畴，而非计算机科学；它们虽熟悉但呈现新特征，例如将不确定性原理从傅里叶与小波分析扩展至一类近奇异的Sigmoid函数类别。
原文摘要 · Abstract (English)
Many scientific problems in the context of machine learning can be reduced to the search of polynomial answers in appropriate variables. The Hevisidization of arbitrary polynomial is actually provided by one-and-the same two-layer expression. What prevents the use of this simple idea is the fatal degeneracy of the Heaviside and sigmoid expansions, which traps the steepest-descent evolution at the bottom of canyons, close to the starting point, but far from the desired true minimum. This problem is unavoidable and can be formulated as a peculiar uncertainty principle -- the sharper the minimum, the smoother the canyons. It is a direct analogue of the usual one, which is the pertinent property of the more familiar Fourier expansion. Standard machine learning software fights with this problem empirically, for example, by testing evolutions, originated at randomly distributed starting points and then selecting the best one. Surprisingly or not, phenomena and problems, encountered in ML application to science are pure scientific and belong to physics, not to computer science. On the other hand, they sound slightly different and shed new light on the well-known phenomena -- for example, extend the uncertainty principle from Fourier and, later, wavelet analysis to a new peculiar class of nearly singular sigmoid functions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。