发现经典神经网络在多任务训练中会自然产生类量子统计行为。
On Emergences of Non-Classical Statistical Characteristics in Classical Neural Networks
- 通过共享隐藏层的梯度竞争,引发非经典统计特性。
- 当资源不足时,S值随资源增加趋近经典上限2,缓解欠拟合。
- 首次揭示训练中隐含的非局域关联,适合研究模型内部机制者阅读。
受量子力学中测量不相容性和贝尔型不等式启发,我们提出一种简单的经典神经网络架构NCnet,其在典型可解释实验设置下稳定表现出非经典统计行为。通过CHSH不等式中的S统计量衡量非经典性,发现其源于多任务共享隐藏层神经元的梯度竞争。值得注意的是,即使无显式通信路径,一个任务头也能通过局部损失振荡间接感知其他任务头的训练状态,导致训练结果出现非局域关联。在低资源情况下,S值随资源增加逐步上升并逼近经典上界2,表明欠拟合随资源增加而缓解;当模型接近充分性能所需临界规模时,S值可能短暂超过2;随着资源持续增长,S值最终渐进衰减至2附近并持续波动。实证表明,当模型容量不足时,S值与泛化性能正相关,且首次接近2的阶段常对应良好泛化。总体而言,我们的结果表明非经典统计可为深度网络内部交互与训练动态提供新视角。
原文摘要 · Abstract (English)
Inspired by measurement incompatibility and Bell-family inequalities in quantum mechanics, we propose the Non-Classical Network (NCnet), a simple classical neural architecture that stably exhibits non-classical statistical behaviors under typical and interpretable experimental setups. We find non-classicality, measured by the $S$ statistic of CHSH inequality, arises from gradient competitions of hidden-layer neurons shared by multi-tasks. Remarkably, even without physical links supporting explicit communication, one task head can implicitly sense the training task of other task heads via local loss oscillations, leading to non-local correlations in their training outcomes. Specifically, in the low-resource regime, the value of $S$ increases gradually with increasing resources and approaches toward its classical upper-bound 2, which implies that underfitting is alleviated with resources increase. As the model nears the critical scale required for adequate performance, $S$ may temporarily exceed 2. As resources continue to grow, $S$ then asymptotically decays down to and fluctuates around 2. Empirically, when model capacity is insufficient, $S$ is positively correlated with generalization performance, and the regime where $S$ first approaches $2$ often corresponding to good generalization. Overall, our results suggest that non-classical statistics can provide a novel perspective for understanding internal interactions and training dynamics of deep networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。