揭示密集关联记忆的鞍点层级结构，提升训练稳定性与效率
Saddle Hierarchy in Dense Associative Memory
- 基于统计物理推导鞍点方程，刻画真实与合成数据下的稳定点
- 提出新正则化方案，显著增强训练稳定性，实现可解释分类
- 发现小网络权重对应大网络的不稳定鞍点，启发高效训练算法
密集关联记忆（DAM）模型因对对抗样本具有鲁棒性且与注意力机制、生成扩散模型密切相关而重新受到关注。本文研究基于三层玻尔兹曼机与庞茨隐层单元的DAM，该单元代表数据簇与类别。通过统计力学分析，推导出描述真实数据训练下DAM驻点及合成数据训练下教师-学生框架内固定点的鞍点方程。基于此，提出一种新型正则化方案，显著提升训练稳定性。实验证明，该DAM能学习到监督与无监督分类任务的可解释解。进一步理论分析发现，相对较小DAM所学权重对应于更大DAM中的不稳定鞍点。据此设计了一种网络增长算法，利用鞍点层级结构大幅降低稠密关联记忆的训练计算成本。
原文摘要 · Abstract (English)
Dense Associative Memory (DAM) models have been attracting renewed attention since they were shown to be robust to adversarial examples and closely related to cutting edge machine learning paradigms, such as the attention mechanism and generative diffusion. We study a DAM built upon a three-layer Boltzmann machine with Potts hidden units, which represent data clusters and classes. Through a statistical mechanics analysis, we derive saddle-point equations that characterize both the stationary points of DAMs trained on real data and the fixed points of DAMs trained on synthetic data within a teacher-student framework. Based on these results, we propose a novel regularization scheme that makes training significantly more stable. Moreover, we show empirically that our DAM learns interpretable solutions to both supervised and unsupervised classification problems. Pushing our theoretical analysis further, we find that the weights learned by relatively small DAMs correspond to unstable saddle points in larger DAMs. We implement a network-growing algorithm that leverages this saddle-point hierarchy to drastically reduce the computational cost of training dense associative memory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。