用随机采样实现无需训练的注意力生成,温度控制记忆检索与自由创作。
Stochastic Attention via Langevin Dynamics on the Modern Hopfield Energy
- 基于霍普菲尔德能量梯度的随机采样,无需训练即可生成。
- 低温时精确回忆,高温时开放生成,五种数据域验证有效。
- 单个掩码可零样本实现类别条件生成,适合小数据场景。
注意力头通过加权平均检索存储的值。我们发现该计算等价于现代霍普菲尔德能量的一次梯度下降,而从对应玻尔兹曼分布中进行朗之万采样,可得到无需训练的随机注意力,仅由单一温度参数控制。降低温度可实现精确检索,提高温度则开启开放式生成。由于能量梯度等于注意力图,无需评分网络、训练循环或学习模型,特别适用于数据稀缺场景。我们推导出熵突变条件,确定任意记忆几何下的检索-生成转换温度,并在五个领域(覆盖两个数量级维度)验证了采样器有效性。在奥利弗蒂人脸数据集上,一个沿记忆轴应用的布尔掩码(类似Transformer因果掩码)使采样器实现零样本类别条件生成,无需重训练或分类器。在MNIST手写数字上,生成样本显著更新颖、更多样,且与马尔可夫链修正的黄金标准相当。在小型Pfam蛋白家族序列上,生成保持了氨基酸组成远优于变分自编码器,在相同新颖度下仍保留家族级保真性。去噪扩散基线在所有记忆规模测试中均失败,生成样本与各向同性噪声无异。该方法无需修改底层注意力架构。
原文摘要 · Abstract (English)
Attention heads retrieve: given a query, they return a weighted average of stored values. We showed that this computation is one step of gradient descent on the modern Hopfield energy, and that Langevin sampling from the corresponding Boltzmann distribution yielded stochastic attention, a training-free sampler controlled by a single temperature parameter. Lowering the temperature gave exact retrieval; raising it gave open-ended generation. Because the energy gradient equals the attention map, no score network, training loop, or learned model was required, making the approach particularly suited to the low-data regime where learned generative models are starved of training signal. We derived an entropy inflection condition that identified the retrieval-to-generation transition temperature for any memory geometry and validated the sampler on five domains spanning two orders of magnitude in dimension. A single Boolean mask on the attention softmax, identical to the causal mask used in transformers but applied along the memory axis rather than the sequence axis, turned the sampler into a zero-shot class-conditional generator on Olivetti faces with no retraining and no learned classifier. On MNIST digit images, stochastic attention produced samples that were markedly more novel and more diverse than the best learned baseline while matching a Metropolis-corrected gold standard. On protein sequences from a small Pfam family, the generation regime preserved amino acid composition far more faithfully than a variational autoencoder at matched novelty, indicating that the training-free score function retained family-level fidelity that learned models lost. A denoising diffusion baseline failed across all memory sizes tested, producing samples indistinguishable from isotropic noise. The approach required no architectural changes to the underlying attention mechanism.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。