用相互作用的伊辛模型改进注意力,提升科学任务精度
Variational-Ising-Attention (VIA):TailoredAttentionMattersfor Science

- 用可学习的耦合项替代softmax,让注意力成集体状态
- 在逆合成反应预测中显著优于传统注意力机制
- 适合需要结构化协同的科学计算场景
注意力通过查询-键打分与softmax归一化实现上下文建模。受工业界长序列需求驱动,主流研究趋向稀疏与高效,但softmax的独立性假设仍普遍存在。对于不受长序列约束的科学任务,更丰富的结构化关联往往至关重要,因此定制化注意力不仅可行,且更为合适。为此,我们提出变分-伊辛注意力(VIA),在softmax归一化基础上引入相互作用的伊辛模型;注意力模式通过可学习的成对耦合,经变分平均场推断产生,将注意力从孤立项排序重新定义为相互作用实体的集体状态。我们在逆合成反应中心预测任务上实现VIA,该任务本质上受协同断键约束支配。跨模型变体的全面实验及机制分析表明,VIA始终显著优于标准softmax注意力。更广泛地,我们的发现提示:对于科学问题,最优解并非通用效率,而是与内在领域结构相匹配的定制注意力。本工作提供了该范式的理论基础与实证验证实例。
原文摘要 · Abstract (English)
Attention enables context modeling via query-key scoring with softmax normalization. Driven by industrial long-context demands, mainstream research has converged toward sparsity and efficiency--yet softmax's independence assumption persists. For scientific tasks unburdened by long-token constraints, however, richer structured coupling may often be essential, making tailored attention both viable and more appropriate. To this end, we propose Variational-Ising-Attention (VIA), which augments softmax normalization with an interacting Ising model; attention patterns emerge from learnable pairwise couplings via variational mean-field inference, redefining attention from a ranking over isolated items to a collective state over interacting entities. We instantiate VIA on retrosynthesis reaction center prediction, a task inherently governed by cooperative bond-breaking constraints. Comprehensive experiments across model variants, coupled with mechanistic analyses, demonstrate that VIA consistently and substantially outperforms standard softmax attention. More broadly, our findings suggest that for scientific problems, the optimal solution is not general-purpose efficiency, but appropriately tailored attention aligned with intrinsic domain structure. This work provides a theoretically grounded and empirically validated instantiation of this paradigm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。