用热带几何设计注意力机制,让神经模型更精准解决组合难题。
Tropical Attention: Neural Algorithmic Reasoning for Combinatorial Algorithms

- 基于热带几何构建分段线性注意力,保持决策结构清晰
- 在长度和数值上均实现更强的分布外泛化能力
- 适合需要高精度与可解释性的复杂组合推理任务
能否通过代数几何为现代神经推理模型引入数学严谨的归纳偏置,从而提升其精确性、鲁棒性和可解释性?为此,我们提出热带注意力(Tropical Attention),一种基于热带几何的注意力机制,将注意力核映射至热带射影空间,使推理过程呈分段线性且1-Lipschitz,从而保留组合推理固有的多面体决策结构。我们证明,多头热带注意力(MHTA)可通用逼近热带电路,并通过组合实现热带传递闭包,仅需多项式资源开销,无需循环机制。这些保证解释了为何生成的多面体决策边界保持锐利且尺度不变,而非被Softmax平滑。实验表明,热带注意力在长度与数值上的分布外泛化能力更强,对扰动噪声具有高度鲁棒性,且推理速度更快、参数更少,优于基于Softmax和循环结构的基线。首次将神经算法推理拓展至NP难与NP完全问题,为构建更精确、更具表现力的大规模推理模型(LRMs)铺平道路,可用于系统发育、密码学、粒子物理与数学发现等复杂组合挑战。
原文摘要 · Abstract (English)
Can algebraic geometry enhance the sharpness, robustness, and interpretability of modern neural reasoning models by equipping them with a mathematically grounded inductive bias? To answer this, we introduce Tropical Attention, an attention mechanism grounded in tropical geometry that lifts the attention kernel into tropical projective space, where reasoning is piecewise-linear and 1-Lipschitz, thus preserving the polyhedral decision structure inherent to combinatorial reasoning. We prove that Multi-Head Tropical Attention (MHTA) stacks universally approximate tropical circuits and realize tropical transitive closure through composition, achieving polynomial resource bounds without invoking recurrent mechanisms. These guarantees explain why the induced polyhedral decision boundaries remain sharp and scale-invariant, rather than smoothed by Softmax. Empirically, we show that Tropical Attention delivers stronger out-of-distribution generalization in both length and value, with high robustness against perturbative noise, and substantially faster inference with fewer parameters compared to Softmax-based and recurrent attention baselines. For the first time, we extend neural algorithmic reasoning beyond PTIME problems to NP-hard and NP-complete problems, paving the way toward sharper and more expressive Large Reasoning Models (LRMs) capable of tackling complex combinatorial challenges in phylogenetics, cryptography, particle physics, and mathematical discovery.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。