提出新型神经算子LANO，用少量代理令牌实现高效高精度微分方程求解。
Efficient High-Accuracy PDEs Solver with the Linear Attention Neural Operator
- 引入少量代理令牌，通过代理注意力机制降低计算复杂度。
- 在标准基准上平均提升19.5%精度，超越现有最优方法。
- 适合需要大规模、高精度科学计算的机器学习应用。
神经算子为函数空间映射提供了强大的数据驱动框架。基于Transformer的神经算子架构面临可扩展性与准确率的根本权衡：软最大注意力虽具优异保真度，但计算复杂度达$/mathcal{O}(N^2 d)$（$N$为网格点数，$d$为隐藏维数）；而线性注意力变体虽将复杂度降至$/mathcal{O}(N d^2)$，却常导致显著精度下降。本文提出一种新型神经算子——线性注意力神经算子（LANO），通过代理机制重构注意力，同时实现可扩展性与高精度。LANO引入一组$M$个代理令牌（$M \ll N$），介导$N$个令牌间的全局交互。该代理注意力机制使算子层复杂度降为$/mathcal{O}(MN d)$，同时保留软最大注意力的表达能力。理论上，证明了其通用逼近性质，展现更优条件与稳定性。实验上，LANO超越当前最优神经PDE求解器（如基于切片的软最大注意力的Transolver），在标准基准上平均提升19.5%精度。通过弥合线性复杂度与软最大注意力性能间的鸿沟，LANO为科学机器学习应用建立了可扩展、高精度的基础。
原文摘要 · Abstract (English)
Neural operators offer a powerful data-driven framework for learning mappings between function spaces, in which the transformer-based neural operator architecture faces a fundamental scalability-accuracy trade-off: softmax attention provides excellent fidelity but incurs quadratic complexity $\mathcal{O}(N^2 d)$ in the number of mesh points $N$ and hidden dimension $d$, while linear attention variants reduce cost to $\mathcal{O}(N d^2)$ but often suffer significant accuracy degradation. To address the aforementioned challenge, in this paper, we present a novel type of neural operators, Linear Attention Neural Operator (LANO), which achieves both scalability and high accuracy by reformulating attention through an agent-based mechanism. LANO resolves this dilemma by introducing a compact set of $M$ agent tokens $(M \ll N)$ that mediate global interactions among $N$ tokens. This agent attention mechanism yields an operator layer with linear complexity $\mathcal{O}(MN d)$ while preserving the expressive power of softmax attention. Theoretically, we demonstrate the universal approximation property, thereby demonstrating improved conditioning and stability properties. Empirically, LANO surpasses current state-of-the-art neural PDE solvers, including Transolver with slice-based softmax attention, achieving average $19.5\%$ accuracy improvement across standard benchmarks. By bridging the gap between linear complexity and softmax-level performance, LANO establishes a scalable, high-accuracy foundation for scientific machine learning applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。