提升3D原子建模效率与精度,支持更复杂的能量面模拟。
EquiformerV3: Scaling Efficient, Expressive, and General SE(3)-Equivariant Graph Attention Transformers
- 优化实现并引入新激活函数与注意力机制,提升模型效率与表达力。
- 在OC20、OMat24等数据集上达到当前最优性能。
- 适合需要高精度能量预测与分子动力学模拟的研究者使用。
随着SE(3)等变图神经网络成为3D原子建模的核心工具,提升其效率、表达力与物理一致性已成为大规模应用的关键挑战。本文提出EquiformerV3,第三代SE(3)等变图注意力变换器,从效率、表达力与泛化性三方面实现突破。相比EquiformerV2,实现1.75倍加速;引入等变合并层归一化、改进前馈网络超参数及平滑半径截断注意力;提出SwiGLU-S²激活函数,有效建模多体相互作用,保持严格等变性同时降低S²采样复杂度。结合平滑截断注意力与SwiGLU-S²,可精确建模平滑变化的势能面(PES),支持能量守恒模拟与高阶导数计算。在含去噪非平衡结构辅助任务(DeNS)训练下,EquiformerV3在OC20、OMat24和Matbench Discovery上取得当前最佳结果。
原文摘要 · Abstract (English)
As $SE(3)$-equivariant graph neural networks mature as a core tool for 3D atomistic modeling, improving their efficiency, expressivity, and physical consistency has become a central challenge for large-scale applications. In this work, we introduce EquiformerV3, the third generation of the $SE(3)$-equivariant graph attention Transformer, designed to advance all three dimensions: efficiency, expressivity, and generality. Building on EquiformerV2, we have the following three key advances. First, we optimize the software implementation, achieving $1.75\times$ speedup. Second, we introduce simple and effective modifications to EquiformerV2, including equivariant merged layer normalization, improved feedforward network hyper-parameters, and attention with smooth radius cutoff. Third, we propose SwiGLU-$S^2$ activations to incorporate many-body interactions for better theoretical expressivity and to preserve strict equivariance while reducing the complexity of sampling $S^2$ grids. Together, SwiGLU-$S^2$ activations and smooth-cutoff attention enable accurate modeling of smoothly varying potential energy surfaces (PES), generalizing EquiformerV3 to tasks requiring energy-conserving simulations and higher-order derivatives of PES. With these improvements, EquiformerV3 trained with the auxiliary task of denoising non-equilibrium structures (DeNS) achieves state-of-the-art results on OC20, OMat24, and Matbench Discovery.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。