用稀疏注意力提升点云模型效率，适配大规模物理系统建模。
Natively Trainable Sparse Attention for Hierarchical Point Cloud Datasets
- 将稀疏注意力机制适配非序列点云数据，突破传统注意力的二次复杂度瓶颈。
- 在宇宙学、分子动力学等3个物理数据集上性能媲美或超越原版Erwin模型。
- 方法可直接训练，适合处理大规模点云的科学计算场景。
将Transformer应用于大规模物理系统数据集，受限于注意力机制的二次计算开销。本文结合Erwin架构与原生稀疏注意力（NSA）机制，提升模型效率与感受野，缓解注意力复杂度问题。针对非序列数据特性，对NSA机制进行适配，实现Erwin NSA模型，并在宇宙学模拟、分子动力学和气压建模三个物理科学数据集上进行评估，性能达到或超过原始Erwin模型表现。此外，复现了Erwin论文中的实验结果以验证其实现可靠性。
原文摘要 · Abstract (English)
Unlocking the potential of transformers on datasets of large physical systems depends on overcoming the quadratic scaling of the attention mechanism. This work explores combining the Erwin architecture with the Native Sparse Attention (NSA) mechanism to improve the efficiency and receptive field of transformer models for large-scale physical systems, addressing the challenge of quadratic attention complexity. We adapt the NSA mechanism for non-sequential data, implement the Erwin NSA model, and evaluate it on three datasets from the physical sciences -- cosmology simulations, molecular dynamics, and air pressure modeling -- achieving performance that matches or exceeds that of the original Erwin model. Additionally, we reproduce the experimental results from the Erwin paper to validate their implementation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。