用轻量采样和空间约束注意力,提升城市时空预测的全局关系捕捉效率
Spatially Constrained Transformer with Efficient Global Relation Modelling for Spatio-Temporal Prediction
- 结合CNN与自注意力,同时捕获局部与远距离时空关系
- 在三个真实数据集上表现优于现有方法,计算成本降低40%
- 适合需要高效高精度城市交通/气象预测的研究者
准确的时空预测对智慧城市建设至关重要。现有方法多依赖卷积神经网络(CNN)捕捉全局关系,但其存在邻域偏差,难以建模遥远区域间的关联。为此,本文提出ST-SampleNet,一种融合CNN与自注意力机制的新型变压器架构,有效捕捉局部与全局关系。随着区域数量增加,自注意力的二次复杂度成为瓶颈。为此,我们引入轻量级区域采样策略,剔除非必要区域,提升效率。此外,设计了空间约束位置编码,将空间邻近信息融入自注意力,增强语义理解并提升性能。在三个真实世界数据集上的实验验证了该方法的有效性。其高效变体实现约40%的计算成本降低,性能仅下降约1%。
原文摘要 · Abstract (English)
Accurate spatio-temporal prediction is crucial for the sustainable development of smart cities. However, current approaches often struggle to capture important spatio-temporal relationships, particularly overlooking global relations among distant city regions. Most existing techniques predominantly rely on Convolutional Neural Networks (CNNs) to capture global relations. However, CNNs exhibit neighbourhood bias, making them insufficient for capturing distant relations. To address this limitation, we propose ST-SampleNet, a novel transformer-based architecture that combines CNNs with self-attention mechanisms to capture both local and global relations effectively. Moreover, as the number of regions increases, the quadratic complexity of self-attention becomes a challenge. To tackle this issue, we introduce a lightweight region sampling strategy that prunes non-essential regions and enhances the efficiency of our approach. Furthermore, we introduce a spatially constrained position embedding that incorporates spatial neighbourhood information into the self-attention mechanism, aiding in semantic interpretation and improving the performance of ST-SampleNet. Our experimental evaluation on three real-world datasets demonstrates the effectiveness of ST-SampleNet. Additionally, our efficient variant achieves a 40% reduction in computational costs with only a marginal compromise in performance, approximately 1%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。