用固定高斯权重替代注意力，加速点云匹配训练并提升稳定性。
Localized Gaussians as Self-Attention Weights for Point Clouds Correspondence
- 将高斯函数作为固定注意力权重注入Transformer，简化计算
- 训练速度提升，优化更稳定，噪声下表现更鲁棒
- 适合追求高效部署的点云匹配场景
当前基于数据驱动的点云匹配方法需要大量训练时间和计算资源，给模型部署带来挑战。在点云匹配任务中，采用仅编码器的Transformer架构最近揭示了注意力头中涌现出语义有意义的模式，尤其表现为以输入形状各点为中心的高斯函数。本文进一步研究该现象，将这些模式作为固定注意力权重集成到Transformer的注意力头中。我们评估了两种变体：一种使用预设方差值的高斯函数，另一种将方差作为可学习参数。此外，我们分析了模型在噪声数据上的表现，并探索提升抗噪能力的方法。结果表明，固定注意力权重不仅加速训练过程，还增强了优化稳定性。我们还通过消融实验识别出信息注入最有效的网络层，并理解了模型对这种信息的依赖程度。
原文摘要 · Abstract (English)
Current data-driven methodologies for point cloud matching demand extensive training time and computational resources, presenting significant challenges for model deployment and application. In the point cloud matching task, recent advancements with an encoder-only Transformer architecture have revealed the emergence of semantically meaningful patterns in the attention heads, particularly resembling Gaussian functions centered on each point of the input shape. In this work, we further investigate this phenomenon by integrating these patterns as fixed attention weights within the attention heads of the Transformer architecture. We evaluate two variants: one utilizing predetermined variance values for the Gaussians, and another where the variance values are treated as learnable parameters. Additionally we analyze the performances on noisy data and explore a possible way to improve robustness to noise. Our findings demonstrate that fixing the attention weights not only accelerates the training process but also enhances the stability of the optimization. Furthermore, we conducted an ablation study to identify the specific layers where the infused information is most impactful and to understand the reliance of the network on this information.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。