蛋白结构数据可直接输入Transformer，让其自动学习空间距离注意力。
Transformers trained on proteins can learn to attend to Euclidean distance
- 用坐标线性嵌入输入Transformer，使其自主建模三维结构信息。
- 在蛋白质掩码预测任务中，结构预训练使性能超越专用结构模型。
- 理论证明注意力可学成3D高斯分布，适用于结构-语言混合建模。
虽然传统Transformer通常处理序列数据，但可与结构模型（如SE(3)不变或等变图神经网络）结合用于3D应用，如蛋白质结构建模。现有方法通常分两步：(1) 将结构特征预处理为输入给Transformer，或(2) 将Transformer嵌入再通过结构表示处理。然而，已有证据表明，Transformer可独立处理结构信息，例如AlphaFold3的结构扩散模型。本文证明，当输入坐标线性嵌入时，Transformer可作为独立的结构模型运行。我们首先从理论上解释了为何Transformer能将注意力机制学习为具有可学习方差的3D高斯函数。随后，通过模拟3D点和蛋白质掩码词元预测任务验证该理论。最后，我们发现使用结构信息预训练蛋白质Transformer编码器，在下游任务中表现优于定制的结构模型。本工作为使用标准Transformer构建结构-语言混合模型提供了理论基础。
原文摘要 · Abstract (English)
While conventional Transformers generally operate on sequence data, they can be used in conjunction with structure models, typically SE(3)-invariant or equivariant graph neural networks (GNNs), for 3D applications such as protein structure modelling. These hybrids typically involve either (1) preprocessing/tokenizing structural features as input for Transformers or (2) taking Transformer embeddings and processing them within a structural representation. However, there is evidence that Transformers can learn to process structural information on their own, such as the AlphaFold3 structural diffusion model. In this work we show that Transformers can function independently as structure models when passed linear embeddings of coordinates. We first provide a theoretical explanation for how Transformers can learn to filter attention as a 3D Gaussian with learned variance. We then validate this theory using both simulated 3D points and in the context of masked token prediction for proteins. Finally, we show that pre-training protein Transformer encoders with structure improves performance on a downstream task, yielding better performance than custom structural models. Together, this work provides a basis for using standard Transformers as hybrid structure-language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。