提出三元交互注意力机制，提升蛋白质序列预测性能
Higher-Order Modular Attention: Fusing Pairwise and Triadic Interactions for Protein Sequences
- 引入三元交互路径，融合成对与三元残基关系
- 在三个TAPE基准上均优于标准注意力和高效变体
- 适合需要捕捉多残基协同效应的蛋白质研究
Transformer自注意力仅计算成对标记交互，但蛋白质序列到表型的关系常涉及三个及以上残基的协同作用，而点积注意力无法显式捕捉。本文提出统一的高阶模块化注意力（HOMA），融合成对注意力与显式的三元交互路径。为使三元注意力适用于长序列，HOMA采用块结构、窗口化的三元注意力机制。在三个TAPE基准（二级结构、荧光、稳定性）上评估，相比标准自注意力及高效变体（如块注意力、Linformer），HOMA在所有任务中均取得一致改进。结果表明，显式的三元项在可控计算成本下提供了互补的表示能力，有助于蛋白质序列预测。
原文摘要 · Abstract (English)
Transformer self-attention computes pairwise token interactions, yet protein sequence to phenotype relationships often involve cooperative dependencies among three or more residues that dot product attention does not capture explicitly. We introduce Higher-Order Modular Attention, HOMA, a unified attention operator that fuses pairwise attention with an explicit triadic interaction pathway. To make triadic attention practical on long sequences, HOMA employs block-structured, windowed triadic attention. We evaluate on three TAPE benchmarks for Secondary Structure, Fluorescence, and Stability. Our attention mechanism yields consistent improvements across all tasks compared with standard self-attention and efficient variants including block-wise attention and Linformer. These results suggest that explicit triadic terms provide complementary representational capacity for protein sequence prediction at controllable additional computational cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。