基于三维结构设计掩码策略，提升蛋白质语言模型对远距离相互作用的建模能力。
Structure-Aware Masking for Protein Representation Learning
- 按三维空间邻近度分组掩码，优先掩盖结构耦合区域。
- 在4个下游任务中最高提升14%，尤其擅长预测高阶突变互作。
- 证明掩码位置是关键诱导偏置，非掩码跨度所致。
掩蔽语言建模（MLM）是训练蛋白质语言模型的标准目标，通常以固定比例（如15%）随机掩蔽单个残基。这一做法隐含假设所有序列位置对表征学习贡献相等。然而，在下游功能预测任务中，蛋白质序列受三维结构依赖性和长程残基接触支配，产生强烈的非局部耦合。本文提出分桶掩码（Bucket Masking），根据残基在三维空间中的接近程度选择残基组，训练时优先掩蔽结构耦合区域。通过将掩码分布与残基接触关联，该策略引导学习目标聚焦于对蛋白质功能至关重要的长程交互。在四个下游蛋白质功能预测任务中，分桶掩码相比标准随机掩码最高提升14%，尤其在预测高阶突变互作方面表现优异。通过受控消融实验，我们证实这些提升源于掩码位置而非掩码跨度，确立了掩码作为位置归纳偏置的作用。
原文摘要 · Abstract (English)
Masked language modeling (MLM) is the standard objective for training protein language models, typically implemented by randomly masking individual residues at a fixed rate (e.g., 15%). This practice implicitly assumes that all sequence positions contribute equally to representation learning. In downstream fitness prediction tasks, however, protein sequences are governed by three-dimensional structural dependencies and long-range residue contacts that induce strong nonlocal couplings between residues. We introduce Bucket Masking, a structure-aware masking strategy that selects groups of residues based on their proximity in three-dimensional space, preferentially masking structurally coupled regions during training. By conditioning the masking distribution on residue contacts, Bucket Masking shifts the learning objective toward modeling long-range interactions that are critical for protein function. Across four downstream protein fitness prediction tasks, Bucket Masking enables up to a 14% improvement over standard random masking, excelling at predicting higher-order mutational interactions. Through controlled ablations, we show that these improvements arise from mask placement rather than span size, establishing masking as a positional inductive bias.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。