推理时加丢弃层，零样本预测蛋白适应度更准
Protein Language Model Zero-Shot Fitness Predictions are Improved by Inference-only Dropout
- 推理时在嵌入层与Transformer间加丢弃层
- 0.1丢弃率下多个模型性能提升
- 无需重训练，适合快速部署的生物计算
蛋白质语言模型（PLMs）如ESM2已证明可实现蛋白质关键标量属性（适应度）的零样本预测。本文表明，在PLM的特征提取器/嵌入层与Transformer之间引入推理时的丢弃层，并像蒙特卡洛丢弃一样对输出进行平均，可在ProteinGym数据集的一个子集上提升零样本性能。这一方法即使模型最初未使用丢弃训练，也无需重新训练或微调即可生效。0.1的丢弃率在所有测试模型中表现良好。
原文摘要 · Abstract (English)
Protein Language Models (PLMs) such as ESM2 have been shown to be capable of zero-shot prediction of critical scalar properties of proteins (fitness). In this work, we show that injecting a dropout layer at inference time between a PLM's featurizer/embedding layer and its transformer, and averaging its output akin to Monte-Carlo dropout increases zero-shot performance on a subset of the ProteinGym dataset. This is the case even when the model was not trained with dropouts to begin with, and does not require retraining or finetuning of the PLM. A dropout of 0.1 seems performant across all models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。