Proust让蛋白语言模型既能高效预测又能生成，打破传统分工。
No Generation without Representation: Efficient Causal Protein Language Models Enable Zero-Shot Fitness Estimation
- 采用分组查询注意力等创新架构，309M参数实现因果建模
- 在替换和插入缺失任务上达到新纪录，仅用1/20计算量超越大模型
- 兼具生成能力与可解释性，适合需要双向功能的生物设计场景
蛋白质语言模型（PLMs）面临根本矛盾：掩码语言模型（MLMs）擅长适应度预测，而因果模型支持生成，迫使使用者维护独立架构。我们提出 extbf{Proust}，一个309M参数的因果PLM，通过借鉴最新大模型研究的架构创新——包括共享键值投影的分组查询注意力、跨层值残差及深度因果卷积——弥合这一鸿沟。在40 B200 GPU小时内训练了330亿个标记。Proust在ProteinGym替换任务上取得Spearman $ρ=0.390$，与需50–200倍算力的MLMs性能相当；在插入缺失任务上创出新纪录，优于最大达20倍的模型。在EVEREST病毒适应度基准上，其表现接近依赖结构的方法，仅使用序列信息。强大的表示能力使其同时保留原生生成能力，这是MLMs因设计缺陷所不具备的。可解释性分析显示，位置熵方差可在一定程度上预测检索增强何时有效或失效。此类洞见随规模扩大将不断深化，可支撑测试时扩展等能力。代码与权重见 https://github.com/Furkan9015/proust-inference
原文摘要 · Abstract (English)
Protein language models (PLMs) face a fundamental divide: masked language models (MLMs) excel at fitness prediction while causal models enable generation, forcing practitioners to maintain separate architectures. We introduce \textbf{Proust}, a 309M-parameter causal PLM that bridges this gap through architectural innovations adapted from recent LLM research, including grouped-query attention with shared K/V projections, cross-layer value residuals, and depthwise causal convolutions. Trained on 33B tokens in 40 B200 GPU-hours, Proust achieves Spearman $ρ= 0.390$ on ProteinGym substitutions, competitive with MLMs requiring 50--200$\times$ the compute. On indels, Proust sets a new state-of-the-art, outperforming models up to 20$\times$ larger. On EVEREST viral fitness benchmarks, it approaches structure-aware methods using sequence alone. These powerful representations position Proust in a sweet spot as it also retains native generative capabilities that MLMs lack by design. Interpretability analysis reveals that per-position entropy variance predicts, to an extent, when retrieval augmentation helps and hurts. Such insights can grow in both quantity and quality at scale and inform capabilities such as test-time scaling. Code and weights are available at https://github.com/Furkan9015/proust-inference
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。