用小模型+LoRA实现高效可控蛋白质生成，性能媲美大模型。
Energy Efficient Protein Language Models: Leveraging Small Language Models with LoRA for Controllable Protein Generation
- 基于Llama-3和Phi-3-mini小模型，结合LoRA技术降低参数量至4%。
- 不可控生成平均pLDDT达69.75，可控生成平均TM-Score达0.84。
- 可在低功耗芯片上部署，能效提升3倍，适合边缘计算场景。
大型语言模型在自然语言处理中表现优异，并在蛋白质序列生成等领域展现出潜力。然而，用于NLP的通用小型模型与专用且庞大的蛋白质语言模型之间仍存在显著差异。本文提出两个基于Llama-3-8B和Phi-3-mini的小型蛋白质语言模型，可实现不可控与可控蛋白质生成。在不可控生成任务中，最佳模型平均pLDDT得分为69.75，表明其生成结构稳定；在可控生成任务中，模型根据提示生成目标性质蛋白，平均TM-Score达0.84，结构相似性高。研究扩展了10类属性（含6类酶）以增强能力。采用低秩适配器（LoRA）技术，将可训练参数降至原始模型的4%，显著降低计算开销。使用UniRef50子集与小模型，整体训练时间减少70%且性能无损。其中Phi-3-mini进一步将可训练参数减少60%，训练成本降低30%，达到与Llama-3相当的0.81平均TM-Score。此外，模型成功部署于节能型ET-SoC-1芯片，能效（TPS/W）提升3倍。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated significant success in natural language processing (NLP) tasks and have shown promising results in other domains such as protein sequence generation. However, there remain salient differences between LLMs used for NLP, which effectively handle multiple tasks and are available in small sizes, and protein language models that are often specialized for specific tasks and only exist in larger sizes. In this work, we introduce two small protein language models, based on Llama-3-8B and Phi-3-mini, that are capable of both uncontrollable and controllable protein generation. For the uncontrollable generation task, our best model achieves an average pLDDT score of 69.75, demonstrating robust performance in generating viable protein structures. For the controllable generation task, in which the model generates proteins according to properties specified in the prompt, we achieve a remarkable average TM-Score of 0.84, indicating high structural similarity to target proteins. We chose 10 properties, including six classes of enzymes, to extend the capabilities of prior protein language models. Our approach utilizes the Low-Rank Adaptor (LoRA) technique, reducing trainable parameters to just 4% of the original model size, lowering computational requirements. By using a subset of the UniRef50 dataset and small models, we reduced the overall training time by 70% without compromising performance. Notably, Phi-3-mini reduced trainable parameters by 60%, decreasing training cost by 30% compared to Llama 3. Consequently, Phi-3 achieved a comparable TM-Score of 0.81, demonstrating that smaller models can match the performance of larger ones, like Llama 3. We also demonstrate the deployment of our models on the energy efficient ET-SoC-1 chip, significantly improving the TPS/W by a factor of 3.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。