让语言模型与结构模型对齐,提升蛋白质表示效果。
Aligning Large Language Models and Geometric Deep Models for Protein Representation
- 用图神经网络结合三维结构信息增强对齐效果。
- 大语言模型和双层投影头能显著提升对齐质量。
- 稀有蛋白影响对齐性能,需针对性优化。
潜在表示对齐已成为构建多模态大语言模型(MLLM)的基础技术,通过将不同模态的嵌入映射到共享空间,通常与大语言模型(LLM)的嵌入空间对齐,以实现有效的跨模态理解。尽管已有初步面向蛋白质的MLLM出现,但主要依赖启发式方法,缺乏对最优对齐策略的根本理解。本文研究蛋白质领域中LLM与几何深度模型(GDM)的多模态表示对齐问题。我们系统评估了三种先进LLM(Gemma2-2B、LLaMa3.1-8B、LLaMa3.1-70B)与四种蛋白质专用GDM(GearNet、GVP、ScanNet、GAT)。从模型和蛋白质角度分析对齐因素,揭示现有方法的挑战,并提出改进策略。关键发现:融合图结构与3D信息的GDM与LLM对齐更优;更大规模的LLM具备更强对齐能力;蛋白质稀有性显著影响对齐表现。此外,增加GDM嵌入维度、使用两层投影头、在蛋白质数据上微调LLM均能显著提升对齐质量。这些策略为蛋白质多模态模型性能提升提供新路径。代码与数据见 https://github.com/Tizzzzy/LLM-GDM-alignment。
原文摘要 · Abstract (English)
Latent representation alignment has become a foundational technique for constructing multimodal large language models (MLLM) by mapping embeddings from different modalities into a shared space, often aligned with the embedding space of large language models (LLMs) to enable effective cross-modal understanding. While preliminary protein-focused MLLMs have emerged, they have predominantly relied on heuristic approaches, lacking a fundamental understanding of optimal alignment practices across representations. In this study, we explore the alignment of multimodal representations between LLMs and Geometric Deep Models (GDMs) in the protein domain. We comprehensively evaluate three state-of-the-art LLMs (Gemma2-2B, LLaMa3.1-8B, and LLaMa3.1-70B) with four protein-specialized GDMs (GearNet, GVP, ScanNet, GAT). Our work examines alignment factors from both model and protein perspectives, identifying challenges in current alignment methodologies and proposing strategies to improve the alignment process. Our key findings reveal that GDMs incorporating both graph and 3D structural information align better with LLMs, larger LLMs demonstrate improved alignment capabilities, and protein rarity significantly impacts alignment performance. We also find that increasing GDM embedding dimensions, using two-layer projection heads, and fine-tuning LLMs on protein-specific data substantially enhance alignment quality. These strategies offer potential enhancements to the performance of protein-related multimodal models. Our code and data are available at https://github.com/Tizzzzy/LLM-GDM-alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。