arXiv:2410.11235cs.CL2024-10被引 2

用大模型统一编码文本与图数据,提升信息检索和问答效果

GT2Vec: Large Language Models as Multi-Modal Encoders for Text and Graph-Structured Data

  • 用MLP适配器将图嵌入映射到文本空间,让大模型同时处理双模态数据
  • 在6个数据集上优于基线,知识图谱问答任务提升达12.3%
  • 适合需要融合结构化关系与自然语言的场景,如智能客服、推荐系统

图结构信息蕴含丰富的上下文语义,可增强语言模型对关系与层级的表达能力,从而提升检索、问答与分类等应用性能。然而,现有融合文本与图嵌入的方法多基于MLP或浅层Transformer,难以充分挖掘异构模态的潜力。为此,本文提出GT2Vec,一种利用大语言模型(LLM)联合编码文本与图数据的简单有效框架。具体地,通过一个MLP适配器将图嵌入投影至与文本嵌入同空间,使LLM能协同处理双模态输入;同时引入对比学习,进一步对齐图文空间,提升联合嵌入质量。在涵盖三个任务(知识图谱上下文问答、图-文本对分类、检索)的六个数据集上的实证结果表明,GT2Vec持续优于现有基线,显著提升多个数据集表现。消融实验进一步验证了方法有效性。

原文摘要 · Abstract (English)

Graph-structured information offers rich contextual information that can enhance language models by providing structured relationships and hierarchies, leading to more expressive embeddings for various applications such as retrieval, question answering, and classification. However, existing methods for integrating graph and text embeddings, often based on Multi-layer Perceptrons (MLPs) or shallow transformers, are limited in their ability to fully exploit the heterogeneous nature of these modalities. To overcome this, we propose GT2Vec, a simple yet effective framework that leverages Large Language Models (LLMs) to jointly encode text and graph data. Specifically, GT2Vec employs an MLP adapter to project graph embeddings into the same space as text embeddings, allowing the LLM to process both modalities jointly. Unlike prior work, we also introduce contrastive learning to align the graph and text spaces more effectively, thereby improving the quality of learned joint embeddings. Empirical results across six datasets spanning three tasks, knowledge graph-contextualized question answering, graph-text pair classification, and retrieval, demonstrate that GT2Vec consistently outperforms existing baselines, achieving significant improvements across multiple datasets. These results highlight GT2Vec's effectiveness in integrating graph and text data. Ablation studies further validate the effectiveness of our method.

图神经网络大模型多模态编码

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。