arXiv:2411.14460cs.CLcs.AI2024-11被引 2

让大模型统一理解表格、图谱等结构化数据,提升问答与对话能力。

LLaSA: Large Language and Structured Data Assistant

  • 用统一超图格式表示各类结构化数据,实现跨类型处理。
  • 自监督预训练超图编码器,可适配多种大模型并提升性能。
  • 轻量微调(LoRA)效果超越全参数调优的现有方法。

结构化数据(如表格、图谱、数据库)在自然语言处理任务中至关重要,例如问答和对话系统。受视觉-语言模型启发,图神经网络(GNNs)被引入大语言模型(LLMs)输入以增强结构化知识定位(SKG)能力。然而现有方法存在两大局限:(1) 使用不同GNN建模不同类型数据,难以统一处理;(2) GNN预训练与特定LLM耦合,无法充分对齐文本空间,限制其在其他模型上的迁移能力。为此,我们提出大型语言与结构化数据助手(LLaSA),一种通用框架,用于增强LLMs对结构化数据的处理能力。具体地,我们将多种结构化数据统一表示为超图格式,通过自监督学习预训练超图编码器,并使用G-Former通过交叉注意力压缩超图表征。压缩后的表征在训练和推理阶段被拼接到序列化输入中。多个SKG任务的实验表明,预训练的超图编码器可适配多种LLMs,显著提升其处理各类结构化数据的能力。此外,结合LoRA微调的LLaSA优于采用全参数微调的先前最优方法。

原文摘要 · Abstract (English)

Structured data, such as tables, graphs, and databases, play a critical role in plentiful NLP tasks such as question answering and dialogue system. Recently, inspired by Vision-Language Models, Graph Neutral Networks (GNNs) have been introduced as an additional modality into the input of Large Language Models (LLMs) to improve their performance on Structured Knowledge Grounding (SKG) tasks. However, those GNN-enhanced LLMs have the following limitations: (1) They employ diverse GNNs to model varying types of structured data, rendering them unable to uniformly process various forms of structured data. (2) The pretraining of GNNs is coupled with specific LLMs, which prevents GNNs from fully aligning with the textual space and limits their adaptability to other LLMs. To address these issues, we propose \textbf{L}arge \textbf{L}anguage and \textbf{S}tructured Data \textbf{A}ssistant (LLaSA), a general framework for enhancing LLMs' ability to handle structured data. Specifically, we represent various types of structured data in a unified hypergraph format, and use self-supervised learning to pretrain a hypergraph encoder, and a G-Former compressing encoded hypergraph representations with cross-attention. The compressed hypergraph representations are appended to the serialized inputs during training and inference stages of LLMs. Experimental results on multiple SKG tasks show that our pretrained hypergraph encoder can adapt to various LLMs and enhance their ability to process different types of structured data. Besides, LLaSA, with LoRA fine-tuning, outperforms previous SOTA method using full parameters tuning.

大模型结构化数据超图多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。