大模型能否利用结构化语义表示？长文本任务中表现更优。
Can LLMs Interpret and Leverage Structured Linguistic Representations? A Case Study with AMRs
- 用抽象语义表示（AMR）编码上下文，测试大模型理解能力
- 长文本任务中性能提升，如对话摘要零样本相似度从66%升至76%
- 新大模型效果更好，且能从线性AMR重构原文，相似度达81%
本文评估大型语言模型（LLMs）利用结构化语言表示形式中的上下文信息的能力。具体而言，我们考察了在多种语言任务中，使用抽象语义表示（AMR）对短和长上下文进行编码的影响。实验基于8位量化且经过指令微调的Llama 3.1 (8B)、Phi-3和Mistral 7B模型。结果显示，对于涉及短上下文的任务，将原始上下文的AMR加入提示通常会降低底层模型性能；但对于涉及长上下文的任务（如SAMSum数据集上的对话摘要），这种增强可显著提升模型表现，例如使Llama 3.1的零样本余弦相似度从66%提高到76%。该提升在较新、较大的模型中更为明显，但不适用于旧版或小型模型。此外，我们发现模型能有效从线性化AMR重建原文，在最佳情况下余弦相似度达到81%。
原文摘要 · Abstract (English)
This paper evaluates the ability of Large Language Models (LLMs) to leverage contextual information in the form of structured linguistic representations. Specifically, we examine the impact of encoding both short and long contexts using Abstract Meaning Representation (AMR) structures across a diverse set of language tasks. We perform our analysis using 8-bit quantized and instruction-tuned versions of Llama 3.1 (8B), Phi-3, and Mistral 7B. Our results indicate that, for tasks involving short contexts, augmenting the prompt with the AMR of the original language context often degrades the performance of the underlying LLM. However, for tasks that involve long contexts, such as dialogue summarization in the SAMSum dataset, this enhancement improves LLM performance, for example, by increasing the zero-shot cosine similarity score of Llama 3.1 from 66% to 76%. This improvement is more evident in the newer and larger LLMs, but does not extend to the older or smaller ones. In addition, we observe that LLMs can effectively reconstruct the original text from a linearized AMR, achieving a cosine similarity of 81% in the best-case scenario.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。