arXiv:2409.00209cs.CLcs.LG2024-09被引 1

用语义因果图提升大模型事件推理能力,效果远超GPT-4。

Enhancing Event Reasoning in Large Language Models through Instruction Fine-Tuning with Semantic Causal Graphs

  • 构建语义因果图捕捉事件触发词与类型间的因果关系。
  • 在事件触发识别上比标准微调提升35.69%,超越GPT-4平均31.01%。
  • 采用低秩适配保留模型通用推理能力,性能下降仅2.03点。

事件检测与文本推理已在多个领域成为关键应用。尽管大语言模型(LLMs)近期在推理能力上取得显著进展,但在事件检测方面仍表现不佳,主要因缺乏考虑事件触发词与类型间因果关系的训练方法。为此,本文提出一种新的指令微调方法,引入语义因果图(SCGs)以捕获文本中的因果关系和上下文信息。基于SCGs,设计了聚焦事件触发词及其与事件类型关系的SCG指令,并采用低秩适配(LoRA)以保持大模型的通用推理能力。实验表明,使用SCG指令微调的模型在事件触发分类任务上平均优于标准指令微调35.69%。值得注意的是,微调后的Mistral 7B模型在关键事件检测指标上平均超越GPT-4:事件触发识别提升31.01%,事件触发分类提升37.40%,事件分类提升16.43%。对通用能力的分析显示,跨六个基准测试平均性能下降仅2.03分。本研究系统评估了多种大模型在不同数据集、提示策略与训练方法下的事件检测表现。

原文摘要 · Abstract (English)

Event detection and text reasoning have become critical applications across various domains. While LLMs have recently demonstrated impressive progress in reasoning abilities, they often struggle with event detection, particularly due to the absence of training methods that consider causal relationships between event triggers and types. To address this challenge, we propose a novel approach for instruction fine-tuning LLMs for event detection. Our method introduces Semantic Causal Graphs (SCGs) to capture both causal relationships and contextual information within text. Building off of SCGs, we propose SCG Instructions for fine-tuning LLMs by focusing on event triggers and their relationships to event types, and employ Low-Rank Adaptation (LoRA) to help preserve the general reasoning abilities of LLMs. Our evaluations demonstrate that training LLMs with SCG Instructions outperforms standard instruction fine-tuning by an average of 35.69\% on Event Trigger Classification. Notably, our fine-tuned Mistral 7B model also outperforms GPT-4 on key event detection metrics by an average of 31.01\% on Event Trigger Identification, 37.40\% on Event Trigger Classification, and 16.43\% on Event Classification. We analyze the retention of general capabilities, observing only a minimal average drop of 2.03 points across six benchmarks. This comprehensive study investigates multiple LLMs for the event detection task across various datasets, prompting strategies, and training approaches.

事件检测因果推理微调大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。