用知识图谱增强大模型,提升金融文本中的数值推理准确率。
Structure First, Reason Next: Enhancing a Large Language Model using Knowledge Graph for Numerical Reasoning in Financial Documents
- 先构建文档专属知识图谱,再结合大模型进行推理
- 在FinQA数据集上使数值计算准确率提升约12%
- 适合需要高精度财务分析的从业者和研究者
数值推理是金融文档分析中的关键任务,有助于从金融文本中理解并逻辑推导出数值结论。尽管大语言模型(LLM)在问答系统中展现出强大的逻辑推理能力,但在处理金融报告中的复杂数字时仍面临挑战:从非结构化文本和半结构化表格中提取数值、并可靠完成计算,仍是当前主流LLM的瓶颈。已有研究表明,引入结构化数据如知识图谱(KG)可显著提升LLM的预测性能与解释性。为此,本文提出一种框架,通过从待处理文档中自动生成特定模式的知识图谱,融合到大模型推理流程中,以支持金融领域的数值推理。在基准数据集FinQA上,使用开源模型Llama 3.1 8B Instruct进行评估,结果显示该框架相较原生LLM将执行准确率提升了约12%。
原文摘要 · Abstract (English)
Numerical reasoning is an important task in the analysis of financial documents. It helps in understanding and performing numerical predictions with logical conclusions for the given query seeking answers from financial texts. Recently, Large Language Models (LLMs) have shown promising results in multiple Question-Answering (Q-A) systems with the capability of logical reasoning. As documents related to finance often consist of long and complex financial contexts, LLMs appear well-suited for building high-quality automated financial question-answering systems. However, LLMs often face challenges in accurately processing the various numbers within financial reports. Extracting numerical data from unstructured text and semi-structured tables, and reliably performing accurate calculations, remains a significant bottleneck for numerical reasoning in most state-of-the-art LLMs. Recent studies have shown that structured data augmentations, such as Knowledge Graphs (KGs), have notably improved the predictions of LLMs along with logical explanations. Thus, it is an important requirement to consider inherent structured information in financial reports while using LLMs for various financial analytics. This paper proposes a framework to incorporate structured information using KGs along with LLM predictions for numerical reasoning tasks. The KGs are extracted using a proposed schema inherently from the document under processing. We evaluated our proposed framework over the benchmark data FinQA, using an open-source LLM, namely Llama 3.1 8B Instruct. We observed that the proposed framework improved execution accuracy by approximately 12% relative to the vanilla LLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。