用知识图谱增强大模型，提升财报问答准确率
Multi-Document Financial Question Answering using LLMs
- 用语义标签+检索增强生成，构建两种财报问答方法
- 新方法在111个复杂问题上显著优于基础RAG
- 适合金融分析、智能投研等需要深度理解的场景
我们提出两种新的多文档财务问答方法。第一种基于语义标签，通过查询索引获取上下文（RAG_SEM）；第二种基于知识图谱（KG_RAG），利用语义标签从图数据库中检索三元组作为上下文。KG_RAG使用小模型通过知识蒸馏训练得到，其知识图谱由大教师模型指导生成。数据包含苹果、微软、谷歌、英伟达、亚马逊和特斯拉2021至2023年共18份1万字级报告，涵盖111个复杂且不易回答的问题。评估指标包括整体得分与分项得分，如忠实性、相关性、正确性、相似度、大模型综合评分及ROUGE分数，以及嵌入相似度。结果表明，两种方法均显著优于普通RAG；KG_RAG在九项指标中的四项表现更优。
原文摘要 · Abstract (English)
We propose two new methods for multi-document financial question answering. First, a method that uses semantic tagging, and then, queries the index to get the context (RAG_SEM). And second, a Knowledge Graph (KG_RAG) based method that uses semantic tagging, and, retrieves knowledge graph triples from a graph database, as context. KG_RAG uses knowledge graphs constructed using a small model that is fine-tuned using knowledge distillation using a large teacher model. The data consists of 18 10K reports of Apple, Microsoft, Alphabet, NVIDIA, Amazon and Tesla for the years 2021, 2022 and 2023. The list of questions in the data consists of 111 complex questions including many esoteric questions that are difficult to answer and the answers are not completely obvious. As evaluation metrics, we use overall scores as well as segmented scores for measurement including the faithfulness, relevance, correctness, similarity, an LLM based overall score and the rouge scores as well as a similarity of embeddings. We find that both methods outperform plain RAG significantly. KG_RAG outperforms RAG_SEM in four out of nine metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。