arXiv:2506.20821cs.CLcs.AI2025-06被引 15

针对金融文档的多模态问答,提出高效精准的检索生成框架。

MultiFinRAG: An Optimized Multimodal Retrieval-Augmented Generation (RAG) Framework for Financial Question Answering

  • 用轻量多模态模型批量处理表格图像,生成结构化数据与摘要
  • 通过模态感知相似度阈值索引,实现跨模态精准检索
  • 动态调用图文上下文,在普通硬件上超越免费版GPT-4o

金融文档如10-K、10-Q和投资者演示文稿通常长达数百页,包含密集文本、结构化表格和复杂图表等多模态内容。回答此类文档中的问题需跨模态联合推理,传统大语言模型(LLMs)和检索增强生成(RAG)管道因令牌限制、布局丢失和跨模态上下文碎片化而受限。我们提出MultiFinRAG,一个专为金融问答设计的多模态检索增强生成框架。该框架首先通过批量处理表格和图表图像,利用轻量级、量化后的开源多模态LLM提取结构化JSON输出和简洁文本摘要。这些输出连同叙述性文本一同嵌入并索引,采用模态感知相似度阈值以实现精确检索。当必要时,采用分层回退策略动态扩展至仅文本、文本+表格+图像的上下文,支持跨模态推理的同时减少无关信息。尽管运行在普通硬件上,MultiFinRAG在涉及文本、表格、图像及综合多模态推理的复杂金融问答任务中,准确率比ChatGPT-4o(免费版)高出19个百分点。

原文摘要 · Abstract (English)

Financial documents--such as 10-Ks, 10-Qs, and investor presentations--span hundreds of pages and combine diverse modalities, including dense narrative text, structured tables, and complex figures. Answering questions over such content often requires joint reasoning across modalities, which strains traditional large language models (LLMs) and retrieval-augmented generation (RAG) pipelines due to token limitations, layout loss, and fragmented cross-modal context. We introduce MultiFinRAG, a retrieval-augmented generation framework purpose-built for financial QA. MultiFinRAG first performs multimodal extraction by grouping table and figure images into batches and sending them to a lightweight, quantized open-source multimodal LLM, which produces both structured JSON outputs and concise textual summaries. These outputs, along with narrative text, are embedded and indexed with modality-aware similarity thresholds for precise retrieval. A tiered fallback strategy then dynamically escalates from text-only to text+table+image contexts when necessary, enabling cross-modal reasoning while reducing irrelevant context. Despite running on commodity hardware, MultiFinRAG achieves 19 percentage points higher accuracy than ChatGPT-4o (free-tier) on complex financial QA tasks involving text, tables, images, and combined multimodal reasoning.

金融AI多模态RAG问答系统

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。