构建首个跨机构银行财报问答数据集,提升金融文本理解能力。
Enhancing Financial Question Answering: A Novel Benchmark Dataset of Banks' financial statements
- 设计跨机构多银行财报问答数据集,覆盖24家银行198k词级长文档
- 引入上下文增强与优化嵌入模型,检索性能NDCG@10提升至0.710
- 验证单块最佳生成策略,适合金融智能分析与合规研究者使用
由于银行财报在结构、语言和数值内容上具有复杂性、冗长性和跨司法管辖区的异质性,自动化问答系统面临巨大挑战。本文提出FinRAG-QA,一个新型金融问答基准数据集,包含999个从业实践者标注的问题,围绕10项标准化指标,基于24家欧美主要银行2019-2023年的209份年报与第三支柱报告构建。该数据集覆盖文档平均长达198,000字,超过现有任何金融QA资源。我们评估了一个多阶段RAG管道,并分离各组件贡献:结合上下文分块增强与检索优化嵌入模型后,NDCG@10从0.322提升至0.710;在召回真值的前提下,推理优化生成器使准确率从44.6%升至79.0%(+34.4个百分点),但代价是约20倍的生成延迟。实验还表明,当第一阶段排序已强时,交叉编码器重排会降低检索效果;且生成时单个顶级片段优于更大上下文。实验于2024年底至2025年初进行,使用当时可用模型。
原文摘要 · Abstract (English)
The comparative analysis of banks' financial statements poses significant challenges for automated question answering systems due to their complexity, substantial length, technical language, and inhomogeneity of both textual and numerical content across different jurisdictions and institutions. We introduce FinRAG-QA, a novel benchmark dataset for financial question answering, which comprises 999 practitioner-curated questions on 10 standardised indicators, grounded in 209 annual and Pillar 3 reports from 24 major European and U.S. banks spanning 2019-2023. Unlike prior financial QA benchmarks, which centre on U.S. filings and single-institution analysis, FinRAG-QA targets cross-institutional retrieval over documents averaging 198k words, longer than any existing financial QA resource. On this benchmark we evaluate a multi-stage RAG pipeline and isolate the contribution of each component. Contextual chunk enrichment combined with a retrieval-optimised embedding model raises NDCG@10 from 0.322 to 0.710; conditional on the ground truth being retrieved, a reasoning-optimised generator raises answer accuracy from 44.6% to 79.0% (+34.4 percentage points), at roughly 20x the generation latency. We further show that cross-encoder reranking degrades retrieval when the first-stage ranking is already strong, and that a single top-ranked chunk outperforms larger contexts at generation time. Experiments were run in late 2024-early 2025 with the models available at that time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。