构建首个企业年报长文本金融信息提取基准，支持精准检索与量化分析。
LEDGER: A Long-Context Benchmark of Corporate Annual Reports for Grounded Financial Retrieval and Extraction
- 基于4999份完整年报，含图表与正文，构建长文本金融信息抽取任务。
- 提供31项财务指标标注，覆盖11.8万条自然语言问题的精准检索评估。
- 支持对话式查找与财报内容关联市场反应，适合金融+AI研究者使用。
金融报告是大模型能力的理想测试场，而近期各类规模模型在长上下文处理上的进步，使该领域亟需更严格的评估体系。现有公开资源多将任务简化为仅含少量问答对的SEC 10-K文本。我们发布LEDGER（长上下文文档接地提取与检索评估），包含4,999份数字化企业年报——完整文档含图表、表格与叙述内容，非仅监管文件。每份报告标注31个整合财务关键绩效指标（KPI），并关联财报发布后的市场反应。据此构建三个难度梯度的评测基准：基于TREC风格相关性判断的页面级KPI检索任务（118,048个自然语言问题）、对话式“大海捞针”单值查找任务，以及从数值密集型长文本中完成完整KPI提取任务。此外，提供人类校对级别的OCR标注、标注者间一致性数据及完整的提取、验证与评分工具链。通过案例研究进一步展示数据集价值：将董事长信件修辞与发布后市场影响关联分析。
原文摘要 · Abstract (English)
Finance reporting is a natural proving ground for large language models, and the very-long-context capabilities of recent models across all sizes make rigorous evaluation in this domain an increasingly pressing need. Yet most public financial resources reduce the task to plain-text SEC 10-K filings paired with a handful of question-answer items. We release LEDGER (Long-context Evaluation of Documents for Grounded Extraction and Retrieval), a corpus of 4,999 digitized corporate annual reports - full documents with figures, tables, and narrative, not just regulatory filings. Each report is labeled with 31 consolidated financial KPIs to be extracted and linked to the market's reaction at the earnings date. From this data we derive three evaluation benchmarks spanning the difficulty spectrum: a pure page-level KPI retrieval task with TREC-style relevance judgments over 118,048 questions in natural language, a conversational "needle-in-a-haystack" single-value lookup, and a full KPI extraction task, both from long, numerically dense reports. We additionally provide human OCR-quality annotations with inter-annotator agreement and the complete extraction, validation, and scoring toolchain. We further demonstrate the dataset's research utility with a case study linking CEO-letter rhetoric to post-publication market impact.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。