用大模型从财报电话会中自动提取关键指标，提升企业绩效追踪效率
Effective Performance Measurement: Challenges and Opportunities in KPI Extraction from Earnings Calls
- 基于大模型的开放抽取系统，无需预设标签即可处理非结构化对话
- 人工评估精度达79.7%，在2460组专家标注上验证有效性
- 提出三个新基准，揭示传统财务模型在财报会场景下的泛化缺陷
财报电话会是公开公司财务信息的重要来源，但其内容无标签、非结构化且使用口语化表达，难以自动化解析。本文通过评估基于SEC文件训练的模型在财报会数据上的表现，建立基准。为此引入三个新基准：SEC文件基准（SECB）、财报会基准（ECB）及包含2,460组专家标注的子集ECB-A，用于定性分析。研究发现，编码器类模型在领域迁移中表现不佳。最终提出一种利用大语言模型实现对非结构化通话转录稿的开放式关键绩效指标（KPI）提取系统，经人工评估验证，精确率达79.7%，为该重要领域提供持续追踪新兴指标的基线方案。
原文摘要 · Abstract (English)
Earnings calls are a key source of financial information about public companies. However, extracting information from these calls is difficult. Unlike the templatic filings required by the U.S. Securities and Exchange Commission (SEC) to report a company's financial situation, earnings conference calls have no built-in labels, are unstructured, and feature conversational language. We explore this challenging domain by assessing the information captured by models trained on SEC filings and in-context learning methods. To establish a baseline, we first evaluate the generalization capabilities of SEC-trained models across established SEC datasets. To support our investigation, we introduce three novel benchmarks: (1) SEC Filings Benchmark (SECB), (2) Earnings Calls Benchmark (ECB), and ECB-A, a subset with 2,460 expert annotation groups to support our qualitative analysis. We find that encoder-based models struggle with the domain shift. Finally, we propose a system utilizing LLMs to perform open-ended extraction from unstructured call transcripts, verified by human evaluation (79.7% precision), providing a baseline for this valuable domain through the consistent tracking of emergent KPIs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。