用大模型自动核查公司事件数据缺失,提升金融数据库准确性。
DelistBench: Evaluating Search-Enabled LLMs for Auditable Corporate-Event Database Completion

- 让大模型通过搜索公开信息重建企业事件记录,实现数据库审计。
- 联网检索使事件日期准确率提升34至48个百分点,最佳系统达81.5%。
- 低成本系统可接近顶尖性能,适合高风险场景优先审查。
金融机构需要独立检测供应商数据库中遗漏、过时或分类错误的企业事件记录。我们提出「Search-to-Record」任务,即利用具备搜索能力的大语言模型,基于公开来源为已知证券池和历史截止时间重建机构定义的事件记录,并构建了包含1,200条记录的DelistBench基准,专门用于证券级退市公告的评估。我们在闭卷与联网两种条件下评估了五种模型。联网访问使七天内事件日期准确率提升34.0至48.0个百分点,事件状态准确率提升约2.8至21.7个百分点;最优系统在七天内达到81.5%的整体联合准确率。经济型网络系统以4.5-6.6%的API成本,实现75.9-78.3%的整体联合准确率。基于风险的筛选可识别低错误子集,但最高覆盖率操作点仍需人工审查27.3%的测试集。评估表明,网络检索是时间精度提升的主要来源,且低成本系统可逼近最优系统表现。综上,Search-to-Record、DelistBench及评估结果提供了明确部署建议:根据本地事件发生率与市场结构校准筛选策略,保持正向事件召回率,并将正向与模糊案例导向针对性审查。
原文摘要 · Abstract (English)
Financial institutions need an independent way to detect missing, stale, and misclassified corporate-event records in vendor databases. We introduce Search-to-Record, a database-assurance task in which search-enabled large language models reconstruct institution-defined event records from public sources for a known security universe and historical cutoff, and DelistBench, a 1,200-record benchmark for security-level delisting announcements. We evaluate five models in paired closed-book and web-enabled conditions. Web access raises announcement-date accuracy within seven days by 34.0 to 48.0 percentage points and event-status accuracy by approximately 2.8 to 21.7 points; the best system achieves 81.5% overall joint accuracy within seven days. Economy web systems achieve 75.9-78.3% overall joint accuracy within seven days at 4.5-6.6% of the API cost of the most expensive web system. Risk-based triage identifies low-error subsets, although the highest-coverage operating point still sends 27.3% of the balanced test set to review. The evaluation identifies web retrieval as the main source of timing gains and shows that low-cost systems can approach the best system's accuracy. Together, Search-to-Record, DelistBench, and the evaluation provide concrete deployment guidance: calibrate triage to local event prevalence and market mix, preserve positive-event recall, and route positive and ambiguous cases to targeted review.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。