金融实体识别在领域迁移下可靠性下降,该研究提出可信度评估与选择性预测方案。
Reliable Financial Named Entity Recognition Under Domain Shift: Confidence Estimation and Selective Prediction

- 采用五种置信度信号评估模型输出可靠性,识别分布偏移下的错误预测
- 在高置信度40%样本上,误句率从34.3%降至2%以下,新闻数据仍有效
- 自一致性置信度更鲁棒且无需后处理校准,适合部署前检测领域偏移
金融AI系统常在特定文本类型上训练信息抽取模型,并部署于招股说明书、财经新闻和社交媒体等不同来源。标准F1指标无法反映输入分布变化时预测的可靠性。本文在三阶段压力测试中评估金融命名实体识别(NER)的置信度估计与选择性预测:涵盖美国证监会文件、财经新闻及通用话题社交媒体(极端域外情况)。使用BERT标签器与LoRA微调的Qwen2.5-0.5B/1.5B模型,结合五种推理时置信度信号、三个训练种子与自助区间进行评估。结果表明:整体输出概率在域内最强但域外失效;实体跨度概率与自一致性更稳健,其中自一致性无需后处理校准且表现更优。选择性弃权可将域内句子错误率从34.3%降至2%以下,对财经新闻仍有效,但在极端社交媒体迁移下无法恢复可用干净子集。研究建议采用分阶段部署策略,在上游检测严重分布偏移后再应用预测级置信度过滤。
原文摘要 · Abstract (English)
Financial AI systems often train information extractors on one textual register and deploy them across filings, news, and user-generated content, and standard F1 scores do not indicate which predictions remain safe to automate when that input distribution changes. We study confidence estimation and selective prediction for financial named entity recognition (NER) on a three-tier stress test spanning SEC filings, financial news, and general-topic social media as an extreme out-of-domain condition, evaluating a BERT tagger and LoRA-tuned Qwen2.5-0.5B/1.5B models with five inference-time confidence signals, three training seeds, and bootstrap intervals. Confidence rankings themselves change under shift: whole-output probability is the strongest in-domain error detector but deteriorates out of domain, whereas entity-span probability and self-consistency are more robust; self-consistency is also better calibrated without post-hoc fitting. Abstention reduces sentence error from 34.3% to below 2% on the highest-confidence 40% of in-domain inputs and remains useful on financial news, but recovers no usefully large clean subset under the extreme social-media shift. These results motivate a staged deployment strategy that detects severe distribution shift upstream before applying prediction-level confidence gating.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。