AI RAG工具可让银行信息披露任务提速10倍，交互式使用效果更佳。
Efficacy of AI RAG Tools for Complex Information Extraction and Data Annotation Tasks: A Case Study Using Banks Public Disclosures
- 用交互式AI工具辅助信息提取，提升准确率与效率。
- 相比纯人工，任务耗时最多减少268小时，提速达10倍。
- 熟练使用AI的分析员表现更好，工具能力与人效紧密相关。
我们采用被试内设计，随机分配任务，研究AI检索增强生成（RAG）工具在复杂信息提取与数据标注任务中的有效性。基于全球系统重要性银行（GSIBs）数千页公开披露文件，复现了一项具有多层级复杂标准的真实世界标注任务。测试两种条件：一是仅使用工具并接受首个答案的“原始”使用模式；二是允许分析员交互式调用工具并自主判断是否补充信息的“交互”模式。相比纯人工基准，使用AI工具使任务执行速度最高提升10倍，尤其在交互模式下准确率显著提高。外推至完整任务后，该方法可节省最多268小时。结果还表明，分析员的技能不仅限于领域知识，还包括对AI工具的熟练程度，两者均影响任务的速度与准确性。
原文摘要 · Abstract (English)
We utilize a within-subjects design with randomized task assignments to understand the effectiveness of using an AI retrieval augmented generation (RAG) tool to assist analysts with an information extraction and data annotation task. We replicate an existing, challenging real-world annotation task with complex multi-part criteria on a set of thousands of pages of public disclosure documents from global systemically important banks (GSIBs) with heterogeneous and incomplete information content. We test two treatment conditions. First, a "naive" AI use condition in which annotators use only the tool and must accept the first answer they are given. And second, an "interactive" AI treatment condition where annotators use the tool interactively, and use their judgement to follow-up with additional information if necessary. Compared to the human-only baseline, the use of the AI tool accelerated task execution by up to a factor of 10 and enhanced task accuracy, particularly in the interactive condition. We find that when extrapolated to the full task, these methods could save up to 268 hours compared to the human-only approach. Additionally, our findings suggest that annotator skill, not just with the subject matter domain, but also with AI tools, is a factor in both the accuracy and speed of task performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。