通过人机协作与检索增强，实现监管行业文档分类的低人力依赖高准确率。
HIRA: A Human-in-the-Loop Retrieval-Augmented Cascade for Document Classification in Regulated Industries
- 用检索融合多模态信息，仅在不确定时触发本地大模型验证
- 仅需6.4%人工修正即达0.8548宏平均F1，远超零样本基线
- 适合数据受限、合规要求严的金融等监管领域应用
监管行业文档分类面临数据本地化、冷启动标签少、审核资源有限和模型治理成本高等挑战。我们提出HIRA，一种无需训练、部署于本地的检索增强级联框架，结合OCR文本的BM25、密集文本嵌入和图像级表示，通过校准后的加权倒数排名融合进行分类。高置信度文档直接由检索结果判定；不确定或视觉易混淆文档则交由本地部署的大语言模型验证器处理，输入包括OCR文本、检索样例、标签说明及混淆关键词。若验证器仍不确定，则提交人工审查。每次修正均作为带权重的检索样例存入，并更新狄利克雷平滑的混淆图谱，实现系统优化而无需重训模型。在包含30,233份文档的私有80类贸易金融语料上，HIRA处理全量数据仅需1,945次（6.4%）人工修正，宏观F1从0.6218提升至0.8548。在Tobacco-3482基准测试中，使用本地DeepSeek-R1-Distill-Qwen-32B验证器，达到0.9423宏平均F1，比零样本大模型基线高出17.4个百分点，且仅对约40%文档调用验证器，减少约60%大模型调用。仅用518次人工修正（占24.8%），便达到全标注池最优性能（2,086个文档），证明选择性人工反馈与检索记忆自适应可替代反复模型重训，是长尾文档分类在监管环境下的可行方案。
原文摘要 · Abstract (English)
Document classification in regulated industries is constrained by data residency, limited cold-start labels, scarce review capacity, and costly model-governance procedures. We present HIRA, a training-free, on-premises retrieval-augmented cascade for document classification in regulated deployments that combines BM25 over OCR text, dense text embeddings, and image-level representations through validation-calibrated weighted reciprocal-rank fusion. Confident documents are classified directly by retrieval; uncertain or visually confusable documents are passed to a locally hosted LLM verifier, which receives the OCR text, retrieved exemplars, label descriptions, and confusion-specific terms. When the verifier remains uncertain, the document is sent to human review. Each correction is stored as a margin-weighted retrieval exemplar and updates a Dirichlet-smoothed confusion graph, letting the system improve without updating model weights. On a private 80-class trade-finance corpus, HIRA processes the full 30,233-document production stream while requesting human correction for only 1,945 documents (6.4%), improving Macro-F1 from 0.6218 to 0.8548. On the corrected Tobacco-3482 benchmark, HIRA reaches 0.9423 Macro-F1 with a locally hosted DeepSeek-R1-Distill-Qwen-32B verifier, 17.4 percentage points above the zero-shot LLM baseline, while invoking the verifier for only about 40% of documents and reducing LLM calls by approximately 60%. With 518 human corrections (24.8% of the pool), HIRA matches the fully labelled pool oracle, in which all 2,086 pool documents are indexed with their ground-truth labels. These results show that selective human feedback and retrieval-memory adaptation can be a practical alternative to repeated model retraining for long-tail document classification in regulated deployments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。