无需训练的文档图文问答系统,提升可解释性与可信度。
DLaVA: Document Language and Vision Assistant for Answer Localization with Enhanced Interpretability and Trustworthiness
- 用唯一框ID组织文本区域,免去反复OCR和思维链推理。
- 在基准数据集上表现媲美顶尖方法,计算复杂度显著降低。
- 融合空间与文本双指标评估,减少AI幻觉,适合高风险场景。
文档视觉问答(VQA)需要强大的文本检测、识别与空间推理能力来理解复杂文档布局。本文提出DLaVA,一种无需训练的新型流水线,利用多模态大语言模型实现零样本答案定位,以提升可信度、可解释性与可说明性。通过创新的无OCR方法,以独特边界框ID组织文本区域,保留空间上下文,避免迭代式OCR或思维链推理,大幅降低计算复杂度。我们进一步改进评估协议,引入交并比(IoU)与平均归一化莱文施泰因相似度(ANLS)双重指标,确保不仅关注文本准确性,也考量空间准确性,从而有效降低AI幻觉风险,增强可信度。在基准数据集上的实验表明,DLaVA性能媲美当前最优技术,计算开销显著更低,且在高可靠性要求的应用中表现出更优精度与稳定性。本研究使用的代码与数据集已开源:https://github.com/ahmad-shirazi/AnnotMLLM。
原文摘要 · Abstract (English)
Document Visual Question Answering (VQA) demands robust integration of text detection, recognition, and spatial reasoning to interpret complex document layouts. In this work, we introduce DLaVA, a novel, training-free pipeline that leverages Multimodal Large Language Models (MLLMs) for zero-shot answer localization in order to improve trustworthiness, interpretability, and explainability. By leveraging an innovative OCR-free approach that organizes text regions with unique bounding box IDs, the proposed method preserves spatial contexts without relying on iterative OCR or chain-of-thought reasoning, thus substantially reducing the computational complexity. We further enhance the evaluation protocol by integrating Intersection over Union (IoU) metrics alongside Average Normalized Levenshtein Similarity (ANLS), thereby ensuring that not only textual accuracy is considered, but spatial accuracy is taken into account, ultimately reducing the risks of AI hallucinations and improving trustworthiness. Experiments on benchmark datasets demonstrate competitive performance compared to state-of-the-art techniques, with significantly lower computational complexity and enhanced accuracies and reliability for high-stakes applications. The code and datasets utilized in this study for DLaVA are accessible at: https://github.com/ahmad-shirazi/AnnotMLLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。