arXiv:2604.09552cs.IRcs.AI2026-04被引 2

用视觉语言检索提升工程文档问答准确率,无需完整文档入库

MCERF: Advancing Multimodal LLM Evaluation of Engineering Documentation with Enhanced Retrieval

论文配图:MCERF: Advancing Multimodal LLM Evaluation of Engineering Documentation with Enhanced Retrieval
图 1 · 摘自论文原文
  • 融合图文检索与模块化推理,动态分配问题到最优处理路径
  • 在DesignQA上相比基线提升41.1%准确率,复杂多模态问题表现显著
  • 适合需要高效理解图纸、表格和文本的工程智能系统开发者

工程规则手册和技术标准包含密集文本、表格和图表等多模态信息,传统检索增强生成(RAG)系统难以有效处理。本文基于DesignQA框架,提出多模态ColPali增强检索与推理框架(MCERF),结合多模态检索器与大模型推理能力,实现对工程文档的精准高效问答。该系统采用ColPali模型同时检索文本与视觉信息,支持四种策略:(i) 混合查找模式用于显式规则匹配,(ii) 视觉转文本融合用于图表示引导查询,(iii) 高级推理大模型处理复杂多模态问题,(iv) 自洽性决策机制稳定输出。模块化设计可适配不同模型架构。此外,对比单路径路由与多智能体路由两种动态分配策略,实验在DesignQA基准上显示,该系统相较最佳基线RAG结果平均准确率提升41.1%,显著改善了无需全文摄入的多模态与推理密集型任务表现,验证了视觉语言检索、模块化推理与自适应路由在工程场景中实现可扩展文档理解的可行性。

原文摘要 · Abstract (English)

Engineering rulebooks and technical standards contain multimodal information like dense text, tables, and illustrations that are challenging for retrieval augmented generation (RAG) systems. Building upon the DesignQA framework [1], which relied on full-text ingestion and text-based retrieval, this work establishes a Multimodal ColPali Enhanced Retrieval and Reasoning Framework (MCERF), a system that couples a multimodal retriever with large language model reasoning for accurate and efficient question answering from engineering documents. The system employs the ColPali, which retrieves both textual and visual information, and multiple retrieval and reasoning strategies: (i) Hybrid Lookup mode for explicit rule mentions, (ii) Vision to Text fusion for figure and table guided queries, (iii) High Reasoning LLM mode for complex multi modal questions, and (iv) SelfConsistency decision to stabilize responses. The modular framework design provides a reusable template for future multimodal systems regardless of underlying model architecture. Furthermore, this work establishes and compares two routing approaches: a single case routing approach and a multi-agent system, both of which dynamically allocate queries to optimal pipelines. Evaluation on the DesignQA benchmark illustrates that this system improves average accuracy across all tasks with a relative gain of +41.1% from baseline RAG best results, which is a significant improvement in multimodal and reasoning-intensive tasks without complete rulebook ingestion. This shows how vision language retrieval, modular reasoning, and adaptive routing enable scalable document comprehension in engineering use cases.

多模态工程文档检索增强视觉语言

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。