用视觉语言模型解析建筑规范中的复杂表格，提升法规查询准确率。
Table Comprehension in Building Codes using Vision Language Models and Domain-Specific Fine-Tuning
- 直接输入图片给视觉模型，比转成LaTeX再处理更有效。
- 对模型进行领域微调后，最高准确率提升超100%。
- 适合需要精准理解建筑规范的工程与合规人员。
建筑规范包含确保安全、合规和决策的重要信息。自动化问答系统可快速准确获取具体条款，提升效率并减少错误。检索增强生成（RAG）系统在该任务中至关重要，结合了信息检索的精度与生成模型的能力。然而，表格数据因布局复杂、合并单元格、多行表头及嵌入语义关系，难以通过传统自然语言处理和视觉语言模型（VLMs）捕捉。本文对比两种从建筑规范表格中提取信息的方法：一是直接将页面图像输入VLMs进行问答；二是将含表格的图像转换为LaTeX代码后输入。实验表明，直接输入方法整体准确率更高。为进一步提升性能，使用领域专用表格数据对各VLM进行低秩适应（LoRA）微调，微调后模型表现显著提升，Qwen2.5-VL-3B-Instruct相对准确率增长超过100%。结果表明，参数高效微调可有效适配强大VLMs，以理解建筑规范等专业领域的复杂结构化数据。
原文摘要 · Abstract (English)
Building codes contain critical information for ensuring safety, regulatory compliance, and informed decision-making in construction and engineering. Automated question answering systems over such codes enable quick and accurate access to specific regulatory clauses, improving efficiency and reducing errors. Retrieval-Augmented Generation (RAG) systems are essential for this task as they combine the precision of information retrieval with the generative capabilities of language models. However, tabular data are challenging to extract as they often involve complex layouts, merged cells, multi-row headers, and embedded semantic relationships that are not easily captured by traditional natural language processing techniques and Vision Language Models (VLMs). This paper explores and compares two methods for extracting information from tabular data in building codes using several pre-trained VLMs. First, a direct input method is used, where the image of the page is input directly into the VLMs, which are then tasked with answering questions based on the image. Second, an indirect input method is introduced, which involves converting an image of a page containing tables into the LaTeX code and then answering inquires based on the LaTeX-based input. The experiments find that the direct input method generally resulted in higher accuracy than the indirect input method. To further improve the performance, we fine-tuned each VLM using Low Rank Adaptation (LoRA) on a domain-specific tabular dataset. The fine-tuned models exhibited substantial improvements, with Qwen2.5-VL-3B-Instruct achieving relative accuracy gains exceeding 100%. Our results highlight the potential of parameter-efficient fine-tuning methods to adapt powerful VLMs for understanding complex structured data in specialized fields, such as building code interpretation and regulatory compliance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。