arXiv:2412.16420cs.CVcs.CL2024-12被引 15

将流程图转为文本再推理,提升可控性与可解释性

Beyond End-to-End VLMs: Leveraging Intermediate Text Representations for Superior Flowchart Understanding

  • 先用视觉转文本生成流程图的文本表示,再用大模型问答
  • 在两个基准上达到最优性能,错误定位更清晰
  • 适合需要定制化和透明决策过程的研究者

流程图通常以图像形式呈现,促使研究者采用端到端视觉语言模型(VLM)进行理解。然而存在两大挑战:(i) 控制力弱——用户只能修改输入图像,而大多数研究者无法调整VLM训练;(ii) 可解释性差——难以判断错误源于视觉编码还是推理阶段。本文提出TextFlow,分两阶段解决:(i) 视觉文本化器——从流程图图像生成文本表示;(ii) 文本推理器——基于文本表示完成问答任务。TextFlow具备三大优势:(i) 用户可选择文本格式(如Graphviz、Mermaid、PlantUML),或转化为可执行图对象调用工具,增强性能与可控性;(ii) 通过分离视觉与文本处理模块,提升错误归因能力;(iii) 支持模块化设计,当VLM表现不佳时,可在推理阶段替换为更强的LLM。在FlowVQA和FlowLearn两个基准上的实验表明,TextFlow达到当前最佳性能且具有鲁棒性。所有代码已公开。

原文摘要 · Abstract (English)

Flowcharts are typically presented as images, driving the trend of using vision-language models (VLMs) for end-to-end flowchart understanding. However, two key challenges arise: (i) Limited controllability--users have minimal influence over the downstream task, as they can only modify input images, while the training of VLMs is often out of reach for most researchers. (ii) Lack of explainability--it is difficult to trace VLM errors to specific causes, such as failures in visual encoding or reasoning. We propose TextFlow, addressing aforementioned issues with two stages: (i) Vision Textualizer--which generates textual representations from flowchart images; and (ii) Textual Reasoner--which performs question-answering based on the text representations. TextFlow offers three key advantages: (i) users can select the type of text representations (e.g., Graphviz, Mermaid, PlantUML), or further convert them into executable graph object to call tools, enhancing performance and controllability; (ii) it improves explainability by helping to attribute errors more clearly to visual or textual processing components; and (iii) it promotes the modularization of the solution, such as allowing advanced LLMs to be used in the Reasoner stage when VLMs underperform in end-to-end fashion. Experiments on the FlowVQA and FlowLearn benchmarks demonstrate TextFlow's state-of-the-art performance as well as its robustness. All code is publicly available.

流程图理解视觉语言模型可解释性文本生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。