让AI看图表更准,通过多智能体协作实现高精度视觉推理。
PixelCraft: A Multi-Agent System for High-Fidelity Visual Reasoning on Structured Images
- 用多智能体系统动态选择工具、讨论推理路径
- 在图表和几何题上显著提升准确率,超越现有模型
- 适合需要精细图像理解的科研与教育场景
结构化图像(如图表和几何图)对多模态大语言模型仍具挑战,感知误差易导致错误结论。现有基于视觉提示的方法受限于低质量图像处理和僵化的线性推理流程。本文提出PixelCraft,一种用于结构化图像的高保真图像处理与灵活视觉推理的多智能体系统。系统包含调度器、规划器、推理器、批评者及一组视觉工具智能体。为实现高保真处理,我们构建高质量语料库,并微调多模态模型为定位模型,将其像素级定位能力与传统计算机视觉算法融合于工具智能体中。在此基础上,PixelCraft通过动态三阶段工作流(工具选择、智能体讨论、自我批判)实现灵活推理。不同于以往简单拼接历史图像的线性模式,PixelCraft维护图像记忆,使规划器可自适应回溯早期视觉步骤,探索替代推理路径,并在讨论中动态调整推理轨迹。在多个具有挑战性的图表与几何基准测试中,实验表明PixelCraft显著提升先进多模态模型的视觉推理性能,树立了结构化图像推理的新标准。代码将开源。
原文摘要 · Abstract (English)
Structured images (e.g., charts and geometric diagrams) remain challenging for multimodal large language models (MLLMs), as perceptual slips can cascade into erroneous conclusions. Intermediate visual cues can steer reasoning; however, existing cue-based methods are constrained with low-fidelity image processing and linear, rigid reasoning patterns, limiting their effectiveness on complex structured-image tasks. In this paper, we propose PixelCraft, a novel multi-agent system for high-fidelity image processing and flexible visual reasoning on structured images. The system comprises a dispatcher, a planner, a reasoner, critics, and a set of visual tool agents. To achieve high-fidelity processing, we construct a high-quality corpus and fine-tune an MLLM into a grounding model, whose pixel-level localizations are integrated with traditional computer vision (CV) algorithms in tool agents. Building on this foundation, PixelCraft facilitates flexible visual reasoning through a dynamic three-stage workflow of tool selection, agent discussion, and self-criticism. Moreover, unlike prior linear reasoning patterns that simply append historical images, PixelCraft maintains an image memory to allow the planner to adaptively revisit earlier visual steps, explore alternative reasoning branches, and dynamically adjust the reasoning trajectory during discussion. Extensive experiments on challenging chart and geometry benchmarks demonstrate that PixelCraft significantly improves visual reasoning performance for advanced MLLMs, setting a new standard for structured image reasoning. Our code will be available at https://github.com/microsoft/PixelCraft.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。