用SVG+多智能体提升大模型看图能力，让图表理解更准。
Socratic Chart: Cooperating Multiple Agents for Robust SVG Chart Understanding
- 将图表转为可解析的SVG格式，结合多智能体提取视觉特征。
- 在无标签、有扰动场景下，性能仍优于GPT-4o等模型30%。
- 适合需要高精度图表分析的研究与应用，如金融、科研自动化。
多模态大语言模型虽表现多样，但在图表推理任务中缺乏真正的视觉理解。现有基准如ChartQA显示模型依赖文本捷径和概率匹配，而非真实视觉推理。为此，我们在ChartQA数据集上移除文本标签并引入图表扰动，构建更具挑战性的测试场景。在此条件下，GPT-4o与Gemini-2.0 Pro性能下降高达30%，凸显其局限性。我们提出Socratic Chart框架，将图表图像转换为可解析的可缩放矢量图形（SVG）表示，使大模型能融合文本与视觉信息以增强理解。该框架采用多智能体流水线：专用生成器提取柱状图高度、折线坐标等原始属性，批判者验证结果，确保符号化表示的高保真度。实验表明，该方法在准确捕捉图表原始属性和提升推理表现方面超越当前最优模型，为推进多模态大模型的视觉理解提供了可靠路径。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have shown remarkable versatility but face challenges in demonstrating true visual understanding, particularly in chart reasoning tasks. Existing benchmarks like ChartQA reveal significant reliance on text-based shortcuts and probabilistic pattern-matching rather than genuine visual reasoning. To rigorously evaluate visual reasoning, we introduce a more challenging test scenario by removing textual labels and introducing chart perturbations in the ChartQA dataset. Under these conditions, models like GPT-4o and Gemini-2.0 Pro experience up to a 30% performance drop, underscoring their limitations. To address these challenges, we propose Socratic Chart, a new framework that transforms chart images into Scalable Vector Graphics (SVG) representations, enabling MLLMs to integrate textual and visual modalities for enhanced chart understanding. Socratic Chart employs a multi-agent pipeline with specialized agent-generators to extract primitive chart attributes (e.g., bar heights, line coordinates) and an agent-critic to validate results, ensuring high-fidelity symbolic representations. Our framework surpasses state-of-the-art models in accurately capturing chart primitives and improving reasoning performance, establishing a robust pathway for advancing MLLM visual understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。