arXiv:2511.02778cs.CVcs.CL2025-11被引 12

用SVG代码重构视觉理解,让模型像人一样画图思考。

VCode: a Multimodal Coding Benchmark with SVG as Symbolic Visual Representation

  • 以SVG为符号化视觉表示,将图像转为可执行代码
  • 多模态任务中顶尖模型生成的SVG仍不准确,存在显著差距
  • 提出可迭代修正与视觉工具增强的智能体框架,性能提升12.3点

代码已成为智能体时代精准且可执行的推理与行动媒介。然而当前进展主要聚焦于语言主导的任务(如程序合成与调试),视觉主导的编码仍被忽视。受人类通过草图推理的启发,我们主张使用SVG代码作为紧凑、可解释且可执行的视觉表示形式。本文提出VCode基准,将多模态理解重新定义为代码生成任务:给定一张图像,模型需输出保留符号语义的SVG代码,以支持下游推理。VCode涵盖三大领域:通用常识(MM-Vet)、专业学科(MMMU)和视觉感知(CV-Bench)。为评估符号保真度,提出CodeVQA新评测协议,由策略模型在渲染后的SVG上回答问题,正确答案表明符号信息完整保留。实证发现,前沿视觉语言模型在生成忠实SVG方面表现不佳,揭示语言与视觉编码能力间的持续差距。为此,我们提出VCoder智能体框架,在两个维度增强模型:(i) 思考与修订,通过迭代分析差异并优化代码;(ii) 使用视觉工具,利用检测器与解析器提供超出模型自身能力的对象、形状、文本等结构化提示。在多个基准测试中,具备强推理能力的前沿模型整体表现良好,但在专业知识与三维推理方面仍有局限。VCoder相较表现最佳的Claude-4-Opus实现12.3分的整体提升。人工实验显示,人类与模型在渲染后SVG上的表现均下降,但一致性表明符号化视觉表示具有潜力。基准与代码已开源:https://github.com/CSU-JPG/VCode。

原文摘要 · Abstract (English)

Code has emerged as a precise and executable medium for reasoning and action in the agent era. Yet, progress has largely focused on language-centric tasks such as program synthesis and debugging, leaving visual-centric coding underexplored. Inspired by how humans reason over sketches, we advocate SVG code as a compact, interpretable, and executable visual representation. We introduce VCode, a benchmark that reframes multimodal understanding as code generation: given an image, a model must produce SVG that preserves symbolic meaning for downstream reasoning. VCode covers three domains - general commonsense (MM-Vet), professional disciplines (MMMU), and visual-centric perception (CV-Bench). To assess symbolic fidelity, we propose CodeVQA, a novel evaluation protocol in which a policy model answers questions over rendered SVGs; correct answers indicate faithful symbolic preservation. Empirically, frontier VLMs struggle to generate faithful SVGs, revealing a persistent gap between language-centric and visual-centric coding. To close this gap, we introduce VCoder, an agentic framework that augments VLMs along two axes: (i) Thinking with Revision, which iteratively analyzes discrepancies and refines SVG code; and (ii) Acting with Visual Tools, where detectors and parsers supply structured cues such as objects, shapes, and text beyond the model's intrinsic capacity. Across benchmarks, frontier VLMs with strong reasoning capabilities score well overall yet remain limited in professional knowledge and 3D reasoning. VCoder delivers a 12.3-point overall gain over the top-performing Claude-4-Opus. Human studies show that both humans and VLMs perform worse on rendered SVGs, their consistency reveals the promise of symbolic visual representation. The benchmark and code are available at https://github.com/CSU-JPG/VCode.

多模态编码SVG生成智能体框架符号表示

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。