构建首个无参考代码的多领域图像转代码评测基准，解决现有方法泛化差、评估不准确问题。
Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generation
- 设计无参考代码的多领域评测框架，避免依赖配对代码或通用评分标准。
- 覆盖15个数据集共2169个样本，涵盖图表、3D场景等多类视觉结构。
- 发现主流模型在空间和化学图等复杂场景表现弱，且可借筛选输出提升性能。
图像转代码任务检验视觉语言模型能否恢复图像结构并生成可执行代码。现有基准或局限于窄领域、依赖成对参考代码，或使用通用评分标准，难以捕捉领域特异性重构错误。本文提出Vision2Code，一个无参考代码的多领域图像转代码评测基准与评估框架。该基准包含来自15个源数据集的2,169个测试样例，涵盖图表、几何、图谱、科学图像、文档及3D空间场景。模型生成可执行程序后，通过视觉语言模型评分器结合数据集特定评分规则和确定性护栏进行渲染比对，以区分代码执行失败与重构质量。人类验证表明，该评估协议比通用视觉评分或嵌入相似性基线更贴近人工判断。在九个开源与专有模型中，我们发现图像转代码性能具有明显领域依赖性：领先模型在规则图表和图类视觉上表现良好，但在空间场景、化学结构、文档及电路图等类型上仍较弱。最后，我们证明经评估器过滤的模型输出可作为训练数据提升能力，例如Qwen3.5-9B在无配对源程序情况下，基准得分从1.60提升至1.86。Vision2Code为测量、诊断和改进图像转代码生成提供了可复现的测试平台。代码与数据公开于https://image2code.github.io/vision2code/。
原文摘要 · Abstract (English)
Image-to-code generation tests whether a vision-language model (VLM) can recover the structure of an image enough to express it as executable code. Existing benchmarks either focus on narrow visual domains, depend on paired executable reference code, or rely on generic rubrics that miss domain-specific reconstruction errors. We introduce Vision2Code, a reference-code-free benchmark and evaluation framework for multi-domain image-to-code generation. Vision2Code contains 2,169 test examples from 15 source datasets that span charts and plots, geometry, graphs, scientific imagery, documents, and 3D spatial scenes. Models generate executable programs, which we render and score against the source image using a VLM rater with dataset-specific rubrics and deterministic guardrails for severe semantic failures. We report render-success diagnostics that separate code execution failures from reconstruction quality. Human validation shows that this evaluation protocol aligns better with human judgments than either a generic visual rubric or embedding-similarity baselines. Across nine open-weight and proprietary models, we find that image-to-code performance is domain-dependent: leading models perform well on regular chart- and graph-like visuals but remain weak on spatial scenes, chemistry, documents, and circuit-style diagrams. Finally, we show that evaluator-filtered model outputs can serve as training data to improve image-to-code capability, with Qwen3.5-9B improving from 1.60 to 1.86 on the benchmark without paired source programs. Vision2Code provides a reproducible testbed for measuring, diagnosing, and improving image-to-code generation. Our code and data are publicly available at https://image2code.github.io/vision2code/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。