让大模型通过视觉反馈自我修正,生成更精准的SVG图像。
RefineSVG: Visual Feedback-Driven Reinforcement Learning for Image-to-SVG Generation

- 引入渲染-观察-修正闭环,用视觉差异图指导模型修正
- 在复杂图像上实现更高保真度与结构准确性,代码效率提升52%以上
- 适合需要高精度矢量图生成的研究者与设计师
我们提出RefineSVG,一种单步闭环视觉反馈框架,使多模态大语言模型通过自我修正实现高保真图像到SVG的生成。现有基于MLLM的方法依赖单次开环推理,模型仅接收一次视觉输入,需一次性生成数千个SVG代码标记而无中间验证,导致几何漂移、错误累积和视觉幻觉。RefineSVG通过在初始生成后调用外部渲染引擎,将生成结果与目标图像对比,生成多维视觉残差图(Diff-Map),作为类似ReAct的纠错信号反馈给模型,驱动针对性修正。为支持该交互流程,我们进一步设计面向SVG的语义词汇表,压缩令牌序列超过52%。通过涵盖监督微调、拒绝采样冷启动数据构建及端到端智能体强化学习的渐进式训练流程,使模型适配闭环视觉修正。大量实验表明,RefineSVG在重建保真度、结构准确性和代码效率方面均持续优于现有基线。代码已开源:https://github.com/liuxiaobo66/RefineSVG。
原文摘要 · Abstract (English)
We propose RefineSVG, a single-step closed-loop visual feedback framework that enables multimodal large language models (MLLMs) to perform high-fidelity image-to-SVG generation through self-correction. Existing MLLM-based approaches rely on single-pass open-loop inference, where the model receives visual input only once and must generate thousands of SVG code tokens without intermediate verification. This paradigm inevitably leads to geometric drift, error accumulation, and visual hallucination on complex images. RefineSVG overcomes this limitation by invoking an external rendering engine after an initial SVG generation pass to compare the rendered output against the target image. The comparison yields a multi-dimensional visual residual map (Diff-Map) that is fed back to the model as a ReAct-style correction signal, driving a targeted correction step. To support this render-observe-correct interaction, we further introduce an SVG-oriented semantic vocabulary that compresses token sequences by over 52%. A progressive training pipeline spanning supervised fine-tuning, rejection-sampling cold-start data construction, and end-to-end agentic reinforcement learning aligns the model with closed-loop visual correction. Extensive experiments show that RefineSVG consistently outperforms existing baselines in reconstruction fidelity, structural accuracy, and code efficiency.Code is available at https://github.com/liuxiaobo66/RefineSVG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。