让AI画画时实时看效果,提升矢量图生成质量
Render-in-the-Loop: Vector Graphics Generation via Visual Self-Feedback
- 引入视觉反馈环,每步生成后渲染画面供模型观察
- 在MMSVGBench上优于主流开源模型,支持文本与图像转矢量图
- 适合需要精准图形生成的AI设计、动画场景
多模态大语言模型在直接代码合成方面展现出生成可缩放矢量图形(SVG)的潜力。然而,现有方法普遍采用开环‘盲画’模式,即模型生成符号代码序列但不感知中间视觉结果,严重低估了模型视觉编码器中蕴含的视觉先验,将SVG生成视为孤立的文本序列建模任务,而非整合的视觉空间任务。因此,模型难以推理部分画布状态和隐含遮挡关系,这些信息在视觉上明显但在文本中模糊。为此,我们提出Render-in-the-Loop,一种将SVG合成重构为分步、视觉上下文感知的过程的新范式。通过将中间代码状态渲染为累积画布,模型在每一步都能显式观察演进中的视觉上下文,并利用实时反馈引导后续生成。然而,我们发现将此视觉循环直接应用于现成模型效果不佳,因其无法有效利用增量式视觉-代码映射。为此,我们首先采用细粒度路径分解构建密集的多步视觉轨迹,再引入视觉自反馈(VSF)训练策略,使下一步原始图形生成基于中间视觉状态。此外,提出渲染-验证(RaV)推理机制,有效过滤退化和冗余图形。我们在多模态基础模型上实现该框架,在标准MMSVGBench上超越强基线,凸显其在文本到SVG和图像到SVG任务中的卓越数据效率与泛化能力。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have shown promising capabilities in generating Scalable Vector Graphics (SVG) via direct code synthesis. However, existing paradigms typically adopt an open-loop "blind drawing" approach, where models generate symbolic code sequences without perceiving intermediate visual outcomes. This methodology severely underutilizes the powerful visual priors embedded in MLLMs vision encoders, treating SVG generation as a disjointed textual sequence modeling task rather than an integrated visuo-spatial one. Consequently, models struggle to reason about partial canvas states and implicit occlusion relationships, which are visually explicit but textually ambiguous. To bridge this gap, we propose Render-in-the-Loop, a novel generation paradigm that reformulates SVG synthesis as a step-wise, visual-context-aware process. By rendering intermediate code states into a cumulative canvas, the model explicitly observes the evolving visual context at each step, leveraging on-the-fly feedback to guide subsequent generation. However, we demonstrate that applying this visual loop naively to off-the-shelf models is suboptimal due to their inability to leverage incremental visual-code mappings. To address this, we first utilize fine-grained path decomposition to construct dense multi-step visual trajectories, and then introduce a Visual Self-Feedback (VSF) training strategy to condition the next primitive generation on intermediate visual states. Furthermore, a Render-and-Verify (RaV) inference mechanism is proposed to effectively filter degenerate and redundant primitives. Our framework, instantiated on a multimodal foundation model, outperforms strong open-weight baselines on the standard MMSVGBench. This result highlights the remarkable data efficiency and generalization capability of our Render-in-the-Loop paradigm for both Text-to-SVG and Image-to-SVG tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。