用大模型生成可精确渲染的矢量图,提升图文对齐能力。
Symbolic Graphics Programming with Large Language Models

- 通过可验证奖励机制,让模型生成格式正确且视觉一致的SVG代码。
- 在多个评估维度上,改进后模型性能接近顶级系统。
- 适合研究跨模态理解、生成式设计与可解释性的人参考。
大型语言模型在程序合成方面表现优异,但在生成可精确渲染的符号图形程序(SGP)以输出视觉内容方面仍待探索。本文聚焦于从自然语言描述生成符号图形程序的任务,特别关注可缩放矢量图形(SVG)。我们构建了SGP-GenBench基准,涵盖物体保真度、场景保真度以及组合性(属性绑定、空间关系、数理能力)等维度。实验发现,前沿闭源模型显著优于开源模型,且性能与通用编程能力高度相关。为此,我们提出一种基于可验证奖励的强化学习方法:利用格式有效性门控确保输出可渲染的SVG,通过强视觉编码器(如SigLIP和DINO)实现文本与图像间的跨模态对齐。该方法应用于Qwen-2.5-7B模型,显著提升生成质量与语义一致性,达到与领先系统相当水平。进一步分析显示,强化学习促使模型更精细地分解对象为可控基本单元,并增强上下文细节以提升场景连贯性。结果表明,符号图形编程为理解跨模态对齐提供了精确且可解释的视角。
原文摘要 · Abstract (English)
Large language models (LLMs) excel at program synthesis, yet their ability to produce symbolic graphics programs (SGPs) that render into precise visual content remains underexplored. We study symbolic graphics programming, where the goal is to generate an SGP from a natural-language description. This task also serves as a lens into how LLMs understand the visual world by prompting them to generate images rendered from SGPs. Among various SGPs, our paper sticks to scalable vector graphics (SVGs). We begin by examining the extent to which LLMs can generate SGPs. To this end, we introduce SGP-GenBench, a comprehensive benchmark covering object fidelity, scene fidelity, and compositionality (attribute binding, spatial relations, numeracy). On SGP-GenBench, we discover that frontier proprietary models substantially outperform open-source models, and performance correlates well with general coding capabilities. Motivated by this gap, we aim to improve LLMs' ability to generate SGPs. We propose a reinforcement learning (RL) with verifiable rewards approach, where a format-validity gate ensures renderable SVG, and a cross-modal reward aligns text and the rendered image via strong vision encoders (e.g., SigLIP for text-image and DINO for image-image). Applied to Qwen-2.5-7B, our method substantially improves SVG generation quality and semantics, achieving performance on par with frontier systems. We further analyze training dynamics, showing that RL induces (i) finer decomposition of objects into controllable primitives and (ii) contextual details that improve scene coherence. Our results demonstrate that symbolic graphics programming offers a precise and interpretable lens on cross-modal grounding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。