arXiv:2606.31732cs.CV2026-06

用符号奖励和参考引导,让模型更准地把图变代码

UniCoder: Unified Visual-to-Code Generation via Symbolic Rewards and Reference-Guided Code Optimization

论文配图:UniCoder: Unified Visual-to-Code Generation via Symbolic Rewards and Reference-Guided Code Optimization
图 1 · 摘自论文原文
  • 用小模型解析代码为颜色坐标等符号属性,实现细粒度奖励
  • 在低效生成路径中注入真实代码,避免探索卡死
  • 在多个数据集上超越开源模型,接近闭源顶尖水平

将科学图表、矢量图形和网页转化为可执行脚本的视觉到代码生成,需要像素级对齐,而现有多模态大模型仅靠监督微调难以达成。尽管强化学习理论上可行,但面临两大挑战:(1)奖励粗糙,语义指标无法惩罚细微元素偏差;(2)探索停滞,代码空间稀疏异构导致策略无法自举有效轨迹。为此,我们提出UniCoder,一个统一的强化学习框架,包含两项新机制。首先,提出符号属性对齐,通过轻量辅助大模型将生成代码解析为离散视觉属性(如十六进制颜色、坐标范围),实现密集的逐元素奖励计算。其次,设计参考引导代码优化,动态向表现差的生成组注入真实轨迹,使盲目探索转为有方向的策略改进。在ChartMimic、UniSVG、Design2Code和ScreenBench四个基准上的实验表明,我们的80亿参数模型不仅超越所有开源基线,性能还达到与专有模型相当的最先进水平,为通用视觉到代码合成树立了新范式。

原文摘要 · Abstract (English)

Visual-to-Code generation, which transforms scientific plots, vector graphics, and webpages into executable scripts, demands a level of pixel-precise alignment that standard Multimodal Large Language Models (MLLMs) fail to achieve through Supervised Fine-Tuning (SFT) alone. While Reinforcement Learning (RL) offers a theoretical pathway to bridge this gap, its application is hindered by two fundamental obstacles: (1) \textit{Reward Coarseness}, where semantic metrics like CLIP scores fail to penalize fine-grained element deviations, and (2) \textit{Exploration Stagnation}, where the sparse, heterogeneous code search space prevents the policy from bootstrapping valid trajectories. To overcome these limitations, we introduce UniCoder, a unified RL framework that integrates two novel mechanisms. First, we propose \textbf{Symbolic Attribute Alignment}, which employs a lightweight auxiliary LLM to parse generated code into discrete visual attributes (e.g., hex colors, coordinate limits), enabling dense, element-wise reward computation. Second, to escape local optima, we devise \textbf{Reference-Guided Code Optimization}, a strategy that dynamically injects ground-truth trajectories into low-performing rollout groups, transforming blind exploration into guided policy improvement. Extensive experiments on ChartMimic, UniSVG, Design2Code and ScreenBench benchmarks demonstrate that our 8B-parameter model not only surpasses all open-source baselines but also achieves state-of-the-art performance comparable to proprietary models, establishing a new paradigm for generalized visual-to-code synthesis.

视觉生成代码强化学习符号奖励代码优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。