统一多模态代码生成,用视觉强化学习提升代码与图像对齐精度
VinciCoder: Unifying Multimodal Code Generation via Coarse-to-fine Visual Reinforcement Learning
- 构建130万条直接生成数据+30万张图优化任务,支持代码自修正
- 提出粗到细视觉强化学习,以图像相似性为奖励,不依赖语法匹配
- 在多个基准上超越现有模型,适合需要图文代码生成的开发者
尽管现有专用多模态代码生成模型在图表转代码等任务上表现优异,但其单一任务训练限制了泛化能力,难以推动视觉代码智能(VI)的发展。本文提出统一大规模多模态代码生成框架VinciCoder。首先构建包含130万条直接生成样本和30万项基于视觉的代码优化任务的SFT语料库,使模型具备直接修正代码以匹配输入图像的能力。随后提出粗到细视觉强化学习(ViRL),通过多尺度图像块间的视觉相似性度量,提供与实现无关的奖励机制,克服文本指标在处理语义相同但语法不同的代码时的脆弱性。大量实验表明,VinciCoder在多种基准上表现卓越,消融实验验证了ViRL策略的有效性。数据、代码与模型已公开于https://github.com/DocTron-hub/VinciCoder。
原文摘要 · Abstract (English)
While recent specialized multimodal code generation models excel in tasks like chart-to-code generation, their reliance on single-task training limits generalization and hinders the development of \textbf{VI}sio\textbf{N} \textbf{C}ode \textbf{I}ntelligence. In this work, we introduce \textbf{VinciCoder}, a unified framework designed for generalized multimodal code generation. We first curate a large-scale SFT corpus comprising 1.3M direct generation pairs and 300k visual-based refinement tasks. This composition fosters self-refinement capabilities, enabling the model to directly rectify code to align with input images. Subsequently, we propose coarse-to-fine Visual Reinforcement Learning (ViRL) to overcome the brittleness of textual metrics in handling semantically equivalent but syntactically diverse code. By quantifying visual similarity across multi-scale patches, ViRL provides an implementation-agnostic reward mechanism that ensures high-fidelity alignment between rendered outputs and input visuals. Extensive experimental results across diverse benchmarks demonstrate that VinciCoder achieves superior performance, while comprehensive ablation studies validate the effectiveness of our proposed ViRL strategy. The data, code and model are available at https://github.com/DocTron-hub/VinciCoder.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。