首个统一图文生成与理解的矢量图数据集,助力大模型精准处理矢量图形。
UniSVG: A Unified Dataset for Vector Graphic Understanding and Generation with Multimodal Large Language Models
- 构建52.5万条矢量图数据,支持文本与图像条件下的统一生成与理解。
- 在多项任务上超越闭源模型GPT-4V,显著提升开源大模型性能。
- 适合研究多模态生成、矢量图形理解及可扩展设计工具的开发者。
与位图不同,可缩放矢量图形(SVG)在缩放时保持高质量,常用于计算机视觉与艺术设计中的代码表示。随着人工智能系统日益普及,让AI理解并生成SVG变得愈发迫切。然而,由于SVG代码由浮点参数控制的曲线和线条构成,其理解与生成需极高精度;同时,生成需满足文本提示或视觉参考等多样化条件,对多模态处理能力要求高。近年来,多模态大语言模型(MLLMs)展现出处理多模态输入与生成复杂控制参数的能力,具备在统一模型中解决SVG U&G任务的潜力。为释放MLLM在该领域的潜力,我们提出面向矢量图的统一数据集UniSVG,包含52.5万条数据,专为MLLM训练与评估设计。据我们所知,这是首个面向统一的文本/图像驱动矢量图生成与多维度理解(颜色、类别、用途等)的综合性数据集。实验证明,基于该数据集训练的开源MLLM在多项任务上超越闭源模型GPT-4V。相关数据集、基准测试、权重、代码与实验细节已公开于https://ryanlijinke.github.io/。
原文摘要 · Abstract (English)
Unlike bitmap images, scalable vector graphics (SVG) maintain quality when scaled, frequently employed in computer vision and artistic design in the representation of SVG code. In this era of proliferating AI-powered systems, enabling AI to understand and generate SVG has become increasingly urgent. However, AI-driven SVG understanding and generation (U&G) remain significant challenges. SVG code, equivalent to a set of curves and lines controlled by floating-point parameters, demands high precision in SVG U&G. Besides, SVG generation operates under diverse conditional constraints, including textual prompts and visual references, which requires powerful multi-modal processing for condition-to-SVG transformation. Recently, the rapid growth of Multi-modal Large Language Models (MLLMs) have demonstrated capabilities to process multi-modal inputs and generate complex vector controlling parameters, suggesting the potential to address SVG U&G tasks within a unified model. To unlock MLLM's capabilities in the SVG area, we propose an SVG-centric dataset called UniSVG, comprising 525k data items, tailored for MLLM training and evaluation. To our best knowledge, it is the first comprehensive dataset designed for unified SVG generation (from textual prompts and images) and SVG understanding (color, category, usage, etc.). As expected, learning on the proposed dataset boosts open-source MLLMs' performance on various SVG U&G tasks, surpassing SOTA close-source MLLMs like GPT-4V. We release dataset, benchmark, weights, codes and experiment details on https://ryanlijinke.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。