让大模型学会理解与生成复杂矢量图,解决语义错乱和图形遮挡问题。
Empowering LLMs to Understand and Generate Complex Vector Graphics
- 用可学习语义标记精准编码矢量图元素及其属性
- 构建58万条高质量图文指令数据,支持模型训练
- 融合几何、外观与语言信息,实现结构化图形生成
大语言模型在自然语言处理上取得突破,但在可缩放矢量图形(SVG)生成领域仍显不足。尽管模型在训练中学习了网页中的部分SVG知识,但其内部的语义模糊与分词表示常导致矢量图生成幻觉,且缺乏对路径渲染顺序的理解,易造成图形重叠。本文提出LLM4SVG,通过可学习语义标记实现对SVG组件的深度理解,精确编码元素及其属性,生成语义一致的输出。我们构建了一个结构化指令数据集,支持理解与生成两大任务。方法采用模块化架构,整合语义标签、矢量指令编码器、微调指令与强大语言模型,紧密融合几何、外观与语言信息。为缓解SVG-文本指令数据稀缺问题,我们开发自动化数据生成流程,构建了包含58万条精心设计指令的SVGX-SFT数据集,支持监督微调策略,推动大模型在矢量图生成中的应用。
原文摘要 · Abstract (English)
The unprecedented advancements in Large Language Models (LLMs) have profoundly impacted natural language processing but have yet to fully embrace the realm of scalable vector graphics (SVG) generation. While LLMs encode partial knowledge of SVG data from web pages during training, recent findings suggest that semantically ambiguous and tokenized representations within LLMs may result in hallucinations in vector primitive predictions. Additionally, LLM training typically lacks modeling and understanding of the rendering sequence of vector paths, which can lead to occlusion between output vector primitives. In this paper, we present LLM4SVG, an initial yet substantial step toward bridging this gap by enabling LLMs to better understand and generate vector graphics. LLM4SVG facilitates a deeper understanding of SVG components through learnable semantic tokens, which precisely encode these tokens and their corresponding properties to generate semantically aligned SVG outputs. Using a series of learnable semantic tokens, a structured dataset for instruction following is developed to support comprehension and generation across two primary tasks. Our method introduces a modular architecture to existing large language models, integrating semantic tags, vector instruction encoders, fine-tuned commands, and powerful LLMs to tightly combine geometric, appearance, and language information. To overcome the scarcity of SVG-text instruction data, we developed an automated data generation pipeline that collected our SVGX-SFT Dataset, consisting of high-quality human-designed SVGs and 580k SVG instruction following data specifically crafted for LLM training, which facilitated the adoption of the supervised fine-tuning strategy popular in LLM development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。