用可执行代码替代语言描述,让文生图更精准可控。
CoCo: Code as CoT for Text-to-Image Preview and Rare Concept Generation
- 将推理过程转为可运行代码,明确规划图像结构
- 在三个评测集上比直接生成提升超40%以上
- 适合需要精确布局和复杂内容的文生图场景
统一多模态模型(UMMs)在文本到图像(T2I)生成方面取得显著进展,尤其通过引入思维链(CoT)推理。然而,现有基于CoT的方法主要依赖抽象自然语言规划,难以满足复杂空间布局、结构化视觉元素和密集文本内容的需求。本文提出CoCo(Code-as-CoT),一种以可执行代码驱动的推理框架,将推理过程表示为可运行代码,实现显式且可验证的中间规划。给定文本提示后,CoCo首先生成指定场景结构布局的可执行代码,随后在沙盒环境中执行以渲染确定性草图图像,再通过细粒度图像编辑进行优化,生成最终高保真结果。为支持该训练范式,我们构建了CoCo-10K数据集,包含10,000对结构化草图-最终图像对,用于训练结构化草图构建与视觉修正能力。在StructT2IBench、OneIG-Bench和LongText-Bench上的实证评估显示,CoCo分别相比直接生成提升68.83%、54.8%和41.23%,同时优于其他增强型CoT生成方法。结果表明,可执行代码是实现精确、可控、结构化文生图的有效可靠范式。
原文摘要 · Abstract (English)
Recent advancements in Unified Multimodal Models (UMMs) have significantly advanced text-to-image (T2I) generation, particularly through the integration of Chain-of-Thought (CoT) reasoning. However, existing CoT-based T2I methods largely rely on abstract natural-language planning, which lacks the precision required for complex spatial layouts, structured visual elements, and dense textual content. In this work, we propose CoCo (Code-as-CoT), a code-driven reasoning framework that represents the reasoning process as executable code, enabling explicit and verifiable intermediate planning for image generation. Given a text prompt, CoCo first generates executable code that specifies the structural layout of the scene. The code is then executed in a sandboxed environment to render a deterministic draft image. Subsequently, the model refines this draft through fine-grained image editing to produce the final high-fidelity result. To support this training paradigm, we construct CoCo-10K, a curated dataset containing structured draft-final image pairs designed to teach both structured draft construction and corrective visual refinement. Empirical evaluations on StructT2IBench, OneIG-Bench, and LongText-Bench show that CoCo achieves improvements of 68.83%, 54.8%, and 41.23%, respectively, over direct generation, while also outperforming other CoT-enhanced generation methods. These results demonstrate that executable code is an effective and reliable reasoning paradigm for precise, controllable, and structured text-to-image generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。