让文字推理贯穿图像生成全过程，提升输出语义质量。
Thinking-while-Generating: Interleaving Textual Reasoning throughout Visual Generation
- 边生成图像边进行文本推理，动态调整前后内容。
- 在自建数据集上实现更连贯、语义丰富的图像生成。
- 适合研究多模态生成与推理融合的学者参考。
视觉生成近期逐步引入推理能力，但现有方法仅在生成前（预规划）或生成后（后修正）进行文本推理，缺乏生成过程中的实时跨模态互动。本文提出首个交织式框架Thinking-while-Generating（TwiG），首次实现生成过程中持续演化的文本推理。随着视觉内容逐步生成，文本推理同步引导后续局部区域并反思已生成部分，形成动态交互，显著提升输出的上下文感知性与语义丰富度。我们探索三种策略：零样本提示、基于自建TwiG-50K数据集的监督微调（SFT），以及定制的TwiG-GRPO强化学习方法，揭示了交织推理的动态特性。代码将公开于：https://github.com/ZiyuGuo99/Thinking-while-Generating。
原文摘要 · Abstract (English)
Recent advances in visual generation have increasingly explored the integration of reasoning capabilities. They incorporate textual reasoning, i.e., think, either before (as pre-planning) or after (as post-refinement) the generation process, yet they lack on-the-fly multimodal interaction during the generation itself. In this preliminary study, we introduce Thinking-while-Generating (TwiG), the first interleaved framework that enables co-evolving textual reasoning throughout the visual generation process. As visual content is progressively generating, textual reasoning is interleaved to both guide upcoming local regions and reflect on previously synthesized ones. This dynamic interplay produces more context-aware and semantically rich visual outputs. To unveil the potential of this framework, we investigate three candidate strategies, zero-shot prompting, supervised fine-tuning (SFT) on our curated TwiG-50K dataset, and reinforcement learning (RL) via a customized TwiG-GRPO strategy, each offering unique insights into the dynamics of interleaved reasoning. We hope this work inspires further research into interleaving textual reasoning for enhanced visual generation. Code will be released at: https://github.com/ZiyuGuo99/Thinking-while-Generating.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。