让图像生成模型真正记住对话历史,支持回溯与跨轮引用。
Non-Markov Multi-Round Conversational Image Generation with History-Conditioned MLLMs
- 设计非马尔可夫对话数据构造方法,强制模型回溯早期视觉状态。
- 引入基于标记的缓存机制,避免多轮交互中身份漂移问题。
- 提升长程一致性,适合需要连贯性的人像定制与复杂指令场景。
对话式图像生成需在多轮交互中遵循用户指令,依赖交错的文本与图像积累的聊天历史。现有大多数多轮基准与训练方法本质上为马尔可夫型:下一输出主要依赖最新图像,导致模型忽略长程历史。本文针对更难的非马尔可夫设定,提出 (i) 非马尔可夫多轮数据构建策略,包括回滚式编辑(强制检索早期视觉状态)与基于名称的多轮个性化(将名称绑定至跨轮外观);(ii) 基于历史条件的训练与推理框架,采用标记级缓存防止多轮身份漂移;(iii) 实现高保真图像重建与可编辑个性化,包含基于重建的DiT解码器与多阶段微调课程。实验表明,显式训练非马尔可夫交互显著提升多轮一致性和指令遵循能力,同时保持单轮编辑与个性化的强性能。
原文摘要 · Abstract (English)
Conversational image generation requires a model to follow user instructions across multiple rounds of interaction, grounded in interleaved text and images that accumulate as chat history. While recent multimodal large language models (MLLMs) can generate and edit images, most existing multi-turn benchmarks and training recipes are effectively Markov: the next output depends primarily on the most recent image, enabling shortcut solutions that ignore long-range history. In this work we formalize and target the more challenging non-Markov setting, where a user may refer back to earlier states, undo changes, or reference entities introduced several rounds ago. We present (i) non-Markov multi-round data construction strategies, including rollback-style editing that forces retrieval of earlier visual states and name-based multi-round personalization that binds names to appearances across rounds; (ii) a history-conditioned training and inference framework with token-level caching to prevent multi-round identity drift; and (iii) enabling improvements for high-fidelity image reconstruction and editable personalization, including a reconstruction-based DiT detokenizer and a multi-stage fine-tuning curriculum. We demonstrate that explicitly training for non-Markov interactions yields substantial improvements in multi-round consistency and instruction compliance, while maintaining strong single-round editing and personalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。