让图像理解和生成像人一样实时互动，自动修正矛盾。
Concurrent Image Understanding and Generation: Self-Correcting Coupled Markov Jump Processes
- 用跨模态注意力动态调整生成过程，实现图文互纠。
- 支持单步采样，图像理解与编辑性能超越现有方法。
- 适合需要实时协同生成的视觉推理任务，如解谜游戏。
人类认知不区分理解与生成。教师在黑板前讲解与作图同步进行，二者相互影响。本文将这种耦合机制引入人工智能系统。掩码扩散模型（MDMs）适合此任务，但现有采样器要么交错处理文本与图像，要么仅共享上一步历史，无法在同一步内利用对方最新决策；加之MDMs无法重掩码，导致跨模态矛盾无法被检测或修复。本文提出自校正耦合马尔可夫跳跃过程（SC-CMJP），其中一模态的转移率由另一模态的置信度加权决定，权重通过跨模态注意力计算。此外，引入重掩码跳跃机制，在发现跨模态证据冲突时立即撤回先前承诺。结合该框架，我们提出无需训练的单步采样器CO₂Jump，用于联合多模态生成。为训练与评估，我们构建并公开三个大规模联合多模态生成数据集：JEdit-1M、JMaze-200K、JNono-200K，包含分布内与分布外基准。CO₂Jump在图像理解与编辑、视觉推理（迷宫与非欧拉图求解）任务中表现最优。采样器性能随去噪步骤增加而单调提升，表明跨模态耦合效应在轨迹中持续累积。
原文摘要 · Abstract (English)
Human cognition does not separate understanding and generation. A teacher at a whiteboard speaks and draws $\textit{together}$, each modality reshapes the other. In this paper, we bring this coupled loop to artificial systems. Masked Diffusion Models (MDMs) are ideally suited to this task, yet existing samplers either decode text and image interleavedly or independently update them in parallel branches that share only previous-step history, but not the other modality's latest decisions $\textit{within}$ the same step; combined with MDMs' inability to remask, cross-modal contradictions are neither detected nor repaired. We introduce $\textbf{Self-Correcting Coupled Markov Jump Processes (SC-CMJP)}$, a framework in which one modality's transition rates are functionals of the other modality's confidence score, as weighted by cross-modal attention. Furthermore, a remasking jump retracts commitments the moment cross-modal evidence turns against them. In conjunction with SC-CMJP, we introduce $\texttt{CO}_\texttt{2}\texttt{Jump}$ (Self-$\underline{\text{CO}}$rrecting $\underline{\text{CO}}$upled $\underline{\text{Jump}}$), a novel training-free single-pass sampler for joint multimodal geneneration. For training and evaluation purposes, we have created and will release three large-scale joint multimodal generation corpora: $\text{JEdit-1M}$, $\text{JMaze-200K}$, $\text{JNono-200K}$, with matching in- and out-of-distribution benchmarks. $\texttt{CO}_\texttt{2}\texttt{Jump}$ achieves best joint performance for image understanding and editing as well as visual reasoning (maze and nonogram solving). The performance of the sampler scales monotonically with the number of denoising steps, evidence that the benefits of cross-modal coupling $\textit{compound}$ across the trajectory. Project page: https://coupled-jump.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。