让视觉模型自动切换策略,精准执行复杂指令
Breaking Dual Bottlenecks: Evolving Unified Multimodal Models into Self-Adaptive Interleaved Visual Reasoners

- 构建分层推理管道,自动选择生成、反思或分步规划
- 在5万+样本数据集上训练,复杂指令生成精度显著提升
- 适合需要高精度图像生成的科研与工业应用
统一多模态模型虽整合了理解与生成能力,但仍存在‘理解-生成鸿沟’:能捕捉用户意图却难以实现像素级精确控制。该鸿沟导致两类瓶颈:注意力纠缠瓶颈(复杂提示下盲选策略)与视觉精修瓶颈(无结构反馈无法高效纠错)。本文提出新框架,使统一模型能根据指令复杂度与自身能力自主切换生成策略。构建分层数据管道,支持三种自适应模式:简单指令直接生成、质量优化时自我反思、复杂场景则多步规划分解。基于此,构建超5万样本高质量数据集,并采用两阶段训练(SFT+RL),设计逐步推理奖励与组内复杂度惩罚机制,确保逻辑一致性并避免冗余计算。大量实验表明,该方法在anything-to-image任务中超越现有基线,在简单至复杂指令下均实现更优生成保真度。
原文摘要 · Abstract (English)
Recent unified models integrate multimodal understanding and generation within a single framework. However, an "understanding-generation gap" persists, where models can capture user intent but often fail to translate this semantic knowledge into precise pixel-level manipulation. This gap results in two bottlenecks in anything-to-image task (X2I): the attention entanglement bottleneck, where blind planning struggles with complex prompts, and the visual refinement bottleneck, where unstructured feedback fails to correct imperfections efficiently. In this paper, we propose a novel framework that empowers unified models to autonomously switch between generation strategies based on instruction complexity and model capability. To achieve this, we construct a hierarchical data pipeline that constructs execution paths across three adaptive modes: direct generation for simple cases, self-reflection for quality refinement, and multi-step planning for decomposing complex scenarios. Building on this pipeline, we contribute a high-quality dataset with over 50,000 samples and implement a two-stage training strategy comprising SFT and RL. Specifically, we design step-wise reasoning rewards to ensure logical consistency and intra-group complexity penalty to prevent redundant computational overhead. Extensive experiments demonstrate that our method outperforms existing baselines on X2I, achieving superior generation fidelity among simple-to-complex instructions. The code is released at https://github.com/WeChatCV/Interleaved_Visual_Reasoner.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。