arXiv:2603.16967cs.CVcs.AI2026-03

让AI理解复杂多步图像编辑指令,自动分解任务并逐步完成。

MSRAMIE: Multimodal Structured Reasoning Agent for Multi-instruction Image Editing

  • 通过树状状态与参考图结构,拆解复杂指令为可执行步骤。
  • 指令越复杂,遵循度提升超15%,单次完成率翻倍以上。
  • 无需重新训练,适配现有模型,结果可解释、可控制。

现有基于指令的图像编辑模型在简单、单步指令下表现良好,但在涉及多步、长篇且相互依赖的现实任务中性能下降。主要原因在于缺乏带有复杂多指令标注的训练数据,而收集此类数据成本高昂,重训练亦不现实。为此,我们提出MSRAMIE——一种基于多模态大语言模型(MLLM)的免训练代理框架。该框架将现有编辑模型作为插件组件,通过结构化多模态推理处理多指令任务。其核心是迭代式交互机制:由基于MLLM的Instructor与图像编辑Actor协同工作,并引入创新的树状状态(Tree-of-States)与参考图(Graph-of-References)拓扑结构。推理时,复杂指令被分解为多个编辑步骤,支持状态转移、跨步信息聚合与原始输入回溯,实现对图像编辑空间的系统探索与灵活渐进式输出优化。可可视化的推理拓扑提供可解释、可控的决策路径。实验表明,随着指令复杂度增加,MSRAMIE在指令遵循度上提升超过15%,单次完成全部修改的概率提升超过100%,同时保持感知质量与视觉一致性。

原文摘要 · Abstract (English)

Existing instruction-based image editing models perform well with simple, single-step instructions but degrade in realistic scenarios that involve multiple, lengthy, and interdependent directives. A main cause is the scarcity of training data with complex multi-instruction annotations. However, it is costly to collect such data and retrain these models. To address this challenge, we propose MSRAMIE, a training-free agent framework built on Multimodal Large Language Model (MLLM). MSRAMIE takes existing editing models as plug-in components and handle multi-instruction tasks via structured multimodal reasoning. It orchestrates iterative interactions between an MLLM-based Instructor and an image editing Actor, introducing a novel reasoning topology that comprises the proposed Tree-of-States and Graph-of-References. During inference, complex instructions are decomposed into multiple editing steps which enable state transitions, cross-step information aggregation, and original input recall, which enables systematic exploration of the image editing space and flexible progressive output refinement. The visualizable inference topology further provides interpretable and controllable decision pathways. Experiments show that as the instruction complexity increases, MSRAMIE can improve instruction following over 15% and increases the probability of finishing all modifications in a single run over 100%, while preserving perceptual quality and maintaining visual consistency.

图像编辑多模态推理框架指令理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。