让扩散模型精准理解并执行细粒度图像修改指令
CogniEdit: Dense Gradient Flow Optimization for Fine-Grained Image Editing
- 通过密集梯度流实现全程优化，提升指令追踪能力
- 在COCO-Edit和ImageNet-Edit上显著优于现有方法
- 适合需要精确控制颜色、位置等细节的编辑场景
基于指令的扩散模型图像编辑已取得显著进展，但现有方法在处理颜色、位置、数量等细粒度指令时仍存在困难。尽管近期方法采用分组相对策略优化（GRPO）进行对齐，但仅在单个采样步骤中优化，反馈稀疏，限制了轨迹级控制。我们提出统一框架CogniEdit，结合多模态推理与密集奖励优化，将梯度跨连续去噪步骤传播，实现采样过程中的轨迹级梯度流动。方法包含三部分：(1) 多模态大语言模型将复杂指令分解为可执行指令；(2) 动态标记焦点重定位，自适应强调细粒度属性；(3) 基于密集GRPO的优化，跨步骤传播梯度以实现轨迹级监督。大量实验表明，CogniEdit在基准数据集上实现了细粒度指令遵循、视觉质量与编辑保持性之间的最佳平衡。
原文摘要 · Abstract (English)
Instruction-based image editing with diffusion models has achieved impressive results, yet existing methods struggle with fine-grained instructions specifying precise attributes such as colors, positions, and quantities. While recent approaches employ Group Relative Policy Optimization (GRPO) for alignment, they optimize only at individual sampling steps, providing sparse feedback that limits trajectory-level control. We propose a unified framework CogniEdit, combining multi-modal reasoning with dense reward optimization that propagates gradients across consecutive denoising steps, enabling trajectory-level gradient flow through the sampling process. Our method comprises three components: (1) Multi-modal Large Language Models for decomposing complex instructions into actionable directives, (2) Dynamic Token Focus Relocation that adaptively emphasizes fine-grained attributes, and (3) Dense GRPO-based optimization that propagates gradients across consecutive steps for trajectory-level supervision. Extensive experiments on benchmark datasets demonstrate that our CogniEdit achieves state-of-the-art performance in balancing fine-grained instruction following with visual quality and editability preservation
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。