动态分配专家模块,让图像编辑更精准地响应不同条件。
CARE-Edit: Condition-Aware Routing of Experts for Contextual Image Editing
- 用轻量级路由机制按条件动态选择文本、掩码等专家
- 在多种编辑任务中实现无色偏、无风格漂移的高质量输出
- 适合需要多条件协同的复杂图像编辑场景
统一扩散编辑器通常依赖固定共享主干处理多样化任务,易受任务干扰且难以适应异质需求(如局部与全局、语义与光度)。现有ControlNet及OmniControl变体通过静态拼接或加性适配融合多模态条件(如文本、掩码、参考图),无法动态优先或抑制冲突模态,导致颜色渗出、身份/风格漂移以及多条件输入下的不可预测行为。为此,我们提出条件感知专家路由(CARE-Edit),将模型计算对齐于特定编辑能力。核心是一个轻量级潜在注意力路由器,根据多模态条件与扩散时间步,将编码的扩散令牌分配至四个专用专家——文本、掩码、参考和基础。具体包括:(i) 掩码重绘模块首先精炼用户定义的粗略掩码,以提供精确空间引导;(ii) 路由器采用稀疏Top-K选择,动态分配计算给最相关专家;(iii) 潜在混合模块随后融合专家输出,一致整合语义、空间与风格信息至基图。实验验证了CARE-Edit在擦除、替换、文本驱动编辑与风格迁移等上下文编辑任务中的优异表现。实证分析揭示了各专家的任务特异性行为,凸显动态、条件感知处理在缓解多条件冲突中的关键作用。
原文摘要 · Abstract (English)
Unified diffusion editors often rely on a fixed, shared backbone for diverse tasks, suffering from task interference and poor adaptation to heterogeneous demands (e.g., local vs global, semantic vs photometric). In particular, prevalent ControlNet and OmniControl variants combine multiple conditioning signals (e.g., text, mask, reference) via static concatenation or additive adapters which cannot dynamically prioritize or suppress conflicting modalities, thus resulting in artifacts like color bleeding across mask boundaries, identity or style drift, and unpredictable behavior under multi-condition inputs. To address this, we propose Condition-Aware Routing of Experts (CARE-Edit) that aligns model computation with specific editing competencies. At its core, a lightweight latent-attention router assigns encoded diffusion tokens to four specialized experts--Text, Mask, Reference, and Base--based on multi-modal conditions and diffusion timesteps: (i) a Mask Repaint module first refines coarse user-defined masks for precise spatial guidance; (ii) the router applies sparse top-K selection to dynamically allocate computation to the most relevant experts; (iii) a Latent Mixture module subsequently fuses expert outputs, coherently integrating semantic, spatial, and stylistic information to the base images. Experiments validate CARE-Edit's strong performance on contextual editing tasks, including erasure, replacement, text-driven edits, and style transfer. Empirical analysis further reveals task-specific behavior of specialized experts, showcasing the importance of dynamic, condition-aware processing to mitigate multi-condition conflicts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。