arXiv:2411.17949cs.CV2024-11CVPR被引 9

让扩散模型精准控制多个物体的位置和属性,提升复杂场景生成能力。

ROICtrl: Boosting Instance Control for Visual Generation

  • 引入ROI-Unpool操作,实现高分辨率特征图上区域的精确控制。
  • 在多实例生成任务中显著提升控制精度,计算开销更低。
  • 兼容主流控制模块,可扩展至ControlNet、IP-Adapter等模型。

自然语言在关联多个实例的位置与属性信息时表现不佳,限制了当前文本驱动视觉生成模型仅能处理少量主导实例的简单构图。为解决此问题,本文通过引入区域实例控制(regional instance control),使每个实例由边界框与自由格式描述共同控制。以往方法依赖隐式位置编码或显式注意力掩码分离感兴趣区域(ROI),导致坐标注入不准确或计算开销过大。受目标检测中ROI-Align启发,本文提出互补操作ROI-Unpool,结合两者可在高分辨率特征图上实现显式、高效且精准的区域操作。基于此,提出ROICtrl——一个适配预训练扩散模型的适配器,实现精确的区域实例控制。ROICtrl兼容社区微调的扩散模型,以及现有的基于空间的附加模块(如ControlNet、T2I-Adapter)和基于嵌入的附加模块(如IP-Adapter、ED-LoRA),拓展其在多实例生成中的应用。实验表明,ROICtrl在区域实例控制任务中表现更优,同时显著降低计算成本。

原文摘要 · Abstract (English)

Natural language often struggles to accurately associate positional and attribute information with multiple instances, which limits current text-based visual generation models to simpler compositions featuring only a few dominant instances. To address this limitation, this work enhances diffusion models by introducing regional instance control, where each instance is governed by a bounding box paired with a free-form caption. Previous methods in this area typically rely on implicit position encoding or explicit attention masks to separate regions of interest (ROIs), resulting in either inaccurate coordinate injection or large computational overhead. Inspired by ROI-Align in object detection, we introduce a complementary operation called ROI-Unpool. Together, ROI-Align and ROI-Unpool enable explicit, efficient, and accurate ROI manipulation on high-resolution feature maps for visual generation. Building on ROI-Unpool, we propose ROICtrl, an adapter for pretrained diffusion models that enables precise regional instance control. ROICtrl is compatible with community-finetuned diffusion models, as well as with existing spatial-based add-ons (\eg, ControlNet, T2I-Adapter) and embedding-based add-ons (\eg, IP-Adapter, ED-LoRA), extending their applications to multi-instance generation. Experiments show that ROICtrl achieves superior performance in regional instance control while significantly reducing computational costs.

扩散模型实例控制多物体生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。