仅用30个样本即可实现扩散模型的通用空间控制，大幅降低训练成本。
Universal Few-Shot Spatial Control for Diffusion Models
- 通过少样本支持对与查询条件匹配，动态生成任务专属控制特征。
- 仅用0.1%数据即达监督基线性能，六项新任务均实现精细控制。
- 适配多种扩散模型结构，无需重新训练，适合快速部署新控制需求。
预训练文生图扩散模型中的空间条件控制已显著提升生成图像的结构精细度。然而，现有控制适配器在面对与训练任务差异较大的新空间控制场景时，泛化能力有限且训练成本高。为此，我们提出通用少样本控制（UFC），一种能适应全新空间条件的灵活少样本适配器。给定少量未见任务的图像-条件对及一个查询条件，UFC利用查询与支持条件间的类比关系，通过匹配机制和小规模参数更新，构建任务特定的控制特征。在六个新空间控制任务上的实验表明，UFC仅需30个标注样本微调，即可实现与空间条件一致的精细控制。值得注意的是，当仅使用完整训练数据的0.1%进行微调时，UFC在多个控制任务中表现媲美全监督基线。我们还验证了UFC对不同扩散主干（UNet与DiT）的无差别适用性。代码已开源：https://github.com/kietngt00/UFC。
原文摘要 · Abstract (English)
Spatial conditioning in pretrained text-to-image diffusion models has significantly improved fine-grained control over the structure of generated images. However, existing control adapters exhibit limited adaptability and incur high training costs when encountering novel spatial control conditions that differ substantially from the training tasks. To address this limitation, we propose Universal Few-Shot Control (UFC), a versatile few-shot control adapter capable of generalizing to novel spatial conditions. Given a few image-condition pairs of an unseen task and a query condition, UFC leverages the analogy between query and support conditions to construct task-specific control features, instantiated by a matching mechanism and an update on a small set of task-specific parameters. Experiments on six novel spatial control tasks show that UFC, fine-tuned with only 30 annotated examples of novel tasks, achieves fine-grained control consistent with the spatial conditions. Notably, when fine-tuned with 0.1% of the full training data, UFC achieves competitive performance with the fully supervised baselines in various control tasks. We also show that UFC is applicable agnostically to various diffusion backbones and demonstrate its effectiveness on both UNet and DiT architectures. Code is available at https://github.com/kietngt00/UFC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。