用少量样本微调即可让扩散Transformer实现高质量上下文生成。
In-Context LoRA for Diffusion Transformers
- 直接拼接图像并联合标注，激活模型内在上下文生成能力。
- 仅需20~100张图微调，生成图像更贴合提示词，质量显著提升。
- 无需修改模型结构，适合快速定制化图像生成任务。
近期研究（arXiv:2410.15027）探索了通过简单拼接注意力标记来实现任务无关图像生成的扩散变换器（DiTs）。然而，尽管投入大量计算资源，生成图像的保真度仍不理想。本文重新评估并简化该框架，假设文本到图像的DiTs本身具备上下文生成能力，仅需极少调优即可激活。通过多种任务实验，我们定性证明现有文本到图像的DiTs可在无需任何调优的情况下有效进行上下文生成。基于此，我们提出一种极为简单的流程：（1）拼接图像而非标记，（2）对多张图像进行联合标注，（3）使用小规模数据集（如20~100样本）进行特定任务的LoRA微调，而非依赖大规模数据的全参数微调。我们将其命名为In-Context LoRA（IC-LoRA）。该方法无需修改原始DiT模型，仅调整训练数据。令人惊讶的是，该流程生成的图像集合具有更高保真度且更符合提示词。虽然微调数据具有任务特异性，但架构与流程保持任务无关，为社区提供强大工具，并为产品级任务无关生成系统的研究提供新思路。代码、数据和模型已开源至https://github.com/ali-vilab/In-Context-LoRA。
原文摘要 · Abstract (English)
Recent research arXiv:2410.15027 has explored the use of diffusion transformers (DiTs) for task-agnostic image generation by simply concatenating attention tokens across images. However, despite substantial computational resources, the fidelity of the generated images remains suboptimal. In this study, we reevaluate and streamline this framework by hypothesizing that text-to-image DiTs inherently possess in-context generation capabilities, requiring only minimal tuning to activate them. Through diverse task experiments, we qualitatively demonstrate that existing text-to-image DiTs can effectively perform in-context generation without any tuning. Building on this insight, we propose a remarkably simple pipeline to leverage the in-context abilities of DiTs: (1) concatenate images instead of tokens, (2) perform joint captioning of multiple images, and (3) apply task-specific LoRA tuning using small datasets (e.g., 20~100 samples) instead of full-parameter tuning with large datasets. We name our models In-Context LoRA (IC-LoRA). This approach requires no modifications to the original DiT models, only changes to the training data. Remarkably, our pipeline generates high-fidelity image sets that better adhere to prompts. While task-specific in terms of tuning data, our framework remains task-agnostic in architecture and pipeline, offering a powerful tool for the community and providing valuable insights for further research on product-level task-agnostic generation systems. We release our code, data, and models at https://github.com/ali-vilab/In-Context-LoRA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。