无需训练即可实现高质量主体驱动图像生成,解放扩散模型的零样本潜力。
FreeCus: Free Lunch Subject-driven Customization in Diffusion Transformers
- 通过注意力共享机制保留主体布局与编辑灵活性。
- 改进特征提取,实现细粒度控制,在多个数据集上达到顶尖效果。
- 兼容现有修复与控制模块,适合快速原型设计与创意应用。
随着扩散变压器(DiT)在文本到图像生成中的突破,主体驱动技术日益用于高保真定制化生成,保持参考输入中的主体身份,推动设计与娱乐创新。现有方法通常依赖针对特定主体的可训练文本嵌入或在大规模数据集上训练专用编码器,严重限制了实际应用。更重要的是,当前方法未能充分发挥现代扩散变压器(如Flux系列)的固有零样本潜力。为此,我们提出FreeCus,一种真正的无训练框架,通过三项关键创新激活DiT能力:1)引入关键注意力共享机制,捕捉主体布局完整性并保留重要编辑灵活性;2)基于对DiT动态偏移的分析,提出优化变体,显著提升细粒度特征提取;3)集成先进多模态大语言模型(MLLM),增强跨模态语义表示。大量实验表明,该方法成功解锁了DiT在多样化场景下的零样本一致性主体生成能力,性能达到或超过需额外训练的方法。值得注意的是,本框架与现有修复管线和控制模块无缝兼容,带来更沉浸式体验。代码已公开:https://github.com/Monalissaa/FreeCus。
原文摘要 · Abstract (English)
In light of recent breakthroughs in text-to-image (T2I) generation, particularly with diffusion transformers (DiT), subject-driven technologies are increasingly being employed for high-fidelity customized production that preserves subject identity from reference inputs, enabling thrilling design workflows and engaging entertainment. Existing alternatives typically require either per-subject optimization via trainable text embeddings or training specialized encoders for subject feature extraction on large-scale datasets. Such dependencies on training procedures fundamentally constrain their practical applications. More importantly, current methodologies fail to fully leverage the inherent zero-shot potential of modern diffusion transformers (e.g., the Flux series) for authentic subject-driven synthesis. To bridge this gap, we propose FreeCus, a genuinely training-free framework that activates DiT's capabilities through three key innovations: 1) We introduce a pivotal attention sharing mechanism that captures the subject's layout integrity while preserving crucial editing flexibility. 2) Through a straightforward analysis of DiT's dynamic shifting, we propose an upgraded variant that significantly improves fine-grained feature extraction. 3) We further integrate advanced Multimodal Large Language Models (MLLMs) to enrich cross-modal semantic representations. Extensive experiments reflect that our method successfully unlocks DiT's zero-shot ability for consistent subject synthesis across diverse contexts, achieving state-of-the-art or comparable results compared to approaches that require additional training. Notably, our framework demonstrates seamless compatibility with existing inpainting pipelines and control modules, facilitating more compelling experiences. Our code is available at: https://github.com/Monalissaa/FreeCus.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。