通过提供干扰项来防止适配器学习错误关联,提升文本图像生成准确性。
Preventing Shortcuts in Adapter Training via Providing the Shortcuts
- 训练时引入辅助模块显式处理干扰因素(如姿态、光照)
- 使适配器专注学习目标属性,生成质量与提示遵循度显著提升
- 适合需要精准风格/身份注入的个性化图像生成场景
基于适配器的训练已成为扩展强大基础图像生成模型能力的关键机制,支持个性化和风格化文本到图像合成。这些适配器通常使用单图重建目标捕捉特定目标属性(如身份),但输入图像不可避免包含多种视觉因素,导致适配器容易将目标属性与姿势、表情、光照等偶然因素混淆。这种虚假相关性限制了泛化能力,并妨碍模型遵循输入文本提示。本文揭示了一种简单而有效的方法:在适配器训练过程中提供希望消除的干扰项。在快捷方式重路由适配器训练中,混淆因素通过控制网或LoRA等辅助模块处理,从而消除适配器内化它们的动机。推理阶段移除辅助模块。应用于面部和全身身份注入任务时,该方法显著提升了生成质量、多样性和提示遵循度。结果表明,在大模型时代,若需解耦表示,最有效的路径或许是为不应学习的内容建立快捷方式。
原文摘要 · Abstract (English)
Adapter-based training has emerged as a key mechanism for extending the capabilities of powerful foundation image generators, enabling personalized and stylized text-to-image synthesis. These adapters are typically trained to capture a specific target attribute, such as subject identity, using single-image reconstruction objectives. However, because the input image inevitably contains a mixture of visual factors, adapters are prone to entangle the target attribute with incidental ones, such as pose, expression, and lighting. This spurious correlation problem limits generalization and obstructs the model's ability to adhere to the input text prompt. In this work, we uncover a simple yet effective solution: provide the very shortcuts we wish to eliminate during adapter training. In Shortcut-Rerouted Adapter Training, confounding factors are routed through auxiliary modules, such as ControlNet or LoRA, eliminating the incentive for the adapter to internalize them. The auxiliary modules are then removed during inference. When applied to tasks like facial and full-body identity injection, our approach improves generation quality, diversity, and prompt adherence. These results point to a general design principle in the era of large models: when seeking disentangled representations, the most effective path may be to establish shortcuts for what should NOT be learned.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。