无需额外计算，提升文本引导修复的准确性和质量。
FreeCond: Free Lunch in the Input Conditions of Text-Guided Inpainting
- 仅调整输入掩码和图像条件，缓解模型训练偏差。
- 在复杂指令下，CLIP得分最高提升60%。
- 适合需要精准控制修复区域的研究与应用。
本研究针对稳定扩散修复（SDI）在遵循文本提示与掩码指令不一致时性能下降的问题，分析了模型内部表示，发现掩码输入会影响交叉注意力层。通过将文本键向量适配至输入掩码，实现对指定区域的精确绘制。基于此提出FreeCond方法，仅调整输入掩码值和图像条件频率，使交叉注意力特征更符合模型训练偏差，在不增加计算成本的前提下显著提升生成质量，尤其在用户输入偏离训练数据分布时表现优异。大量实验表明，FreeCond可适用于任意SDI模型，如在CLIP分数上分别提升60%（SDI）和58%（SDXLI）。
原文摘要 · Abstract (English)
In this study, we aim to determine and solve the deficiency of Stable Diffusion Inpainting (SDI) in following the instruction of both prompt and mask. Due to the training bias from masking, the inpainting quality is hindered when the prompt instruction and image condition are not related. Therefore, we conduct a detailed analysis of the internal representations learned by SDI, focusing on how the mask input influences the cross-attention layer. We observe that adapting text key tokens toward the input mask enables the model to selectively paint within the given area. Leveraging these insights, we propose FreeCond, which adjusts only the input mask condition and image condition. By increasing the latent mask value and modifying the frequency of image condition, we align the cross-attention features with the model's training bias to improve generation quality without additional computation, particularly when user inputs are complicated and deviate from the training setup. Extensive experiments demonstrate that FreeCond can enhance any SDI-based model, e.g., yielding up to a 60% and 58% improvement of SDI and SDXLI in the CLIP score.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。