发现扩散模型生成动作图像时会错乱角色关系，提出简单方法有效纠正。
Role Bias in Diffusion Models: Diagnosing and Mitigating through Intermediate Decomposition
- 通过中间构图训练模型理解动作角色方向
- 修复后人类偏好度超78%优于现有方法
- 适合改进图像生成中角色混淆问题的研究者
文本到图像扩散模型虽能生成逼真图像，但在组合性图像生成上表现不佳。本文提出RoleBench基准，评估基于动作关系的组合泛化能力（如‘鼠标追猫’）。结果显示，主流模型普遍错误地生成高频反向关系（如‘猫追鼠’），称为角色坍缩。以往研究归因于架构限制或数据不足，我们发现当反向关系常见时，模型即使对类似中间关系（如‘鼠追男孩’）也难以生成，说明问题源于分布不对称而非罕见组合缺失。据此提出轻量级框架ReBind，通过精心设计的主动/被动中间构图进行微调，实现角色绑定学习。实验表明，简单微调可显著缓解角色坍缩，人类偏好度超过78%，优于当前最优方法。研究揭示了组合失败中的分布不对称机制，提供了一条简单有效的泛化改进路径。
原文摘要 · Abstract (English)
Text-to-image (T2I) diffusion models exhibit impressive photorealistic image generation capabilities, yet they struggle in compositional image generation. In this work, we introduce RoleBench, a benchmark focused on evaluating compositional generalization in action-based relations (e.g., "mouse chasing cat"). We show that state-of-the-art T2I models and compositional generation methods consistently default to frequent reversed relations (i.e., "cat chasing mouse"), a phenomenon we call role collapse. Related works attribute this to the model's architectural limitation or underrepresentation in the data. Our key insight reveals that while models fail on rare compositions when their inversions are common, they can successfully generate similar intermediate compositions (e.g., "mouse chasing boy"), suggesting that this limitation is also due to the presence of frequent counterparts rather than just the absence of rare compositions. Motivated by this, we hypothesize that directional decomposition can gradually mitigate role collapse. We test this via ReBind, a lightweight framework that teaches role bindings using carefully selected active/passive intermediate compositions. Experiments suggest that intermediate compositions through simple fine-tuning can significantly reduce role collapse, with humans preferring ReBind more than 78% compared to state-of-the-art methods. Our findings highlight the role of distributional asymmetries in compositional failures and offer a simple, effective path for improving generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。