让文字和图像融合生成新物体，解决模型偏倚问题。
Novel Object Synthesis via Adaptive Text-Image Harmony
- 引入动态缩放因子与注入步骤，平衡文本与图像特征。
- 设计带噪声参数的损失函数，兼顾编辑性和图像保真度。
- 提出相似度评分函数，自适应调参实现图文和谐融合。
本文研究将物体文本与物体图像结合以生成新物体图像的任务。然而，大多数扩散模型在此任务中表现不佳，常因输入不平衡而过度依赖文本或图像。为此，我们提出一种简单有效的方法——自适应文本-图像和谐（ATIH），用于生成新颖且出人意料的物体。首先，在交叉注意力中引入尺度因子，于自注意力中设置注入步骤，以平衡跨模态特征并保留图像信息。其次，设计含噪声参数的均衡损失函数，确保物体图像兼具最优可编辑性与高保真度。第三，提出新型相似度评分函数，同时最大化生成图像与输入文本/图像的相似性，并动态平衡二者以实现和谐融合。大量实验验证了该方法的有效性，成功生成如‘疣猴-玻璃罐’等惊艳组合。项目主页：https://xzr52.github.io/ATIH/
原文摘要 · Abstract (English)
In this paper, we study an object synthesis task that combines an object text with an object image to create a new object image. However, most diffusion models struggle with this task, \textit{i.e.}, often generating an object that predominantly reflects either the text or the image due to an imbalance between their inputs. To address this issue, we propose a simple yet effective method called Adaptive Text-Image Harmony (ATIH) to generate novel and surprising objects. First, we introduce a scale factor and an injection step to balance text and image features in cross-attention and to preserve image information in self-attention during the text-image inversion diffusion process, respectively. Second, to better integrate object text and image, we design a balanced loss function with a noise parameter, ensuring both optimal editability and fidelity of the object image. Third, to adaptively adjust these parameters, we present a novel similarity score function that not only maximizes the similarities between the generated object image and the input text/image but also balances these similarities to harmonize text and image integration. Extensive experiments demonstrate the effectiveness of our approach, showcasing remarkable object creations such as colobus-glass jar. Project page: https://xzr52.github.io/ATIH/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。