arXiv:2409.12576cs.CV2024-09被引 32

让多角色图像生成保持脸、衣、发、体型一致,讲好连贯故事

StoryMaker: Towards Holistic Consistent Characters in Text-to-image Generation

  • 用面部+裁剪角色图融合特征,保留整体形象一致性
  • 分区域约束注意力,防止角色与背景混淆,提升角色独立性
  • 支持多角色叙事生成,适合需要连贯角色的创意应用

免微调的个性化图像生成方法在保持多角色面部一致性方面已取得显著进展。然而,场景中缺乏整体一致性(如服装、发型、体型)限制了其构建连贯叙事的能力。本文提出StoryMaker,一种可保持面部、服装、发型和身体一致性的个性化方案,从而支持通过一系列图像讲述完整故事。StoryMaker结合面部身份信息与裁剪的角色图像(包含服装、发型、体态),利用位置感知的Perceiver Resampler(PPR)提取独特角色特征。为避免多角色与背景之间的干扰,采用分割掩码配合均方误差损失,分别约束不同角色及背景的交叉注意力影响区域。同时,训练生成网络时引入姿态条件,促进姿态解耦。此外,使用LoRA提升生成质量与保真度。实验验证了方法的有效性。StoryMaker具备广泛应用潜力,且兼容其他社会性插件。代码与模型权重已公开于https://github.com/RedAIGC/StoryMaker。

原文摘要 · Abstract (English)

Tuning-free personalized image generation methods have achieved significant success in maintaining facial consistency, i.e., identities, even with multiple characters. However, the lack of holistic consistency in scenes with multiple characters hampers these methods' ability to create a cohesive narrative. In this paper, we introduce StoryMaker, a personalization solution that preserves not only facial consistency but also clothing, hairstyles, and body consistency, thus facilitating the creation of a story through a series of images. StoryMaker incorporates conditions based on face identities and cropped character images, which include clothing, hairstyles, and bodies. Specifically, we integrate the facial identity information with the cropped character images using the Positional-aware Perceiver Resampler (PPR) to obtain distinct character features. To prevent intermingling of multiple characters and the background, we separately constrain the cross-attention impact regions of different characters and the background using MSE loss with segmentation masks. Additionally, we train the generation network conditioned on poses to promote decoupling from poses. A LoRA is also employed to enhance fidelity and quality. Experiments underscore the effectiveness of our approach. StoryMaker supports numerous applications and is compatible with other societal plug-ins. Our source codes and model weights are available at https://github.com/RedAIGC/StoryMaker.

图像生成角色一致性多角色叙事LoRA

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。