提出双流扩散变换器，实现文本与结构图的高效融合生成高保真人脸。
MMFace-DiT: A Dual-Stream Diffusion Transformer for High-Fidelity Multimodal Face Generation
- 双流架构并行处理文本与结构图，通过共享位置编码注意力深度融合
- 在六种模型上提升40%视觉保真度和提示对齐度，显著改善空间-语义一致性
- 可动态适应不同结构条件，无需重新训练，适合可控人脸生成场景
近期多模态人脸生成模型通过引入分割掩码、草图或边缘图等空间先验，克服了文本到图像扩散模型的空间控制局限。这种多模态融合使生成结果同时符合高层语义意图和低层结构布局。然而，现有方法大多在预训练文本到图像管道基础上附加控制模块或拼接独立单模态网络，存在架构限制、参数冗余，且在模态冲突或潜在空间不匹配时表现不佳，难以实现语义与空间域的协同融合。我们提出MMFace-DiT，一种统一的双流扩散变换器，专为协同多模态人脸合成设计。其核心创新在于双流变换器块，能并行处理空间（掩码/草图）和语义（文本）令牌，并通过共享旋转位置嵌入（RoPE）注意力机制深度融合。该设计避免模态主导，确保对文本和结构先验的强约束，实现前所未有的空间-语义一致性。此外，新颖的模态嵌入器使单一模型可动态适配不同空间条件，无需重训练。在六种先进多模态人脸生成模型上，MMFace-DiT实现了40%的视觉保真度与提示对齐度提升，确立了端到端可控生成建模的新范式。代码与数据集见项目页：https://vcbsl.github.io/MMFace-DiT/
原文摘要 · Abstract (English)
Recent multimodal face generation models address the spatial control limitations of text-to-image diffusion models by augmenting text-based conditioning with spatial priors such as segmentation masks, sketches, or edge maps. This multimodal fusion enables controllable synthesis aligned with both high-level semantic intent and low-level structural layout. However, most existing approaches typically extend pre-trained text-to-image pipelines by appending auxiliary control modules or stitching together separate uni-modal networks. These ad hoc designs inherit architectural constraints, duplicate parameters, and often fail under conflicting modalities or mismatched latent spaces, limiting their ability to perform synergistic fusion across semantic and spatial domains. We introduce MMFace-DiT, a unified dual-stream diffusion transformer engineered for synergistic multimodal face synthesis. Its core novelty lies in a dual-stream transformer block that processes spatial (mask/sketch) and semantic (text) tokens in parallel, deeply fusing them through a shared Rotary Position-Embedded (RoPE) Attention mechanism. This design prevents modal dominance and ensures strong adherence to both text and structural priors to achieve unprecedented spatial-semantic consistency for controllable face generation. Furthermore, a novel Modality Embedder enables a single cohesive model to dynamically adapt to varying spatial conditions without retraining. MMFace-DiT achieves a 40% improvement in visual fidelity and prompt alignment over six state-of-the-art multimodal face generation models, establishing a flexible new paradigm for end-to-end controllable generative modeling. The code and dataset are available on our project page: https://vcbsl.github.io/MMFace-DiT/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。