解决多人姿态生成中肢体扭曲与干扰问题,提升复杂场景下图像质量。
TrioPose: Native Triple-Stream Diffusion Transformers for Pose-Guided Text-to-Image Generation

- 将姿态作为独立模态,用双残差注入保持预训练分布稳定。
- 在Human-Art数据集上达到64.33的AP,比之前方法提升30%。
- 适合需要高精度多人姿态生成的应用,如虚拟人设计、动画制作。
姿态引导的文生图生成在多人复杂场景中常出现肢体扭曲和特征干扰问题。现有基于UNet的适配器难以处理长程空间依赖,而新兴的多模态扩散变换器(MM-DiTs)虽具全局建模优势,但简单的信号拼接会严重破坏预训练潜在分布。为此,我们提出TrioPose,基于SD3.5M架构的原生姿态驱动框架。具体而言,引入分层激活与零初始化双残差注入的三流姿态感知扩散变换器(TSPA-DiT),将姿态视为独立模态,有效施加几何约束同时保持潜空间稳定性。针对严重多实例遮挡问题,设计可学习的关系偏置掩码,将拓扑连接细分为物理状态并映射为连续注意力软约束,显著解耦实例间干扰。此外,提出姿态引导的空间损失加权策略,利用热图误差图调制扩散目标,仅在易扭曲区域强化解剖监督。大量实验表明,TrioPose在Human-Art、CrowdPose和OCHuman等挑战性基准上达到顶尖性能,尤其在Human-Art上取得64.33的AP,较先前方法提升30%,并在复杂多人生成中树立了视觉保真度与图文语义对齐的新标准。
原文摘要 · Abstract (English)
Pose-guided text-to-image generation often suffers from limb distortions and feature crosstalk in complex multi-person scenarios. While existing UNet-based adapters struggle with long-range spatial dependencies, emerging Multimodal Diffusion Transformers (MM-DiTs) offer superior global modeling. However, naive signal concatenation in MM-DiTs severely disrupts pre-trained latent distributions. To address this, we propose TrioPose, a native pose-driven framework built upon the SD3.5M architecture. Specifically, we introduce a Triple-Stream Pose-Aware DiT (TSPA-DiT) that treats pose as an independent modality. It employs layer-wise activation and zero-initialized dual-residual injection to smoothly enforce geometric constraints while preserving pre-trained latent stability. To resolve severe multi-instance occlusions, we design a Learnable Relational Bias Mask that categorizes topological connectivity into fine-grained physical states, mapping them into continuous attention soft constraints to effectively decouple inter-instance interference. Furthermore, a Pose-Guided Spatial Loss Weighting strategy modulates the native diffusion objective using heatmap-derived error maps, focusing anatomical supervision strictly on distortion-prone regions. Extensive experiments demonstrate that TrioPose achieves state-of-the-art performance across challenging benchmarks, including Human-Art, CrowdPose, and OCHuman. Notably, it attains an AP of $64.33$ on Human-Art, representing a $30\%$ improvement over prior arts, while setting new standards for visual fidelity and text-image semantic alignment in complex multi-human generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。