arXiv:2503.02127cs.CV2025-03被引 3

用空间信息提升扩散模型手部生成质量,避免手指错乱

HanDrawer: Leveraging Spatial Information to Render Realistic Hands Using a Conditional Diffusion Model in Single Stage

  • 通过图卷积提取手部网格的空间结构和物理约束
  • 融合空间特征后,生成手部姿态更自然、无错指现象
  • 适合需要高精度手部生成的研究者与开发者

尽管扩散模型在文本到图像生成中表现优异,但手部姿态生成仍存在严重缺陷,如手指数量错误或动作不自然。为使扩散模型学习手部空间信息以提升生成质量,本文提出HanDrawer模块,利用图卷积层从MANO手部网格顶点中提取隐含的内在空间结构与物理约束,并通过交叉注意力将这些空间特征与其他模态对齐融合。融合后的空间特征被用于引导单阶段扩散模型的去噪过程,从而高质量生成手部区域。为提升特征融合精度,提出位置保持零填充(PPZP)策略,确保空间特征准确注入扩散模型相关层中。通过额外加入手部重建损失与去噪损失,使模型在学习全局图像特征的同时重点关注手部区域。为精确训练与评估,对广泛使用的HaGRID手部姿态数据集进行了细致清洗与重标注,获得高质量多模态数据。定量与定性分析表明,本方法在HaGRID数据集上达到当前最优性能。源代码与增强数据集将在论文接收后公开。

原文摘要 · Abstract (English)

Although diffusion methods excel in text-to-image generation, generating accurate hand gestures remains a major challenge, resulting in severe artifacts, such as incorrect number of fingers or unnatural gestures. To enable the diffusion model to learn spatial information to improve the quality of the hands generated, we propose HanDrawer, a module to condition the hand generation process. Specifically, we apply graph convolutional layers to extract the endogenous spatial structure and physical constraints implicit in MANO hand mesh vertices. We then align and fuse these spatial features with other modalities via cross-attention. The spatially fused features are used to guide a single stage diffusion model denoising process for high quality generation of the hand region. To improve the accuracy of spatial feature fusion, we propose a Position-Preserving Zero Padding (PPZP) fusion strategy, which ensures that the features extracted by HanDrawer are fused into the region of interest in the relevant layers of the diffusion model. HanDrawer learns the entire image features while paying special attention to the hand region thanks to an additional hand reconstruction loss combined with the denoising loss. To accurately train and evaluate our approach, we perform careful cleansing and relabeling of the widely used HaGRID hand gesture dataset and obtain high quality multimodal data. Quantitative and qualitative analyses demonstrate the state-of-the-art performance of our method on the HaGRID dataset through multiple evaluation metrics. Source code and our enhanced dataset will be released publicly if the paper is accepted.

手部生成扩散模型空间建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。