arXiv:2412.06174cs.CV2024-12中稿 · publication in IEE…被引 4

通过融合多尺度光流与神经纹理,提升单张图像动作迁移的鲁棒性。

One-shot Human Motion Transfer via Occlusion-Robust Flow Prediction and Neural Texturing

  • 融合多尺度光流与神经纹理,增强外观与几何一致性
  • 在存在严重自遮挡时仍保持高质量生成结果
  • 适合需要高鲁棒性的动作迁移场景

人体动作迁移旨在用驱动视频动画化静态源图像。尽管近期一拍即合的方法已取得显著进展,但基于2D身体关键点、骨架和语义掩码的方法仍难以准确捕捉源图像与驱动姿态间的对应关系,尤其在运动变化大、关节复杂的情况下。此外,密集姿态(DensePose)的精度下降会损害基于神经渲染方法的图像质量。为此,本文提出统一框架,结合多尺度特征变形与神经纹理映射,以恢复更优的2D外观与2.5D几何结构,部分利用DensePose信息,同时适应其固有的精度局限。模型通过联合训练与融合多种模态,获得对几何误差具有鲁棒性的神经纹理特征,以及能更好保留外观的多尺度密集运动光流。在完整视图与半身视图人体视频数据集上的实验表明,该模型泛化能力强,结果具有竞争力,尤其在处理严重自遮挡等挑战性情况时表现优异。

原文摘要 · Abstract (English)

Human motion transfer aims at animating a static source image with a driving video. While recent advances in one-shot human motion transfer have led to significant improvement in results, it remains challenging for methods with 2D body landmarks, skeleton and semantic mask to accurately capture correspondences between source and driving poses due to the large variation in motion and articulation complexity. In addition, the accuracy and precision of DensePose degrade the image quality for neural-rendering-based methods. To address the limitations and by both considering the importance of appearance and geometry for motion transfer, in this work, we proposed a unified framework that combines multi-scale feature warping and neural texture mapping to recover better 2D appearance and 2.5D geometry, partly by exploiting the information from DensePose, yet adapting to its inherent limited accuracy. Our model takes advantage of multiple modalities by jointly training and fusing them, which allows it to robust neural texture features that cope with geometric errors as well as multi-scale dense motion flow that better preserves appearance. Experimental results with full and half-view body video datasets demonstrate that our model can generalize well and achieve competitive results, and that it is particularly effective in handling challenging cases such as those with substantial self-occlusions.

动作迁移神经纹理遮挡鲁棒

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。