arXiv:2411.19459cs.CVcs.LG2024-11被引 4

用文字和图片生成人体动作视频,更灵活可控。

Fleximo: Towards Flexible Text-to-Human Motion Video Generation

  • 仅凭图片和文字生成动作视频,无需参考视频
  • 自研骨架重缩放与适配器,解决尺寸不匹配问题
  • 支持任意长度动作生成,适合影视创作与虚拟人

现有方法依赖从参考视频中提取姿态序列,限制了灵活性与控制力,且姿态检测误差常导致视频质量下降。本文提出新任务:仅通过参考图像和自然语言生成人体动作视频,提升可操作性。为训练端到端模型,需海量高质量图文-动作视频对,难以获取。为此,我们提出Fleximo框架,利用大规模预训练的文本到3D动作模型。该方法面临挑战:生成的骨架与参考图像尺度不一致,且细节缺失。为此,我们设计基于锚点的重缩放方法,以及骨架适配器以填补细节并弥合文本到动作与动作到视频生成之间的差距。同时引入视频精炼流程提升画质。采用大语言模型将自然语言分解为离散动作序列,实现任意长度动作视频生成。为评估性能,我们构建新基准MotionBench(含20身份、20动作共400视频),并提出新指标MotionScore评估动作跟随准确性。定性与定量结果均表明,本方法优于现有文本条件图像到视频生成方法。所有代码与模型权重将公开。

原文摘要 · Abstract (English)

Current methods for generating human motion videos rely on extracting pose sequences from reference videos, which restricts flexibility and control. Additionally, due to the limitations of pose detection techniques, the extracted pose sequences can sometimes be inaccurate, leading to low-quality video outputs. We introduce a novel task aimed at generating human motion videos solely from reference images and natural language. This approach offers greater flexibility and ease of use, as text is more accessible than the desired guidance videos. However, training an end-to-end model for this task requires millions of high-quality text and human motion video pairs, which are challenging to obtain. To address this, we propose a new framework called Fleximo, which leverages large-scale pre-trained text-to-3D motion models. This approach is not straightforward, as the text-generated skeletons may not consistently match the scale of the reference image and may lack detailed information. To overcome these challenges, we introduce an anchor point based rescale method and design a skeleton adapter to fill in missing details and bridge the gap between text-to-motion and motion-to-video generation. We also propose a video refinement process to further enhance video quality. A large language model (LLM) is employed to decompose natural language into discrete motion sequences, enabling the generation of motion videos of any desired length. To assess the performance of Fleximo, we introduce a new benchmark called MotionBench, which includes 400 videos across 20 identities and 20 motions. We also propose a new metric, MotionScore, to evaluate the accuracy of motion following. Both qualitative and quantitative results demonstrate that our method outperforms existing text-conditioned image-to-video generation methods. All code and model weights will be made publicly available.

动作生成文本驱动视频合成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。