用线性注意力提升图像动画效率与流畅度,保持外观一致且可控制。
Consistent and Controllable Image Animation with Motion Linear Diffusion Transformers
- 改用线性注意力替代自注意力,降低计算开销。
- 通过运动残差学习实现帧间平滑过渡,减少突变。
- 支持动作迁移与视频编辑,适合需要稳定动画的场景。
图像动画虽借助扩散模型取得显著进展,但保持输入图像外观一致性及避免运动突变仍是难题。当前方法多依赖U-Net架构,难以媲美最新的文本到视频生成模型。此外,传统Transformer中的自注意力机制呈二次复杂度,计算成本高。为此,我们提出MiraMo框架:(1)采用基础文本到视频架构,以线性注意力替代原生自注意力,降低计算开销并保持生成质量;(2)引入新型运动残差学习范式,聚焦建模运动动态而非直接预测帧,提升时序一致性;(3)推理阶段结合DCT噪声优化策略抑制突发动态伪影,并配备动态控制模块平衡运动平滑性与表现力。大量实验表明,MiraMo在生成一致性、流畅性和可控性方面优于现有方法,且推理速度更快。此外,我们在动作迁移和视频编辑任务中验证了其泛化能力。
原文摘要 · Abstract (English)
Image animation has seen significant progress, driven by the powerful generative capabilities of diffusion models. However, maintaining appearance consistency with static input images and mitigating abrupt motion transitions in generated animations remain persistent challenges. While text-to-video (T2V) generation has demonstrated impressive performance with diffusion transformer models, the image animation field still largely relies on U-Net-based diffusion models, which lag behind the latest T2V approaches. Moreover, the quadratic complexity of vanilla self-attention mechanisms in Transformers imposes heavy computational demands, making image animation particularly resource-intensive. To address these issues, we propose MiraMo, a framework designed to enhance efficiency, appearance consistency, and motion smoothness in image animation. Specifically, MiraMo introduces three key elements: (1) A foundational text-to-video architecture replacing vanilla self-attention with efficient linear attention to reduce computational overhead while preserving generation quality; (2) A novel motion residual learning paradigm that focuses on modeling motion dynamics rather than directly predicting frames, improving temporal consistency; and (3) A DCT-based noise refinement strategy during inference to suppress sudden motion artifacts, complemented by a dynamics control module to balance motion smoothness and expressiveness. Extensive experiments against state-of-the-art methods validate the superiority of MiraMo in generating consistent, smooth, and controllable animations with accelerated inference speed. Additionally, we demonstrate the versatility of MiraMo through applications in motion transfer and video editing tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。