通过音频和文本微调，让视频模型精准生成特定动作。
Sound Sparks Motion: Audio and Text Tuning for Video Editing
- 仅调整音频隐变量和文本残差，无需修改模型权重。
- 能实现原模型难以达成的局部动作编辑，且保持画面质量。
- 适合想在不训练模型前提下精确控制视频动作的创作者。
以运动为中心的视频编辑对大型生成视频模型仍具挑战，它们虽擅长外观变化，却难以在现有片段中生成特定、局部的动作或状态转换。我们提出 Sound Sparks Motion，一种无需训练的框架，通过在测试时微调模型内部的多模态条件信号，实现音视频生成模型中的运动编辑。该方法仅调整两个轻量级变量：来自源视频的音频隐变量与文本条件中的残差扰动。实验表明，这种组合可促使模型实现原本在纯提示控制下难以实现的运动编辑。由于无法直接评估文本与动作的时间对齐，我们使用视觉-语言模型提供反馈，判断目标动作是否出现在生成视频中，以此作为语义优化目标。正则化与感知-时间约束确保内容一致性和视觉质量。此外，所学的隐变量控制可在不同视频间迁移，表明其捕捉的是可复用的运动编辑方向而非过拟合单例。结果表明，特别是通过音频路径的多模态条件微调，是实现运动感知视频编辑的有前景方向，且测试时微调可作为揭示模型中嵌入的运动控制机制的轻量探测工具。代码与数据见项目主页：https://amirhossein-razlighi.github.io/Sound_Sparks_Motion/
原文摘要 · Abstract (English)
Motion-centric video editing remains difficult for large generative video models, which often respond well to appearance changes but struggle to produce specific, localized actions or state transitions in an existing clip. We introduce Sound Sparks Motion, a training-free framework that enables motion editing in an audio-visual video generation model by tuning its internal multimodal conditioning signals at test time. Rather than modifying model weights, our method tunes only two lightweight variables: an audio latent derived from the source video and a residual perturbation in the text-conditioning. We find that this combination can encourage motion edits that the underlying model often struggles to realize under prompt-only control. Since there is no direct way to evaluate temporal alignment between text and motion, we guide the tuning process using a vision-language model that provides feedback indicating whether the intended motion appears in the generated video. This simple supervision yields an effective semantic objective for motion editing, while regularization and perceptual-temporal constraints help preserve content and visual quality. Beyond per-video tuning, we show that the learned latent controls are transferable across videos, suggesting that they capture reusable motion-edit directions rather than overfitting to a single example. Our results highlight multimodal conditioning tuning, particularly through the audio pathway, as a promising direction for motion-aware video editing, and suggest that test-time tuning can serve as a lightweight probing mechanism that helps reveal latent motion controls embedded in the model's multimodal conditioning. Code and data are available via our project page: https://amirhossein-razlighi.github.io/Sound_Sparks_Motion/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。