用轻量框架实现多模态音视频控制,无需修改模型结构
AVControl: Efficient Framework for Training Audio-Visual Controls
- 为每种控制模态独立训练LoRA,通过并行画布注入参考信号
- 在VACE基准上超越所有基线,深度与姿态控制效果最佳
- 适合需要快速扩展新控制模态的研究者和开发者
音视频生成需支持多种控制模态,如深度、姿态、相机轨迹和音频变换,但现有方法或仅训练单一模型支持固定控制,或为每新增模态引入昂贵的架构改动。我们提出AVControl,基于LTX-2联合音视频基础模型的轻量可扩展框架,将每个控制模态作为独立LoRA训练,并在注意力层中以额外标记形式注入参考信号,无需额外架构修改。实验表明,简单将图像上下文方法拓展至视频无法实现结构化控制,而我们的并行画布方案有效解决此问题。在VACE基准上,我们在深度与姿态引导生成、修复和外推任务中均优于所有对比基线,相机控制和音视频任务表现也具竞争力。该框架支持多种独立训练的模态:空间对齐控制(深度、姿态、边缘)、含内参的相机轨迹、稀疏运动控制、视频编辑,以及目前已知首个针对联合生成模型的模块化音视频控制。方法兼具计算与数据效率:每种模态仅需小规模数据集,在数百至数千步内收敛,远低于单体模型的开销。代码与训练好的LoRA检查点已公开。
原文摘要 · Abstract (English)
Controlling video and audio generation requires diverse modalities, from depth and pose to camera trajectories and audio transformations, yet existing approaches either train a single monolithic model for a fixed set of controls or introduce costly architectural changes for each new modality. We introduce AVControl, a lightweight, extendable framework built on LTX-2, a joint audio-visual foundation model, where each control modality is trained as a separate LoRA on a parallel canvas that provides the reference signal as additional tokens in the attention layers, requiring no architectural changes beyond the LoRA adapters themselves. We show that simply extending image-based in-context methods to video fails for structural control, and that our parallel canvas approach resolves this. On the VACE Benchmark, we outperform all evaluated baselines on depth- and pose-guided generation, inpainting, and outpainting, and show competitive results on camera control and audio-visual benchmarks. Our framework supports a diverse set of independently trained modalities: spatially-aligned controls such as depth, pose, and edges, camera trajectory with intrinsics, sparse motion control, video editing, and, to our knowledge, the first modular audio-visual controls for a joint generation model. Our method is both compute- and data-efficient: each modality requires only a small dataset and converges within a few hundred to a few thousand training steps, a fraction of the budget of monolithic alternatives. We publicly release our code and trained LoRA checkpoints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。