用分拆思维链和多维奖励,让视频生成音频更准更自然。
PrismAudio: Decomposed Chain-of-Thoughts and Multi-dimensional Rewards for Video-to-Audio Generation
- 分四个模块分别处理语义、时间、音质和空间逻辑,每块配专属奖励
- 在真实场景数据集上四项指标全领先,跨域表现也强
- 适合做音视频生成、人机交互或内容创作的开发者参考
视频到音频(V2A)生成需平衡语义一致性、音画时序同步性、听觉美感和空间准确性四大感知维度;但现有方法因单一损失函数导致目标纠缠且缺乏人类偏好对齐。我们提出PrismAudio,首个将强化学习引入V2A生成的框架,采用专属性思维链(CoT)规划。通过将整体推理分解为语义、时间、美学与空间四个专用CoT模块,并分别匹配对应奖励函数,实现多维强化学习优化,解决目标纠缠问题并保持可解释性。为提升计算效率,提出Fast-GRPO,采用混合常微分方程-随机微分方程采样,显著降低训练开销。同时构建AudioCanvas基准,覆盖300类单事件与501个复杂多事件样本,分布更均衡、场景更真实。实验表明,PrismAudio在VGGSound(域内)与AudioCanvas(域外)上均达到四项维度最优性能。
原文摘要 · Abstract (English)
Video-to-Audio (V2A) generation requires balancing four critical perceptual dimensions: semantic consistency, audio-visual temporal synchrony, aesthetic quality, and spatial accuracy; yet existing methods suffer from objective entanglement that conflates competing goals in single loss functions and lack human preference alignment. We introduce PrismAudio, the first framework to integrate Reinforcement Learning into V2A generation with specialized Chain-of-Thought (CoT) planning. Our approach decomposes monolithic reasoning into four specialized CoT modules (Semantic, Temporal, Aesthetic, and Spatial CoT), each paired with targeted reward functions. This CoT-reward correspondence enables multidimensional RL optimization that guides the model to jointly generate better reasoning across all perspectives, solving the objective entanglement problem while preserving interpretability. To make this optimization computationally practical, we propose Fast-GRPO, which employs hybrid ODE-SDE sampling that dramatically reduces the training overhead compared to existing GRPO implementations. We also introduce AudioCanvas, a rigorous benchmark that is more distributionally balanced and covers more realistically diverse and challenging scenarios than existing datasets, with 300 single-event classes and 501 multi-event samples. Experimental results demonstrate that PrismAudio achieves state-of-the-art performance across all four perceptual dimensions on both the in-domain VGGSound test set and out-of-domain AudioCanvas benchmark. The project page is available at https://PrismAudio.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。