用大模型驱动的多模态框架，实现视频音频的零样本修复。
VAInpaint: Zero-Shot Video-Audio inpainting framework with LLMs-driven Module
- 分步处理：先分割物体，再结合全局与局部描述生成文本指令。
- 音视频修复效果接近当前最佳基准，支持跨场景泛化。
- 适合多媒体编辑、影视后期等需要精准内容删除的场景。
针对音视频混合内容的编辑需求，如何精准移除特定物体及其对应音频仍具挑战。本文提出VAInpaint框架，首先通过分割模型生成掩码，引导视频修复模型去除目标物体；同时，大语言模型（LLM）全局分析场景，区域模型提供局部描述，二者融合后生成文本查询指令，输入至文本驱动的音频分离模型。该音频模型在自定义数据集（包含分割后的MUSIC乐器图像和VGGSound背景）上微调，提升泛化能力。实验表明，该方法在音视频修复任务上性能接近现有基准。
原文摘要 · Abstract (English)
Video and audio inpainting for mixed audio-visual content has become a crucial task in multimedia editing recently. However, precisely removing an object and its corresponding audio from a video without affecting the rest of the scene remains a significant challenge. To address this, we propose VAInpaint, a novel pipeline that first utilizes a segmentation model to generate masks and guide a video inpainting model in removing objects. At the same time, an LLM then analyzes the scene globally, while a region-specific model provides localized descriptions. Both the overall and regional descriptions will be inputted into an LLM, which will refine the content and turn it into text queries for our text-driven audio separation model. Our audio separation model is fine-tuned on a customized dataset comprising segmented MUSIC instrument images and VGGSound backgrounds to enhance its generalization performance. Experiments show that our method achieves performance comparable to current benchmarks in both audio and video inpainting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。