智能协调多个视频模型,自动完成生成与编辑任务。
SPAgent: Adaptive Task Decomposition and Model Selection for General Video Generation and Editing
- 分三步自动分解任务并选择最优模型
- 可自主评估新模型并加入工具库
- 适合需要快速生成/编辑视频的用户
尽管开源视频生成与编辑模型已取得显著进展,但单个模型通常仅适用于特定任务,难以满足用户多样化需求。有效协调这些模型可释放广泛的视频生成与编辑能力。然而,手动协调复杂且耗时,需用户深入理解任务要求及各模型性能、适用性与局限性,提高了使用门槛。为此,我们提出由语义规划代理(SPAgent)驱动的新系统。SPAgent整合了前沿开源图像与视频生成及编辑模型作为工具,并在人工标注数据集上微调后,通过三步框架——解耦意图识别、原则引导路径规划、能力导向执行模型选择——实现自动化模型协调。此外,我们增强了SPAgent的视频质量评估能力,使其能自主评估并无需人工干预即把新模型纳入工具库。实验表明,SPAgent能有效协调模型生成或编辑视频,展现出在多种视频任务中的广泛适应性与灵活性。
原文摘要 · Abstract (English)
While open-source video generation and editing models have made significant progress, individual models are typically limited to specific tasks, failing to meet the diverse needs of users. Effectively coordinating these models can unlock a wide range of video generation and editing capabilities. However, manual coordination is complex and time-consuming, requiring users to deeply understand task requirements and possess comprehensive knowledge of each model's performance, applicability, and limitations, thereby increasing the barrier to entry. To address these challenges, we propose a novel video generation and editing system powered by our Semantic Planning Agent (SPAgent). SPAgent bridges the gap between diverse user intents and the effective utilization of existing generative models, enhancing the adaptability, efficiency, and overall quality of video generation and editing. Specifically, the SPAgent assembles a tool library integrating state-of-the-art open-source image and video generation and editing models as tools. After fine-tuning on our manually annotated dataset, SPAgent can automatically coordinate the tools for video generation and editing, through our novelly designed three-step framework: (1) decoupled intent recognition, (2) principle-guided route planning, and (3) capability-based execution model selection. Additionally, we enhance the SPAgent's video quality evaluation capability, enabling it to autonomously assess and incorporate new video generation and editing models into its tool library without human intervention. Experimental results demonstrate that the SPAgent effectively coordinates models to generate or edit videos, highlighting its versatility and adaptability across various video tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。