用分步任务链和评分对齐优化,提升视频模型细粒度理解能力
VidChain: Chain-of-Tasks with Metric-based Direct Preference Optimization for Dense Video Captioning
- 将复杂视频描述任务拆解为连续子任务,激发模型推理潜力
- 在两个数据集上显著超越现有方法,尤其在时间定位精度上提升明显
- 适合需要精准事件识别与定位的视频分析场景
尽管视频大模型在多种任务中取得进展,但在细粒度时间理解方面仍表现不足,如密集视频字幕(Dense Video Captioning, DVC)任务。DVC需描述视频中所有事件并准确定位其时间位置,融合了视频分割、字幕生成和时间定位等多个细粒度任务。以往视频大模型采用单步处理,未能有效利用其推理能力;且训练目标未充分反映评估指标,缺乏与任务目标直接对齐的监督。为此,本文提出VidChain框架,包含任务链(CoTasks)与基于度量的直接偏好优化(M-DPO)。CoTasks将复杂任务分解为一系列子任务,使视频大模型能更有效地发挥推理能力;M-DPO通过与评估指标对齐,为每个子任务提供精细监督。在两种不同视频大模型上应用VidChain,均显著提升其细粒度视频理解能力,在两个DVC基准测试及时间视频定位任务上优于先前方法。代码已开源。
原文摘要 · Abstract (English)
Despite the advancements of Video Large Language Models (VideoLLMs) in various tasks, they struggle with fine-grained temporal understanding, such as Dense Video Captioning (DVC). DVC is a complicated task of describing all events within a video while also temporally localizing them, which integrates multiple fine-grained tasks, including video segmentation, video captioning, and temporal video grounding. Previous VideoLLMs attempt to solve DVC in a single step, failing to utilize their reasoning capability. Moreover, previous training objectives for VideoLLMs do not fully reflect the evaluation metrics, therefore not providing supervision directly aligned to target tasks. To address such a problem, we propose a novel framework named VidChain comprised of Chain-of-Tasks (CoTasks) and Metric-based Direct Preference Optimization (M-DPO). CoTasks decompose a complex task into a sequence of sub-tasks, allowing VideoLLMs to leverage their reasoning capabilities more effectively. M-DPO aligns a VideoLLM with evaluation metrics, providing fine-grained supervision to each task that is well-aligned with metrics. Applied to two different VideoLLMs, VidChain consistently improves their fine-grained video understanding, thereby outperforming previous VideoLLMs on two different DVC benchmarks and also on the temporal video grounding task. Code is available at \url{https://github.com/mlvlab/VidChain}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。