用分层时空对齐优化视频模型，减少幻觉和理解偏差。
VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Models
- 分三层次对齐视频内容、时间语义和空间对象与语言描述
- 在7.2K数据集上提升视频问答与字幕生成效果，显著降低幻觉率
- 适合需要精准视频理解的AI研究者和开发者使用
基于大语言模型的大视频模型在视频理解方面展现出潜力，但常与人类直觉不符并出现视频幻觉问题。为解决这些问题，我们提出VistaDPO，一种面向大视频模型的分层时空直接偏好优化框架。该框架在三个层次实现文本-视频偏好对齐：实例级（整体视频内容与回答对齐）、时序级（视频时序语义与事件描述对齐）、感知级（空间物体与语言标记对齐）。由于缺乏细粒度视频-语言偏好对齐数据集，我们构建了VistaDPO-7k，包含7200个问答对，标注了优选与次优响应，并附带时间戳、关键帧和边界框等时空定位信息。在视频幻觉、视频问答和字幕生成等多个基准任务上的实验表明，VistaDPO能显著提升现有大视频模型性能，有效缓解视频-语言错配与幻觉问题。代码与数据已开源。
原文摘要 · Abstract (English)
Large Video Models (LVMs) built upon Large Language Models (LLMs) have shown promise in video understanding but often suffer from misalignment with human intuition and video hallucination issues. To address these challenges, we introduce VistaDPO, a novel framework for Video Hierarchical Spatial-Temporal Direct Preference Optimization. VistaDPO enhances text-video preference alignment across three hierarchical levels: i) Instance Level, aligning overall video content with responses; ii) Temporal Level, aligning video temporal semantics with event descriptions; and iii) Perceptive Level, aligning spatial objects with language tokens. Given the lack of datasets for fine-grained video-language preference alignment, we construct VistaDPO-7k, a dataset of 7.2K QA pairs annotated with chosen and rejected responses, along with spatial-temporal grounding information such as timestamps, keyframes, and bounding boxes. Extensive experiments on benchmarks such as Video Hallucination, Video QA, and Captioning performance tasks demonstrate that VistaDPO significantly improves the performance of existing LVMs, effectively mitigating video-language misalignment and hallucination. The code and data are available at https://github.com/HaroldChen19/VistaDPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。