Neptune构建了可扩展的长视频理解基准，挑战现有模型在多模态推理上的极限。
Neptune: The Long Orbit to Benchmarking Long Video Understanding
- 用大模型自动生成密集时序标注与难题问答对，解决数据成本高问题。
- 15分钟长视频测试显示主流模型在时间顺序、计数等任务上表现差。
- 提供开源评分模型GEM，适合研究长视频理解与多模态推理的团队使用。
我们提出Neptune，一个面向长视频理解的基准，要求模型在长时间跨度和跨模态条件下进行推理。现有视频数据集和模型多聚焦于短片段（10-30秒），部分长视频数据集虽存在，但常可通过单帧图像模型解决，且人工标注成本高。为缓解上述问题，我们设计了一套可扩展的数据集生成流程，利用视觉语言模型（VLMs）和大语言模型（LLMs）自动生成密集、时间对齐的视频字幕，以及针对视频片段（最长可达15分钟）的高难度问答干扰集。Neptune涵盖广泛的长视频推理能力，其中子集强调多模态推理。由于现有开放问答评估指标多为规则基础或依赖专有模型，我们引入新的开源模型基评估指标GEM，用于评分开放回答。基准评估显示，当前多数开源长视频模型在Neptune上表现不佳，尤其在时间排序、计数和状态变化等任务上。通过Neptune，我们旨在推动更先进长视频理解模型的发展。数据集已开源：https://github.com/google-deepmind/neptune
原文摘要 · Abstract (English)
We introduce Neptune, a benchmark for long video understanding that requires reasoning over long time horizons and across different modalities. Many existing video datasets and models are focused on short clips (10s-30s). While some long video datasets do exist, they can often be solved by powerful image models applied per frame (and often to very few frames) in a video, and are usually manually annotated at high cost. In order to mitigate both these problems, we propose a scalable dataset creation pipeline which leverages large models (VLMs and LLMs), to automatically generate dense, time-aligned video captions, as well as tough question answer decoy sets for video segments (up to 15 minutes in length). Our dataset Neptune covers a broad range of long video reasoning abilities and consists of a subset that emphasizes multimodal reasoning. Since existing metrics for open-ended question answering are either rule-based or may rely on proprietary models, we provide a new open source model-based metric GEM to score open-ended responses on Neptune. Benchmark evaluations reveal that most current open-source long video models perform poorly on Neptune, particularly on questions testing temporal ordering, counting and state changes. Through Neptune, we aim to spur the development of more advanced models capable of understanding long videos. The dataset is available at https://github.com/google-deepmind/neptune
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。