用智能代理自动标注视频事件,让模型学会推理时间、地点、原因和后果。
MAVEN: A Multi-stage Agentic Annotation Pipeline for Video Reasoning Tasks

- 分阶段智能代理生成带思维链的多尺度时空事件描述,统一驱动标注流程。
- 在5300段交通视频上标注后,模型准确率比零样本高38.8点,超越Gemini系列。
- 支持自动适应新场景,无需人工重写提示,适合需要快速扩展数据的应用。
训练视觉语言模型进行视频事件推理需要高质量的结构化标注,涵盖事件发生的时间、地点、原因及后果,但人工标注无法满足规模需求。我们提出MAVEN(多阶段智能体视频事件标注),一个将原始视频转化为多任务训练数据的多阶段智能体流水线,包含链式思维(CoT)推理轨迹,围绕指定事件焦点组织。其核心是融合三个互补描述层级生成多尺度时空事件描述(MSTED),作为下游问答生成的唯一输入。关键优势在于支持代理驱动的领域自适应:给定新数据集和目标问题示例,代理可自顶向下重设计所有提示,无需人工干预。通过层次化精炼循环,对标注错误按分类法归因至源头阶段,并针对性修改提示或调整流水线结构,持续提升数据质量。我们在超过5300段交通视频上应用MAVEN,微调Cosmos-Reason2-8B模型。在私有CCTV评估集上,微调结果超越Gemini 2.5 Pro和3.1 Flash,MCQ准确率较零样本提升38.8点;在AccidentBench上,仅使用CCTV数据训练使模型提升10.7点,匹配Gemini 2.5 Pro,加入代理适配的行车记录仪标注后差距缩小至接近Gemini 3.1 Flash,强化学习后训后性能超越两者。仓库监控与公共安全视频的定性结果表明,该智能体流程可轻松迁移到新领域。
原文摘要 · Abstract (English)
Training Vision Language Models (VLMs) for video event reasoning requires high-quality structured annotations capturing not only what happened, but when, where, why, and with what consequence, at a scale manual labelling cannot support. We present MAVEN (Multi-stage Agentic Video Event aNnotation), a multi-stage agentic pipeline that turns raw videos into multi-task training data with Chain-of-Thought (CoT) reasoning traces, organized around a designated Event of Focus. At its core, MAVEN synthesizes a Multi-Scale Spatio-Temporal Event Description (MSTED) from three complementary caption levels; this explicit intermediate serves as the sole input to downstream Q&A generation across multiple task formats. Crucially, MAVEN supports agent-driven domain adaptation: given a new video dataset and target question examples, the agent redesigns all prompts top-down without manual re-engineering. A hierarchical refinement loop further classifies annotation errors against a taxonomy, traces root causes to the originating pipeline stage, and applies targeted edits that rewrite prompts or modify the pipeline structure itself, iteratively improving data quality. We apply MAVEN to label over 5,300 traffic videos and fine-tune Cosmos-Reason2-8B on the resulting data. On a private CCTV evaluation set, fine-tuning surpasses both Gemini 2.5 Pro and 3.1 Flash, including a $+38.8$-point gain in MCQ accuracy over zero-shot. On AccidentBench, CCTV-only training lifts Cosmos-Reason2 by $+10.7$ MCQ points and matches Gemini 2.5 Pro despite seeing no dashcam videos; adding agent-adapted dashcam annotations narrows the gap to Gemini 3.1 Flash, and RL post-training pushes overall performance past both Gemini baselines. Qualitative results on warehouse surveillance and public safety videos further show the agentic workflow readily adapts the pipeline to new domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。