用人类与AI协作提升视频字幕精度，让模型更懂专业影像细节。
Building a Precise Video Language with Human-AI Oversight
- 构建结构化描述规范，融合影视专业人士定义的数百个视觉基本单元。
- 通过专家评审模型生成的初稿，实现高质量字幕，准确率超闭源模型Gemini-3.1-Pro。
- 适用于电影、广告等专业视频重标注，支持400字长提示精细控制镜头语言。
视频语言模型（VLMs）通过自然语言理解动态视觉世界。本文提出一套开放数据集、评测基准与可扩展监督方案，实现高精度视频字幕生成。首先，定义包含主体、场景、运动、空间及摄像机动态的结构化描述规范，基于数百个由专业影视创作者共同设计的视觉基本单元。为获取高质量字幕，引入CHAI（基于批评的人机协同监督）框架：训练专家对模型生成的初稿进行评审并修订为优化后的终稿。该分工机制提升标注效率与准确性，同时提供丰富监督信号，用于改进开源模型Qwen3-VL在字幕生成、奖励建模和批评生成上的表现，分别通过SFT、DPO和推理时缩放实现。消融实验表明，批评的质量（精确性、召回率、建设性）直接影响下游性能。仅需少量专家监督，最终模型超越闭源模型Gemini-3.1-Pro。进一步将该方法应用于大规模专业视频（如电影、广告、游戏）重标注，并微调视频生成模型Wan，使其能精准响应长达400字的详细提示，实现对运镜、角度、镜头、焦距、视角、构图等影像语言的精细控制。结果表明，精准规范与人机协同监督是实现专业级视频理解与生成的关键。数据与代码见项目页：https://linzhiqiu.github.io/papers/chai/
原文摘要 · Abstract (English)
Video-language models (VLMs) learn to reason about the dynamic visual world through natural language. We introduce a suite of open datasets, benchmarks, and recipes for scalable oversight that enable precise video captioning. First, we define a structured specification for describing subjects, scenes, motion, spatial, and camera dynamics, grounded by hundreds of carefully defined visual primitives developed with professional video creators such as filmmakers. Next, to curate high-quality captions, we introduce CHAI (Critique-based Human-AI Oversight), a framework where trained experts critique and revise model-generated pre-captions into improved post-captions. This division of labor improves annotation accuracy and efficiency by offloading text generation to models, allowing humans to better focus on verification. Additionally, these critiques and preferences between pre- and post-captions provide rich supervision for improving open-source models (Qwen3-VL) on caption generation, reward modeling, and critique generation through SFT, DPO, and inference-time scaling. Our ablations show that critique quality in precision, recall, and constructiveness, ensured by our oversight framework, directly governs downstream performance. With modest expert supervision, the resulting model outperforms closed-source models such as Gemini-3.1-Pro. Finally, we apply our approach to re-caption large-scale professional videos (e.g., films, commercials, games) and fine-tune video generation models such as Wan to better follow detailed prompts of up to 400 words, achieving finer control over cinematography including camera motion, angle, lens, focus, point of view, and framing. Our results show that precise specification and human-AI oversight are key to professional-level video understanding and generation. Data and code are available on our project page: https://linzhiqiu.github.io/papers/chai/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。