用视频字幕验证生成步骤,让大模型在无标注数据上学会正确排序操作流程。
RECIPE: Procedural Planning via Grounding in Instructional Video

- 用字幕匹配度当奖励,让模型从海量噪声视频中自动生成合理步骤序列。
- 在7个基准上零样本提升最高16点,比有标注训练还强,且不依赖人工标签。
- 适用于图文或纯视频输入,可作为智能助手的规划模块,适合做自动化流程设计。
视觉规划要求模型根据部分视频和目标,生成后续的操作步骤。该任务受限于标注数据:现有数据集小、领域窄,每例仅含单一执行路径,而实际存在多种有效顺序。大规模教学视频语料库虽内容丰富,但基于其噪声语音转写(ASR)伪标签进行监督微调会传播分割与对齐错误,且仍为单路径。我们发现关键不对称性:从噪声视频中提取干净步骤标签困难,但验证生成步骤序列是否在时间上与ASR字幕对齐却廉价且可扩展至百万级视频。为此提出RECIPE框架,将接地质量作为GRPO强化学习的奖励信号,使噪声语料库变为验证器而非标签源。该方法统一适用于两种输入(文本历史+冻结视觉语言模型,或直接处理视频令牌),并兼容标注与弱监督场景。在7个程序性任务上采用基于参考的LLM评判协议,评估6项程序标准。RECIPE-RL在所有规模(0.5B, 3B, 7B)和所有基准上均优于基线,域内宏准确率提升+7至+8点,零样本最高提升+16点。优于有标注及伪标注监督微调(后者反而损害基线),且无需人工标注。作为先验提议阶段,其在Visual Planning for Assistance上各预测时长均超越最强零样本基线;在COIN上保持生成多样性,而传统微调会压缩多样性。
原文摘要 · Abstract (English)
Visual planning asks a model to generate the remaining steps of a procedure in natural language given a partial video context and a goal. Progress on this task is bottlenecked by annotation: clean labeled datasets are small, domain-narrow, and encode a single execution trajectory per example, even though many valid orderings exist. Large-scale instructional video corpora offer orders of magnitude more procedural content, but supervised fine-tuning on pseudo-labels from their noisy ASR narrations propagates segmentation and alignment errors and stays single-trajectory. We identify a key asymmetry: extracting clean step labels from noisy video is hard, but verifying whether a generated step sequence is temporally grounded in ASR transcripts is cheap and scales to millions of videos via precomputed text embeddings. We exploit this asymmetry in RECIPE, which uses grounding quality as a reward for GRPO, turning the noisy corpus into a verifier rather than a label source. The framework applies uniformly to two planner input configurations (Socratic, with a textual history extracted by a frozen VLM, and Video, consuming video tokens directly) and to annotated and weakly supervised regimes. We evaluate on 7 procedural benchmarks using a reference-based LLM-as-judge protocol scoring plans across 6 procedural criteria. RECIPE-RL improves over the base checkpoint at all scales (0.5B, 3B, 7B) and every benchmark, with macro-accuracy gains of +7 to +8 points in-domain and up to +16 points zero-shot. It outperforms supervised fine-tuning on both annotated and pseudo-labeled plans (the latter degrades the base) and remains robust without human annotations. Used as the proposal stage of a prior propose-assess-search planner, it improves over the strongest zero-shot baseline at every horizon on Visual Planning for Assistance, and on COIN it preserves the generation diversity that SFT collapses.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。