arXiv:2602.18884cs.AI2026-02中稿 · ICLR被引 1

用新数据集和强化学习,让小模型学会理解视频时序和操作流程。

TPRU: Advancing Temporal and Procedural Understanding in Large Multimodal Models

  • 构建包含时序重排等三类任务的大规模数据集TPRU
  • 小模型TPRU-7B准确率从50.33%提升至75.70%,超越大模型GPT-4o
  • 适合做轻量级机器人、GUI导航等实时交互场景的开发者

多模态大语言模型,尤其是小型可部署版本,在理解时间序列和操作流程类视觉数据方面存在明显缺陷,制约了其在真实世界具身AI中的应用。这一瓶颈主要源于训练范式缺乏大规模、程序连贯的数据。为此,我们提出TPRU,一个源自机器人操作、GUI导航等多样化具身场景的大规模数据集。TPRU通过时序重排、下一帧预测、前一帧回顾三类互补任务系统性培养模型的时序推理能力。关键特征是引入具有挑战性的负样本,迫使模型从被动观察转向主动跨模态验证。我们采用强化学习微调方法,专注于提升资源高效模型性能。实验表明,该方法取得显著成效:在人工构建的TPRU-Test上,TPRU-7B的准确率从50.33%提升至75.70%,达到当前最优水平,显著优于参数量更大的基线模型(如GPT-4o)。这些能力具备良好泛化性,在现有基准测试中也表现出显著提升。代码已开源于https://github.com/Stephen-gzk/TPRU/。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs), particularly smaller, deployable variants, exhibit a critical deficiency in understanding temporal and procedural visual data, a bottleneck hindering their application in real-world embodied AI. This gap is largely caused by a systemic failure in training paradigms, which lack large-scale, procedurally coherent data. To address this problem, we introduce TPRU, a large-scale dataset sourced from diverse embodied scenarios such as robotic manipulation and GUI navigation. TPRU is systematically designed to cultivate temporal reasoning through three complementary tasks: Temporal Reordering, Next-Frame Prediction, and Previous-Frame Review. A key feature is the inclusion of challenging negative samples, compelling models to transition from passive observation to active, cross-modal validation. We leverage TPRU with a reinforcement learning (RL) fine-tuning methodology, specifically targeting the enhancement of resource-efficient models. Experiments show our approach yields dramatic gains: on our manually curated TPRU-Test, the accuracy of TPRU-7B soars from 50.33\% to 75.70\%, a state-of-the-art result that significantly outperforms vastly larger baselines, including GPT-4o. Crucially, these capabilities generalize effectively, demonstrating substantial improvements on established benchmarks. The codebase is available at https://github.com/Stephen-gzk/TPRU/ .

时序理解具身AI小模型优化数据集

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。