用自然语言生成空间时间规则，让小模型学会精准理解视频中的物体运动与交互。
Spatio-Temporal Grounding of Large Language Models from Perception Streams
- 将自然语言查询转为可验证的空间时间表达式（SpRE），注入大模型进行监督训练。
- 在2.7万条数据上训练30亿参数模型，帧级准确率从48.5%提升至87.5%，媲美GPT-4。
- 适合需要轻量级视频理解能力的场景，如机器人、智能监控等应用。
具身人工智能代理需对三维空间中物体的运动与交互进行时序推理，但现有小型前沿大语言模型仍难以处理精细的空间关系、度量距离和时间顺序。本文提出通用框架FESTS，通过将自然语言查询编译为结合正则表达式语法与S4u空间逻辑的时空表达式（SpRE），并引入全称与存在量词扩展，实现可验证的时空监督。该流程将每个SpRE匹配到结构化视频日志，输出（查询, 帧, 匹配结果, 解释）三元组，无需人工标注即可生成无限训练数据。在27,000个此类样本上训练30亿参数模型，帧级F1得分从48.5%提升至87.5%，达到GPT-4.1在复杂时空推理任务上的表现，且规模小两个数量级，从而实现视频大模型的时空智能。
原文摘要 · Abstract (English)
Embodied-AI agents must reason about how objects move and interact in 3-D space over time, yet existing smaller frontier Large Language Models (LLMs) still mis-handle fine-grained spatial relations, metric distances, and temporal orderings. We introduce the general framework Formally Explainable Spatio-Temporal Scenes (FESTS) that injects verifiable spatio-temporal supervision into an LLM by compiling natural-language queries into Spatial Regular Expression (SpRE) -- a language combining regular expression syntax with S4u spatial logic and extended here with universal and existential quantification. The pipeline matches each SpRE against any structured video log and exports aligned (query, frames, match, explanation) tuples, enabling unlimited training data without manual labels. Training a 3-billion-parameter model on 27k such tuples boosts frame-level F1 from 48.5% to 87.5%, matching GPT-4.1 on complex spatio-temporal reasoning while remaining two orders of magnitude smaller, and, hence, enabling spatio-temporal intelligence for Video LLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。