让大模型学会追踪物体4维动态，通过对话理解运动轨迹。
LMM-Track4D: Eliciting 4D Dynamic Reasoning in LMMs via Trajectory-Grounded Dialogue
- 用轨迹引导的多轮对话任务，要求模型输出3D运动路径。
- 在23.5k帧、7.5k标注上提升4维推理能力，超越强基线。
- 适合研究视频理解、动态建模与多模态推理的学者。
近期大型多模态模型（LMMs）在图像和视频理解方面日益强大，但在持续的4维时空动态推理上仍存在不足。为研究这一能力差距，我们提出轨迹引导的多轮时空对话任务：模型需在回答时空问题的同时，返回整个短片段或长片段指定段落中的结构化3D目标轨迹，并引入Track4D-Bench基准，包含526个片段级对话样本，覆盖23.5k帧和7.5k个物体标注，用于训练与评估。基于此任务，我们提出LMM-Track4D，结合射线-时间几何编码（RTGE）、用于长时序动态传播的流式状态令牌TRK，以及在遮挡和视角变化下保持稳定的4步3D状态估计的物体槽运动残差锚定（OSK-RA）解码器。在Track4D-Bench上的实验显示，该方法持续优于强基线，表明显式动态状态建模是激发LMMs 4D动态推理的有效设计原则。代码与数据集将公开于https://github.com/mikubaka88/LMM-Track4D。
原文摘要 · Abstract (English)
Recent large multimodal models (LMMs) have become increasingly capable on image and video understanding, yet still struggle to sustain 4D continuous spatiotemporal dynamic reasoning. To study this capability gap, we formulate trajectory-grounded multi-turn spatiotemporal dialogue, a new task in which a model must answer spatiotemporal queries while returning structured 3D target trajectories over an entire short clip or a specified segment of a longer clip, and introduce Track4D-Bench, a benchmark with 526 clip-level dialogue samples spanning 23.5k frames and 7.5k object annotations, for training and evaluation. Building on this task, we propose LMM-Track4D, which combines RTGE (Ray--Time Geometry Encoding), a dedicated streaming state token TRK for long-horizon dynamic propagation, and an Object-Slot Kinematic, Residual-Anchor (OSK-RA) decoder for stable 4-step 3D state estimation under occlusion and viewpoint variation. Experiments on Track4D-Bench show consistent improvements over strong baselines, suggesting that explicit dynamic state modeling is a useful design principle for eliciting 4D dynamic reasoning in LMMs. Our code and dataset will be publicly available at https://github.com/mikubaka88/LMM-Track4D.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。