用轨迹信息提升视频检索精度,让模型更好区分驾驶中的细微动作。
TraVEL: Trajectory-Guided Video Embedding Learning for Driving-Video Retrieval

- 引入轨迹监督优化视频嵌入,让模型关注运动特征而非静态场景。
- 在2B和8B模型上,纵向与横向检索准确率分别提升9.8和4.7点。
- 无需复杂规则或额外感知模块,适合自动驾驶数据检索场景。
从大规模驾驶日志中高效检索相关片段,对数据整理、模型开发和安全分析至关重要。传统基于规则的系统需专家定义规则和多阶段感知,而通用多模态嵌入模型常依赖静态场景线索,难以区分以运动为中心的事件(如左转/右转、加速/减速)。本文提出TraVEL(轨迹引导的视频嵌入学习),通过在nuReasoning数据集上使用配对片段与推理轨迹微调Qwen3-VL-Embedding,并采用组相对策略优化,以自车轨迹相似性作为奖励信号。轨迹仅用于训练监督,检索仍基于单向量视频嵌入,不依赖自车位姿、专家规则或辅助输出。我们进一步构建了基于nuReasoning的驾驶视频检索基准。实验表明,相较SFT,TraVEL在2B模型上纵向与横向mAP分别提升9.8和4.7点,在8B模型上分别提升7.2和1.5点,实现了物理可解释监督与高效嵌入搜索的结合。
原文摘要 · Abstract (English)
Efficiently retrieving relevant clips from large-scale driving logs is essential for data curation, model development, and safety analysis. Structured and rule-based retrieval systems can explicitly target driving events, but typically require expert-defined rules, auxiliary data, and multi-stage perception pipelines. Multimodal embedding models offer a simpler and more efficient alternative by representing each video with a single searchable vector. However, general-purpose models often rely on shortcuts from static scene context and struggle to distinguish motion-centric events, such as turning left versus right or accelerating versus decelerating. In this work, we study how to adapt a general-purpose multimodal embedding model to driving-video retrieval. We first fine-tune Qwen3-VL-Embedding on paired clips and reasoning traces from nuReasoning using an InfoNCE objective. While this stage substantially improves overall retrieval, caption supervision alone remains insufficient for fine-grained motion understanding. We therefore introduce TraVEL (Trajectory-Guided Video Embedding Learning), a motion-aware fine-tuning framework that uses ego-trajectory similarity as a reward within Group Relative Policy Optimization. Trajectories serve only as privileged training supervision; retrieval still operates on single-vector video embeddings without ego poses, expert rules, or auxiliary perception outputs. We further construct a driving-video retrieval benchmark from nuReasoning. Experiments show that TraVEL improves motion-centric retrieval across model scales: relative to SFT, it raises longitudinal and lateral mAP by 9.8 and 4.7 points at 2B, with corresponding gains of 7.2 and 1.5 points at 8B. TraVEL thus combines physically grounded supervision with efficient embedding-based search.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。