提出T-MASK方法,提升驾驶监控中跨视角的模型泛化能力。
T-MASK: Temporal Masking for Probing Foundation Models across Camera Views in Driver Monitoring
- 通过时序标记掩码,聚焦视频动态区域进行轻量级探针适配。
- 在Drive&Act数据集上跨视角准确率提升1.23%,较参数高效微调高8.0%。
- 特别增强对次要驾驶行为的识别,低数据场景下效果显著。
相机视角变化是驾驶监控中的常见挑战。尽管深度学习与预训练基础模型通过轻量级末端层适配(即“探针”)展现出强大泛化潜力,但其在未见视角下的鲁棒性仍待深入研究。本文通过单一训练视角适配图像基础模型,并直接评估其在未见视角下的表现,无需进一步微调。对比了线性探针、先进探针策略与两种基础模型(DINOv2和CLIP)在参数高效微调(PEFT)和全微调下的性能。基于此,提出T-MASK——一种利用时序标记掩码、侧重动态视频区域的新图像到视频探针方法。在公开数据集Drive&Act上,T-MASK相较强探针基线提升跨视角top-1准确率1.23%,相比PEFT方法提升8.0%,且不增加任何参数。尤其在低频次次要行为识别上,训练视角下提升5.42%,跨视角下提升1.36%。结果表明,采用轻量探针如T-MASK在细粒度驾驶行为分析中具有巨大潜力,特别是在跨视角与低数据条件下。该工作强调了时序标记选择在构建鲁棒驾驶监控系统中的关键作用。代码与模型将开源于https://github.com/th-nesh/T-MASK。
原文摘要 · Abstract (English)
Changes of camera perspective are a common obstacle in driver monitoring. While deep learning and pretrained foundation models show strong potential for improved generalization via lightweight adaptation of the final layers ('probing'), their robustness to unseen viewpoints remains underexplored. We study this challenge by adapting image foundation models to driver monitoring using a single training view, and evaluating them directly on unseen perspectives without further adaptation. We benchmark simple linear probes, advanced probing strategies, and compare two foundation models (DINOv2 and CLIP) against parameter-efficient fine-tuning (PEFT) and full fine-tuning. Building on these insights, we introduce T-MASK -- a new image-to-video probing method that leverages temporal token masking and emphasizes more dynamic video regions. Benchmarked on the public Drive&Act dataset, T-MASK improves cross-view top-1 accuracy by $+1.23\%$ over strong probing baselines and $+8.0\%$ over PEFT methods, without adding any parameters. It proves particularly effective for underrepresented secondary activities, boosting recognition by $+5.42\%$ under the trained view and $+1.36\%$ under cross-view settings. This work provides encouraging evidence that adapting foundation models with lightweight probing methods like T-MASK has strong potential in fine-grained driver observation, especially in cross-view and low-data settings. These results highlight the importance of temporal token selection when leveraging foundation models to build robust driver monitoring systems. Code and models will be made available at https://github.com/th-nesh/T-MASK to support ongoing research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。