提出视频轨迹数据集,让模型在真实时序场景下更好适应。
ADVMEM: Adversarial Memory Initialization for Realistic Test-Time Adaptation via Tracklet-Based Benchmarking
- 用目标追踪数据构建带时间依赖的视频轨迹数据集
- 发现现有方法在时序依赖场景下性能显著下降
- 设计对抗性记忆初始化,提升多类模型在该场景表现
我们提出一种新型基于轨迹的测试时自适应(TTA)基准数据集,旨在模拟现实世界中手持摄像头、自动驾驶等场景的复杂挑战。当前的TTA基准主要关注模型部署时的分布偏移问题,以及对机器学习中独立同分布(i.i.d.)假设的违背,但未能真实反映连续视频帧中存在的自然时序依赖关系——例如同一物体在视频流中持续出现。为此,我们构建了名为「固有时序依赖」(ITD)的数据集,通过从目标追踪数据集中提取物体中心图像序列(即轨迹)来确保实例天然具备时序关联。利用ITD,我们对现有TTA方法进行了全面实验分析,揭示了其在面对时序依赖时的局限性。基于这些发现,我们进一步提出一种新的对抗性记忆初始化策略,显著提升了多种记忆型TTA方法在该挑战性基准上的表现。
原文摘要 · Abstract (English)
We introduce a novel tracklet-based dataset for benchmarking test-time adaptation (TTA) methods. The aim of this dataset is to mimic the intricate challenges encountered in real-world environments such as images captured by hand-held cameras, self-driving cars, etc. The current benchmarks for TTA focus on how models face distribution shifts, when deployed, and on violations to the customary independent-and-identically-distributed (i.i.d.) assumption in machine learning. Yet, these benchmarks fail to faithfully represent realistic scenarios that naturally display temporal dependencies, such as how consecutive frames from a video stream likely show the same object across time. We address this shortcoming of current datasets by proposing a novel TTA benchmark we call the "Inherent Temporal Dependencies" (ITD) dataset. We ensure the instances in ITD naturally embody temporal dependencies by collecting them from tracklets-sequences of object-centric images we compile from the bounding boxes of an object-tracking dataset. We use ITD to conduct a thorough experimental analysis of current TTA methods, and shed light on the limitations of these methods when faced with the challenges of temporal dependencies. Moreover, we build upon these insights and propose a novel adversarial memory initialization strategy to improve memory-based TTA methods. We find this strategy substantially boosts the performance of various methods on our challenging benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。