通过自举机制提升测试时适应性能,支持多任务场景。
Self-Bootstrapping for Versatile Test-Time Adaptation
- 在傅里叶域随机掩码低频幅度作为增强手段
- 在图像分类、分割与3D检测上均超越现有方法
- 适用于Transformer与CNN,可即插即用
本文提出一种通用的测试时适应(TTA)目标,适用于图像级、物体级和像素级的分类与回归任务。通过自举策略,优化测试图像与其退化视图之间的预测一致性。核心挑战在于设计有效的退化方式:既要保留几何信息(如物体尺寸与位置),又需提供充足的学习信号。为此,我们分析了常见分布偏移对图像在傅里叶域各频段信息功率的影响,发现低频成分携带高能量,掩码低频能提供更多学习信号,而掩码高频则无效。基于此,我们在傅里叶域随机掩码图像的低频振幅作为增强;同时引入噪声注入以补偿高频信息缺失,提升其信息功率。实验表明,无论独立使用或作为模块集成,该方法在分类、分割和3D单目检测任务中,对Transformer与CNN模型均取得更优性能。
原文摘要 · Abstract (English)
In this paper, we seek to develop a versatile test-time adaptation (TTA) objective for a variety of tasks - classification and regression across image-, object-, and pixel-level predictions. We achieve this through a self-bootstrapping scheme that optimizes prediction consistency between the test image (as target) and its deteriorated view. The key challenge lies in devising effective augmentations/deteriorations that: i) preserve the image's geometric information, e.g., object sizes and locations, which is crucial for TTA on object/pixel-level tasks, and ii) provide sufficient learning signals for TTA. To this end, we analyze how common distribution shifts affect the image's information power across spatial frequencies in the Fourier domain, and reveal that low-frequency components carry high power and masking these components supplies more learning signals, while masking high-frequency components can not. In light of this, we randomly mask the low-frequency amplitude of an image in its Fourier domain for augmentation. Meanwhile, we also augment the image with noise injection to compensate for missing learning signals at high frequencies, by enhancing the information power there. Experiments show that, either independently or as a plug-and-play module, our method achieves superior results across classification, segmentation, and 3D monocular detection tasks with both transformer and CNN models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。