针对移动端测试时自适应,构建了四类真实场景的评估基准。
BoTTA: Benchmarking on-device Test Time Adaptation
- 设计四个资源受限场景评估测试时自适应方法
- 发现多数算法在小样本和新类别上表现差
- 推荐周期性更新而非持续调整,适合手机设备
深度学习模型性能高度依赖运行时测试样本,训练数据分布偏移会显著降低准确率。测试时自适应(TTA)通过推理期间无标签数据调整模型,无需访问原始训练集。尽管已有研究从算法复杂度、数据分布偏移、模型结构等角度探索TTA,但移动与边缘设备的实际约束仍被忽视。本文提出BoTTA基准,用于评估受限设备上的TTA方法。评估聚焦四个现实挑战:(i) 测试样本有限,(ii) 接触类别少,(iii) 分布偏移多样,(iv) 样本内多重偏移。使用真实设备测试,报告系统级指标。不同于以往连续自适应,我们提倡周期性更新。实验表明:多数先进TTA算法在小样本下失效,无法泛化到未见类别,且对偏移多样性敏感。例如,SHOT在512次适配样本下提升精度2.25倍,但在树莓派上内存占用达基线的1.08倍。该基准为资源受限部署提供实用指导。
原文摘要 · Abstract (English)
The performance of deep learning models depends heavily on test samples at runtime, and shifts from the training data distribution can significantly reduce accuracy. Test-time adaptation (TTA) addresses this by adapting models during inference without requiring labeled test data or access to the original training set. While research has explored TTA from various perspectives like algorithmic complexity, data and class distribution shifts, model architectures, and offline versus continuous learning, constraints specific to mobile and edge devices remain underexplored. We propose BoTTA, a benchmark designed to evaluate TTA methods under practical constraints on mobile and edge devices. Our evaluation targets four key challenges caused by limited resources and usage conditions: (i) limited test samples, (ii) limited exposure to categories, (iii) diverse distribution shifts, and (iv) overlapping shifts within a sample. We assess state-of-the-art TTA methods under these scenarios using benchmark datasets and report system-level metrics on a real testbed. Furthermore, unlike prior work, we align with on-device requirements by advocating periodic adaptation instead of continuous inference-time adaptation. Experiments reveal key insights: many recent TTA algorithms struggle with small datasets, fail to generalize to unseen categories, and depend on the diversity and complexity of distribution shifts. BoTTA also reports device-specific resource use. For example, while SHOT improves accuracy by $2.25\times$ with $512$ adaptation samples, it uses $1.08\times$ peak memory on Raspberry Pi versus the base model. BoTTA offers actionable guidance for TTA in real-world, resource-constrained deployments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。