提出无需反向传播的零阶提示优化方法,提升模型在低资源设备上的实时适应能力。
FOZO: Forward-Only Zeroth-Order Prompt Optimization for Test-Time Adaptation
- 采用零阶优化策略,仅通过前向传播调整提示参数,避免修改模型权重。
- 在ImageNet-C上达59.52%准确率,优于主流基于梯度的方法和现有前向-only方法。
- 适用于量化模型(INT8),适合部署在边缘设备等资源受限场景。
测试时适应(TTA)对使深度学习模型应对真实世界数据分布偏移至关重要。然而,现有方法存在显著局限:基于反向传播的方法因计算与内存开销大,且在适应过程中修改模型权重,不适用于低端设备;而传统无反向传播技术则适应能力受限。本文提出前向仅零阶优化(FOZO),一种新颖且实用的无反向传播TTA范式。FOZO利用内存高效的零阶提示优化,目标函数同时优化中间特征统计量与预测熵。为确保在分布外数据流上的高效稳定适应,引入动态衰减的扰动尺度以进行零阶梯度估计,并在TTA数据流假设下理论上证明其收敛性。在ImageNet-C、ImageNet-R和ImageNet-Sketch上的连续适应实验表明,FOZO表现优异,在ImageNet-C(5K,level 5)上达到59.52%的Top-1准确率,超越主要基于梯度的方法及当前最优前向仅方法(58.13%)。此外,FOZO在量化(INT8)模型上也展现强泛化能力。结果表明,FOZO是资源受限场景下TTA部署的有力解决方案。
原文摘要 · Abstract (English)
Test-Time Adaptation (TTA) is essential for enabling deep learning models to handle real-world data distribution shifts. However, current approaches face significant limitations: backpropagation-based methods are not suitable for low-end deployment devices, due to their high computation and memory requirements, as well as their tendency to modify model weights during adaptation; while traditional backpropagation-free techniques exhibit constrained adaptation capabilities. In this work, we propose Forward-Only Zeroth-Order Optimization (FOZO), a novel and practical backpropagation-free paradigm for TTA. FOZO leverages a memory-efficient zeroth-order prompt optimization, which is led by objectives optimizing both intermediate feature statistics and prediction entropy. To ensure efficient and stable adaptation over the out-of-distribution data stream, we introduce a dynamically decaying perturbation scale during zeroth-order gradient estimation and theoretically prove its convergence under the TTA data stream assumption. Extensive continual adaptation experiments on ImageNet-C, ImageNet-R, and ImageNet-Sketch demonstrate FOZO's superior performance, achieving 59.52% Top-1 accuracy on ImageNet-C (5K, level 5) and outperforming main gradient-based methods and SOTA forward-only FOA (58.13%). Furthermore, FOZO exhibits strong generalization on quantized (INT8) models. These findings demonstrate that FOZO is a highly competitive solution for TTA deployment in resource-limited scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。