发现测试时自适应模型的黑盒攻击漏洞，仅靠重复错误预测即可导致性能下降。
R.I.P.: A Simple Black-box Attack on Continual Test-time Adaptation
- 利用测试数据中重复错误预测的机制实施攻击
- 无需模型参数或样本修改，即可显著降低连续测试时自适应模型性能
- 首个无需白盒信息的黑盒攻击方法，适用于评估模型鲁棒性
测试时自适应（TTA）通过在无标签测试数据上进行自监督学习，动态调整模型参数，以应对持续的领域偏移问题。然而，该机制也带来了潜在的安全风险。本文提出一种简单的理论连续TTA模型，揭示了测试数据采样过程中存在一个关键风险：重复错误预测（RIP）。RIP攻击者或普通用户无意间重复使用错误预测结果，即可导致模型性能持续下降。该风险无需了解模型参数或修改测试样本，因此成为首个真正意义上的黑盒攻击方法，区别于以往依赖白盒信息的攻击。我们对最新连续TTA方法在面对RIP攻击时的表现进行了广泛基准测试，分析其脆弱性并为未来提升系统抗攻击能力提供方向。
原文摘要 · Abstract (English)
Test-time adaptation (TTA) has emerged as a promising solution to tackle the continual domain shift in machine learning by allowing model parameters to change at test time, via self-supervised learning on unlabeled testing data. At the same time, it unfortunately opens the door to unforeseen vulnerabilities for degradation over time. Through a simple theoretical continual TTA model, we successfully identify a risk in the sampling process of testing data that could easily degrade the performance of a continual TTA model. We name this risk as Reusing of Incorrect Prediction (RIP) that TTA attackers can employ or as a result of the unintended query from general TTA users. The risk posed by RIP is also highly realistic, as it does not require prior knowledge of model parameters or modification of testing samples. This simple requirement makes RIP as the first black-box TTA attack algorithm that stands out from existing white-box attempts. We extensively benchmark the performance of the most recent continual TTA approaches when facing the RIP attack, providing insights on its success, and laying out potential roadmaps that could enhance the resilience of future continual TTA systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。