用关键点图增强模型抗干扰能力,提升视觉模型鲁棒性。
SIFT-Graph: Benchmarking Multimodal Defense Against Image Adversarial Attacks With Robust Feature Graph
- 融合手工与学习特征,构建抗扰动的结构化特征图
- 在白盒攻击下显著提升模型鲁棒性,干净准确率损失小
- 适用于视觉变换器和卷积网络,适配主流视觉模型
对抗攻击暴露了现代深度视觉模型的根本缺陷:依赖对微小扰动高度敏感的密集像素表示。传统防御策略通常在脆弱的像素域内操作,缺乏整合固有鲁棒视觉特征的机制。本文提出SIFT-Graph,一种多模态防御框架,通过结合手工提取(SIFT)与学习得到的特征,从原始图像中聚合具有结构性意义的特征。具体地,将尺度不变特征变换关键点与图注意力网络结合,捕捉尺度与旋转不变的局部结构,这些结构对扰动具有韧性。随后将此类鲁棒特征嵌入传统视觉模型(如视觉变压器和卷积神经网络),形成统一的结构感知、抗扰动模型。初步结果表明,该方法能有效提升模型对基于梯度的白盒对抗攻击的鲁棒性,同时仅带来轻微的干净准确率下降。
原文摘要 · Abstract (English)
Adversarial attacks expose a fundamental vulnerability in modern deep vision models by exploiting their dependence on dense, pixel-level representations that are highly sensitive to imperceptible perturbations. Traditional defense strategies typically operate within this fragile pixel domain, lacking mechanisms to incorporate inherently robust visual features. In this work, we introduce SIFT-Graph, a multimodal defense framework that enhances the robustness of traditional vision models by aggregating structurally meaningful features extracted from raw images using both handcrafted and learned modalities. Specifically, we integrate Scale-Invariant Feature Transform keypoints with a Graph Attention Network to capture scale and rotation invariant local structures that are resilient to perturbations. These robust feature embeddings are then fused with traditional vision model, such as Vision Transformer and Convolutional Neural Network, to form a unified, structure-aware and perturbation defensive model. Preliminary results demonstrate that our method effectively improves the visual model robustness against gradient-based white box adversarial attacks, while incurring only a marginal drop in clean accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。