给Transformer加个插件层，无需训练就能大幅增强抗攻击能力。
ProTransformer: Robustify Transformers via Plug-and-Play Paradigm
- 设计可即插即用的鲁棒注意力机制，直接嵌入现有模型。
- 在文本攻击下，BERT等模型性能提升11.4%至28.3%。
- 适用文本、视觉、图结构任务，尤其适合部署高危场景。
基于Transformer的架构近年来在机器学习多个领域占据主导地位。本文提出一种新型鲁棒注意力机制，旨在提升Transformer架构的抗干扰能力。该方法可作为即插即用模块无缝集成到现有Transformer中，无需额外训练或微调即可显著增强模型鲁棒性。通过大量实验与消融分析，我们证明ProTransformer在多种预测任务、攻击方式、主干架构和数据领域中均有效提升模型鲁棒性。值得注意的是，在经典TextFooler攻击下，无需微调，ProTransformer使BERT、ALBERT、DistilBERT和RoBERTa的性能分别提升19.5%、28.3%、16.1%和11.4%。此外，该方法在大语言模型（LLMs）上对提示攻击也表现出色：在T5和LLaMA上分别提升24.8%和17.8%，对Vicuna在越狱攻击下平均提升10.4%。在视觉与图数据领域，ProTransformer同样展现优异鲁棒性。
原文摘要 · Abstract (English)
Transformer-based architectures have dominated various areas of machine learning in recent years. In this paper, we introduce a novel robust attention mechanism designed to enhance the resilience of transformer-based architectures. Crucially, this technique can be integrated into existing transformers as a plug-and-play layer, improving their robustness without the need for additional training or fine-tuning. Through comprehensive experiments and ablation studies, we demonstrate that our ProTransformer significantly enhances the robustness of transformer models across a variety of prediction tasks, attack mechanisms, backbone architectures, and data domains. Notably, without further fine-tuning, the ProTransformer consistently improves the performance of vanilla transformers by 19.5%, 28.3%, 16.1%, and 11.4% for BERT, ALBERT, DistilBERT, and RoBERTa, respectively, under the classical TextFooler attack. Furthermore, ProTransformer shows promising resilience in large language models (LLMs) against prompting-based attacks, improving the performance of T5 and LLaMA by 24.8% and 17.8%, respectively, and enhancing Vicuna by an average of 10.4% against the Jailbreaking attack. Beyond the language domain, ProTransformer also demonstrates outstanding robustness in both vision and graph domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。