用微分方程思想提升BERT抗攻击能力，不加参数也不训练。
IM-BERT: Enhancing Robustness of BERT through the Implicit Euler Method
- 将BERT层视为微分方程解，用隐式欧拉法构建更稳定的连接
- 在对抗性数据集上性能提升8.3%、低资源下高5.9%准确率
- 适合追求模型鲁棒性的实际应用，尤其资源受限场景
预训练语言模型（PLMs）在多种自然语言任务中表现优异，但其在有限下游数据上微调时易受对抗攻击影响，导致过拟合。本文从动态系统视角出发，将BERT的一层视为常微分方程（ODE）的解，分析显式与隐式欧拉方法在初值扰动下的数值稳定性。基于此，提出无需额外参数或对抗训练的数值稳定连接策略——IM-connection。该方法显著增强模型对对抗攻击的鲁棒性，尤其在低资源场景下仍保持优势。在对抗性GLUE（AdvGLUE）数据集上的实验表明，相比原始BERT，IM-BERT性能提升约8.3%绝对值；在低资源条件下，准确率高出5.9%绝对值。
原文摘要 · Abstract (English)
Pre-trained Language Models (PLMs) have achieved remarkable performance on diverse NLP tasks through pre-training and fine-tuning. However, fine-tuning the model with a large number of parameters on limited downstream datasets often leads to vulnerability to adversarial attacks, causing overfitting of the model on standard datasets. To address these issues, we propose IM-BERT from the perspective of a dynamic system by conceptualizing a layer of BERT as a solution of Ordinary Differential Equations (ODEs). Under the situation of initial value perturbation, we analyze the numerical stability of two main numerical ODE solvers: the explicit and implicit Euler approaches. Based on these analyses, we introduce a numerically robust IM-connection incorporating BERT's layers. This strategy enhances the robustness of PLMs against adversarial attacks, even in low-resource scenarios, without introducing additional parameters or adversarial training strategies. Experimental results on the adversarial GLUE (AdvGLUE) dataset validate the robustness of IM-BERT under various conditions. Compared to the original BERT, IM-BERT exhibits a performance improvement of approximately 8.3\%p on the AdvGLUE dataset. Furthermore, in low-resource scenarios, IM-BERT outperforms BERT by achieving 5.9\%p higher accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。