提出可验证的深层修复框架,让Transformer模型更抗对抗攻击。
WARP: Guaranteed Inner-Layer Repair of NLP Transformers
- 基于凸二次规划构建修复优化,支持全层参数调整。
- 实测在多种架构上提升抗扰动能力,且修复结果可验证。
- 适合关注模型安全与可靠性的研究人员使用。
基于Transformer的NLP模型仍易受对抗扰动影响,现有修复方法存在根本权衡:梯度方法灵活但不可验证且易过拟合;提供可验证修复的方法仅限于最后一层或小型网络,严重限制参数搜索空间。本文提出WARP(Weight-Adjusted Repair with Provability),一种基于约束的修复框架,将修复扩展至Transformer模型的任意内层。WARP通过逻辑输出差距的一阶线性化构建凸二次规划问题,实现高维参数空间上的可计算优化。在该一阶近似成立的前提下,该方法对每个样本提供三类保证:(i) 正 margin 约束以确保修复后输入分类正确;(ii) 对指定保留集的保持约束;(iii) 基于Lipschitz连续性的认证鲁棒半径。为确保跨不同模型架构的可行性,引入基于敏感性的预处理步骤以调节优化景观。进一步证明,在温和假设下,迭代优化过程收敛至满足所有修复约束的解。在不同层结构的编码器型Transformer上的实证评估表明,这些保证在实践中成立,并显著提升对对抗输入的鲁棒性。结果表明,通过原则性的约束优化,实现可验证且泛化的Transformer修复是可行的。
原文摘要 · Abstract (English)
Transformer-based NLP models remain vulnerable to adversarial perturbations, yet existing repair methods face a fundamental trade-off: gradient-based approaches offer flexibility but lack verifiability and often overfit; methods that do provide repair guarantees are restricted to the final layer or small networks, significantly limiting the parameter search space available for repair. We present WARP (Weight-Adjusted Repair with Provability), a constraint-based repair framework that extends repair beyond the last layer of Transformer models. WARP formulates repair as a convex quadratic program derived from a first-order linearization of the logit gap, enabling tractable optimization over a high-dimensional parameter space. Under the condition that the first-order approximation holds, this formulation induces three per-sample guarantees: (i) a positive margin constraint ensuring correct classification on repaired inputs, (ii) preservation constraints over a designated remain set, and (iii) a certified robustness radius derived from Lipschitz continuity. To ensure feasibility across varying model architectures, we introduce a sensitivity-based preprocessing step that conditions the optimization landscape accordingly. We further show that the iterative optimization procedure converges to solutions satisfying all repair constraints under mild assumptions. Empirical evaluation on encoder-only Transformers with varying layer architectures validates that these guarantees hold in practice while improving robustness to adversarial inputs. Our results demonstrate that guaranteed, generalizable Transformer repair is achievable through principled constraint-based optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。