用伪投影器提升Transformer对噪声的鲁棒性，无需改动模型结构。
Correction of Transformer-Based Models with Smoothing Pseudo-Projector
- 基于多网格思想设计轻量级伪投影器，修正隐藏层表示
- 在文本分类和合成任务中显著改善训练动态与抗噪能力
- 适合希望增强现有模型鲁棒性的研究者与工程师
伪投影器是一种轻量级修改，可无缝集成到现有语言模型及其他神经网络中，无需改变其核心架构。它可视为隐藏表示的修正器，通过抑制由标签无关输入内容引发的方向，降低对噪声的敏感性。该设计灵感源自多网格（MG）范式，最初用于加速偏微分方程和边界值问题的迭代求解，后扩展至更一般的线性系统。我们称之为伪投影器，因其线性原型对应于严格幂等的正交投影算子，而实际实现采用可学习的限制与延拓算子，因此一般不满足精确正交投影性质。我们在基于Transformer的文本分类任务及受控合成基准上评估该方法，证明其在提升训练动态与鲁棒性方面的有效性。实验结果结合支持性的理论启发，表明在多种设置下均有稳定改进，未观察到负面效应。下一步将把该方法扩展至语言模型。
原文摘要 · Abstract (English)
The pseudo-projector is a lightweight modification that can be integrated into existing language models and other neural networks without altering their core architecture. It can be viewed as a hidden-representation corrector that reduces sensitivity to noise by suppressing directions induced by label-irrelevant input content. The design is inspired by the multigrid (MG) paradigm, originally developed to accelerate the convergence of iterative solvers for partial differential equations and boundary value problems, and later extended to more general linear systems through algebraic multigrid methods. We refer to the method as a pseudo-projector because its linear prototype corresponds to a strictly idempotent orthogonal projector, whereas the practical formulation employs learnable restriction and prolongation operators and therefore does not, in general, satisfy the properties of an exact orthogonal projection. We evaluate the proposed approach on transformer-based text classification tasks, as well as controlled synthetic benchmarks, demonstrating its effectiveness in improving training dynamics and robustness. Experimental results, together with supporting theoretical heuristics, indicate consistent improvements in training behavior across a range of settings, with no adverse effects observed otherwise. Our next step will be to extend this approach to language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。