arXiv:2505.23868cs.LGcs.AI2025-05NeurIPS被引 4

用生成的噪声数据训练模型,让其自动抗干扰,无需清洗原始数据。

Noise-Robustness Through Noise: A Framework combining Asymmetric LoRA with Poisoning MoE

  • 设计不对称LoRA结构,加入专门处理噪声的专家模块。
  • 训练时注入噪声提升专家识别能力,推理时屏蔽该专家输出纯净结果。
  • 仅靠低成本噪声注入就实现强鲁棒性,无需数据预处理。

当前参数高效微调方法在面对噪声数据时易受干扰。传统处理方式依赖繁琐的数据预处理或易积累误差的模型结构修改。本文提出一种新型框架LoPE(Asymmetric LoRA Poisoning Experts),通过生成噪声数据实现模型对噪声的鲁棒性。受混合专家架构启发,LoPE在非对称LoRA配置中引入专用污染专家。采用两阶段策略:训练时对污染专家注入噪声以增强其噪声辨识与处理能力;推理时选择性屏蔽该专家,利用正常专家学习到的净化知识生成鲁棒输出。大量实验表明,仅通过低代价噪声注入,LoPE即可实现优异性能与鲁棒性,完全消除数据清洗需求。

原文摘要 · Abstract (English)

Current parameter-efficient fine-tuning methods for adapting pre-trained language models to downstream tasks are susceptible to interference from noisy data. Conventional noise-handling approaches either rely on laborious data pre-processing or employ model architecture modifications prone to error accumulation. In contrast to existing noise-process paradigms, we propose a noise-robust adaptation method via asymmetric LoRA poisoning experts (LoPE), a novel framework that enhances model robustness to noise only with generated noisy data. Drawing inspiration from the mixture-of-experts architecture, LoPE strategically integrates a dedicated poisoning expert in an asymmetric LoRA configuration. Through a two-stage paradigm, LoPE performs noise injection on the poisoning expert during fine-tuning to enhance its noise discrimination and processing ability. During inference, we selectively mask the dedicated poisoning expert to leverage purified knowledge acquired by normal experts for noise-robust output. Extensive experiments demonstrate that LoPE achieves strong performance and robustness purely through the low-cost noise injection, which completely eliminates the requirement of data cleaning.

参数高效噪声鲁棒LoRA专家系统

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。