让小数据场景下的模型更抗干扰，提升参数高效微调的鲁棒性
Small Data, Big Noise: Adversarial Training for Robust Parameter-Efficient Fine-Tuning
- 将对抗训练引入参数高效微调，增强模型对噪声的抵抗力
- 在低资源设置下显著提升性能，字符级和词级扰动下均有效
- 无需额外参数，适合真实世界中小样本且语言多变的场景
参数高效微调（PEFT）已成为适配基础模型至下游自然语言任务的关键技术。然而，现有方法在有限训练数据下易受噪声影响，性能下降明显。本文提出SDBN（Small Data Big Noise）框架，首次将对抗训练引入PEFT，显著提升模型鲁棒性与泛化能力，优于现有方法。提出两种变体：SDBN-h通过枚举字符级扰动并利用梯度选择最坏情况；SDBN-p采用大模型生成扰动变体，适用于生成任务的鲁棒优化。在多个基准测试中，尤其在低资源设置及词级、字符级扰动下表现优异。该框架不引入额外参数或仅增加少量计算开销，填补了对抗训练与参数高效适配交叉研究的空白，使PEFT在数据稀缺与语言多样性并存的真实场景中更具可靠性。
原文摘要 · Abstract (English)
Parameter-Efficient Fine-Tuning (PEFT) has become essential for adapting foundation models to downstream NLP tasks. However, current PEFT methods often struggle with robustness to noise and performance degradation on limited training data. We propose SDBN (Small Data Big Noise), a unified framework that brings adversarial training to PEFT - a combination that remains less studied in the PEFT setting despite its complementary strengths - to enhance model robustness and generalization, outperforming alternative approaches. We also introduce two variants of the method that use discrete uncertainty sets: SDBN-h, which enumerates character-level edits and selects worst-case variants using gradients, and SDBN-p, which uses LLM-generated variants for robust optimization in generative tasks. Experiments across multiple benchmarks reveal substantial improvements, particularly in low-resource settings and under both word-level and character-level corruptions. This framework addresses the less explored intersection of adversarial training and parameter-efficient adaptation, without introducing additional parameters or only modest computational overhead, making PEFT deployments more reliable in real-world scenarios where data scarcity and linguistic variability often coexist
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。