用随机掩码输入词元，让Transformer文本分类更稳健、更准确。
Token Masking Improves Transformer-Based Text Classification
- 训练时随机替换词元为[MASK]，引入随机扰动。
- 在多任务上提升性能，最佳掩码率常为10%。
- 适合追求模型泛化力的NLP研究者和工程师。
尽管基于Transformer的模型在文本分类上表现优异，我们探讨了输入词元掩码是否能进一步提升效果。提出一种简单的词元掩码正则化方法：以概率p将输入词元随机替换为特殊[MASK]标记。该方法在训练中引入随机扰动，实现隐式梯度平均，促使模型捕捉更深层的词间依赖关系。在语言识别与情感分析任务上，对mBERT、Qwen2.5-0.5B、TinyLlama-1.1B等多种模型的实验均显示，其效果优于标准正则化技术。我们发现不同任务存在最优掩码率，其中p=0.1作为通用默认值表现良好。性能提升主要归因于两点：(1) 输入扰动降低过拟合，(2) 梯度层面平滑起到隐式集成作用。
原文摘要 · Abstract (English)
While transformer-based models achieve strong performance on text classification, we explore whether masking input tokens can further enhance their effectiveness. We propose token masking regularization, a simple yet theoretically motivated method that randomly replaces input tokens with a special [MASK] token at probability p. This introduces stochastic perturbations during training, leading to implicit gradient averaging that encourages the model to capture deeper inter-token dependencies. Experiments on language identification and sentiment analysis -- across diverse models (mBERT, Qwen2.5-0.5B, TinyLlama-1.1B) -- show consistent improvements over standard regularization techniques. We identify task-specific optimal masking rates, with p = 0.1 as a strong general default. We attribute the gains to two key effects: (1) input perturbation reduces overfitting, and (2) gradient-level smoothing acts as implicit ensembling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。