通过生成合成特征提升语言模型在不平衡数据下的泛化能力
Synthetic Feature Augmentation Improves Generalization Performance of Language Models
- 在嵌入空间生成合成样本，缓解数据类别不平衡问题
- 在多个文本分类基准上显著提升少数类表现，改善整体泛化性能
- 适合处理标注数据稀缺或分布不均的自然语言任务
在有限且不平衡的数据集上训练和微调深度学习模型，尤其是大语言模型（LLMs），面临巨大挑战。这些问题常导致模型过拟合于主导类别，对少数类别表现不佳，从而产生偏差预测，并降低真实场景中的鲁棒性。为此，我们提出在嵌入空间中通过多种技术生成合成样本以增强特征。通过上采样代表性不足的类别，该方法提升了模型性能并缓解了数据不平衡。我们在多个开源文本分类基准上验证了该方法的有效性，证明其在不平衡数据场景下具有提升模型鲁棒性和泛化能力的潜力。
原文摘要 · Abstract (English)
Training and fine-tuning deep learning models, especially large language models (LLMs), on limited and imbalanced datasets poses substantial challenges. These issues often result in poor generalization, where models overfit to dominant classes and underperform on minority classes, leading to biased predictions and reduced robustness in real-world applications. To overcome these challenges, we propose augmenting features in the embedding space by generating synthetic samples using a range of techniques. By upsampling underrepresented classes, this method improves model performance and alleviates data imbalance. We validate the effectiveness of this approach across multiple open-source text classification benchmarks, demonstrating its potential to enhance model robustness and generalization in imbalanced data scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。