用生成方法解决小样本文本数据不平衡问题。
SMOTExT: SMOTE meets Large Language Models
- 基于BERT嵌入插值生成新文本，结合xRAG解码
- 仅用生成数据训练模型，性能接近真实数据
- 适合低资源场景与隐私保护下的模型训练
数据稀缺和类别不平衡是训练稳健NLP模型的长期挑战，尤其在专业领域或低资源场景下。我们提出一种新方法SMOTExT，将合成少数类过采样（SMOTE）思想拓展至文本数据。该方法通过插值两个现有样本的BERT嵌入生成新向量，并利用xRAG跨模态检索-生成架构将其解码为连贯文本。尽管目前仅为初步研究，仅依赖定性输出，但该方法在少样本设置下展现出显著的知识蒸馏与数据增强潜力。早期实验表明，仅使用生成数据训练的模型性能可媲美使用原始数据训练的模型，暗示其在数据保护约束下实现安全高效学习的可行性。
原文摘要 · Abstract (English)
Data scarcity and class imbalance are persistent challenges in training robust NLP models, especially in specialized domains or low-resource settings. We propose a novel technique, SMOTExT, that adapts the idea of Synthetic Minority Over-sampling (SMOTE) to textual data. Our method generates new synthetic examples by interpolating between BERT-based embeddings of two existing examples and then decoding the resulting latent point into text with xRAG architecture. By leveraging xRAG's cross-modal retrieval-generation framework, we can effectively turn interpolated vectors into coherent text. While this is preliminary work supported by qualitative outputs only, the method shows strong potential for knowledge distillation and data augmentation in few-shot settings. Notably, our approach also shows promise for privacy-preserving machine learning: in early experiments, training models solely on generated data achieved comparable performance to models trained on the original dataset. This suggests a viable path toward safe and effective learning under data protection constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。