用预训练模型和规则生成孟加拉语数据，提升小样本分类效果
BDA: Bangla Text Data Augmentation Framework
- 结合预训练模型与规则生成新文本，保持原意并增加词汇多样性
- 仅用50%数据+增强后，分类F1值达使用100%原始数据的水平
- 适合资源有限的低语料语言文本任务，尤其适合小样本场景
数据增强通过生成与现有数据相似的合成样本，提升训练数据量。在高质量数据稀缺的资源受限领域，该技术尤为重要。本文提出孟加拉语文本数据增强（BDA）框架，结合预训练模型与规则方法生成文本新变体，并通过筛选机制确保新文本语义一致且词汇多样。我们在五个不同孟加拉语文本分类数据集上全面评估了该框架的效果。结果显示，使用仅50%原始训练数据并经BDA增强后，模型在多个数据集上的F1分数显著提升，性能等同于使用100%数据训练的模型。此外，我们通过逐步减少训练数据并进行增强，验证了其在数据稀缺条件下的有效性，带来明显的F1分数提升。研究还深入分析了影响性能的关键因素，明确了框架的优势与局限。
原文摘要 · Abstract (English)
Data augmentation involves generating synthetic samples that resemble those in a given dataset. In resource-limited fields where high-quality data is scarce, augmentation plays a crucial role in increasing the volume of training data. This paper introduces a Bangla Text Data Augmentation (BDA) Framework that uses both pre-trained models and rule-based methods to create new variants of the text. A filtering process is included to ensure that the new text keeps the same meaning as the original while also adding variety in the words used. We conduct a comprehensive evaluation of the framework's effectiveness in Bangla text classification tasks. Our framework achieved significant improvement in F1 scores across five distinct datasets, delivering performance equivalent to models trained on 100% of the data while utilizing only 50% of the training dataset. Additionally, we explore the impact of data scarcity by progressively reducing the training data and augmenting it through BDA, resulting in notable F1 score enhancements. The study offers a thorough examination of BDA's performance, identifying key factors for optimal results and addressing its limitations through detailed analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。