用预训练模型自评难度，动态调整训练顺序提升NLU效果
Your Pretrained Model Tells the Difficulty Itself: A Self-Adaptive Curriculum Learning Paradigm for Natural Language Understanding
- 让预训练模型自己判断数据难易度，自动排序训练样本
- 在4个NLU数据集上收敛更快，准确率更高
- 适合想提升微调效率的研究者和工程师
课程学习是自然语言处理中常用的训练策略，通过按难度递增顺序呈现样本以提升学习效率与性能。然而，现有方法多依赖人工定义的难度指标（如文本长度），可能无法反映模型真实认知难度。为此，我们提出一种自适应课程学习范式，利用预训练语言模型自身预测的难度得分来优先排序微调样本。基于这些得分，我们探索了从易到难、从难到易以及混合采样等不同训练策略。在涵盖二分类与多分类任务的四个自然语言理解数据集上评估表明，该方法相比标准随机采样能实现更快收敛并提升性能。
原文摘要 · Abstract (English)
Curriculum learning is a widely adopted training strategy in natural language processing (NLP), where models are exposed to examples organized by increasing difficulty to enhance learning efficiency and performance. However, most existing approaches rely on manually defined difficulty metrics -- such as text length -- which may not accurately reflect the model's own perspective. To overcome this limitation, we present a self-adaptive curriculum learning paradigm that prioritizes fine-tuning examples based on difficulty scores predicted by pre-trained language models (PLMs) themselves. Building on these scores, we explore various training strategies that differ in the ordering of examples for the fine-tuning: from easy-to-hard, hard-to-easy, to mixed sampling. We evaluate our method on four natural language understanding (NLU) datasets covering both binary and multi-class classification tasks. Experimental results show that our approach leads to faster convergence and improved performance compared to standard random sampling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。