检验测试集无标签数据预训练对少样本分类的公平性影响
Evaluating the fairness of task-adaptive pretraining on unlabeled test data before few-shot text classification
- 在25个任务上对比使用测试集与独立数据预训练的效果
- 未发现利用测试集无标签数据导致结果过度乐观
- 建议少样本基准包含多次采样以提升可靠性
少样本学习基准对于评估现代NLP技术至关重要。然而，若研究者可利用测试集中的无标签文本进行模型预训练，则可能使某些方法获得不公平优势。鉴于对此潜在问题的研究匮乏，本文通过在25个分类任务和3个语言模型（BERT、GPT-2、Mistral 7B）上开展受控的少样本与零样本实验，量化了使用测试集无标签文本预训练所带来的偏差。结果显示，未发现因预训练数据来源导致结果过度乐观的证据。此外，本文强调在少样本文本分类研究中重复子采样的重要性，并建议少样本学习基准应包含多个训练折（training folds）。代码与数据已公开于https://github.com/kddubey/pretrain-on-test/。
原文摘要 · Abstract (English)
Few-shot learning benchmarks are critical for evaluating modern NLP techniques. It is possible, however, that benchmarks favor methods which easily make use of unlabeled text, because researchers can use unlabeled text from the test set to pretrain their models. Given the dearth of research on this potential problem, we run experiments to quantify the bias caused by pretraining on unlabeled test set text instead of on unlabeled, independently drawn text. Controlled few-shot and zero-shot experiments on 25 classification tasks and 3 language models -- BERT, GPT-2, and Mistral 7B -- do not find evidence of overoptimism. Furthermore, we demonstrate the importance of repeated subsampling when studying few-shot text classification, and recommend that few-shot learning benchmarks include multiple training folds. Code and data are available at https://github.com/kddubey/pretrain-on-test/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。