用训练动态划分数据集，无法提升SQuAD任务的抗干扰能力。
When is dataset cartography ineffective? Using training dynamics does not improve robustness against Adversarial SQuAD
- 基于训练动态将SQuAD分为易学、模糊、难学三类样本。
- 在对抗数据集AddSent上，分组训练未提升泛化性能。
- 对难学样本单独训练，仅轻微改善对AddOneSent的准确率。
本文研究了在SQuAD数据集上使用数据集制图（dataset cartography）对抽取式问答任务的有效性。通过分析SQuAD中的标注缺陷，评估了两个对抗性数据集AddSent和AddOneSent对ELECTRA-small模型的影响。利用训练动态将SQuAD划分为易学、模糊和难学三类子集，并对比在这些子集上训练的模型与随机采样同等规模样本训练的模型表现。结果表明，基于制图的训练并未提升模型在SQuAD验证集或AddSent对抗集上的泛化能力。尽管难学子集在AddOneSent上带来略高F1值，但整体增益有限。研究认为，数据集制图在类似SQuAD的问答任务中对增强对抗鲁棒性作用甚微。最后，将结果与SNLI任务的前期发现对比，探讨可能差异原因。
原文摘要 · Abstract (English)
In this paper, I investigate the effectiveness of dataset cartography for extractive question answering on the SQuAD dataset. I begin by analyzing annotation artifacts in SQuAD and evaluate the impact of two adversarial datasets, AddSent and AddOneSent, on an ELECTRA-small model. Using training dynamics, I partition SQuAD into easy-to-learn, ambiguous, and hard-to-learn subsets. I then compare the performance of models trained on these subsets to those trained on randomly selected samples of equal size. Results show that training on cartography-based subsets does not improve generalization to the SQuAD validation set or the AddSent adversarial set. While the hard-to-learn subset yields a slightly higher F1 score on the AddOneSent dataset, the overall gains are limited. These findings suggest that dataset cartography provides little benefit for adversarial robustness in SQuAD-style QA tasks. I conclude by comparing these results to prior findings on SNLI and discuss possible reasons for the observed differences.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。