用对比数据微调模型，让AI真正理解语义而非靠表面模式
From Superficial Patterns to Semantic Understanding: Fine-Tuning Language Models on Contrast Sets
- 在训练中加入少量对比样本，引导模型识别细微语义变化
- ELECTRA-small模型在对比集上准确率从75%提升至近90%
- 适合关注模型鲁棒性与真实语义理解的研究者
大规模预训练语言模型在标准自然语言推理（NLI）数据集上表现优异，但其评估结果可能具有误导性。尽管模型在分布内数据上表现良好，但在分布外测试集（如对比集）上性能显著下降。对比集包含经微小但关键改动的输入实例，其标签发生变化，暴露出模型依赖训练数据中的表面模式而非深层语言理解的问题。例如，ELECTRA-small模型在SNLI数据集上准确率达近90%，但在分布外对比集上降至75%。本研究探索通过在训练过程中引入少量复杂对比样本，提升模型鲁棒性。实验表明，该方法使模型在对比集上的准确率恢复至近90%，凸显了多样化且具挑战性训练数据的重要性。
原文摘要 · Abstract (English)
Large-scale pre-trained language models have demonstrated high performance on standard datasets for natural language inference (NLI) tasks. Unfortunately, these evaluations can be misleading, as although the models can perform well on in-distribution data, they perform poorly on out-of-distribution test sets, such as contrast sets. Contrast sets consist of perturbed instances of data that have very minor, but meaningful, changes to the input that alter the gold label, revealing how models can learn superficial patterns in the training data rather than learning more sophisticated language nuances. As an example, the ELECTRA-small language model achieves nearly 90% accuracy on an SNLI dataset but drops to 75% when tested on an out-of-distribution contrast set. The research carried out in this study explores how the robustness of a language model can be improved by exposing it to small amounts of more complex contrast sets during training to help it better learn language patterns. With this approach, the model recovers performance and achieves nearly 90% accuracy on contrast sets, highlighting the importance of diverse and challenging training data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。