用弱化标签提升模型抗噪声能力,让检索更稳健。
Learning to Retrieve with Weakened Labels: Robust Training under Label Noise
- 用模型置信度生成多个可能标签,替代单一错误标签
- 在4个数据集上优于10种主流损失函数,显著提升检索效果
- 适合标注噪声大、需鲁棒训练的检索任务场景
神经编码器常用于自然语言处理中的密集检索任务,如问答系统中为查询生成候选文档。然而,训练数据中的稀疏标注和标签噪声使检索模型的训练或微调变得困难。尽管已有工作通过改进损失函数或数据清洗来缓解这些问题,但这些方法往往需要调整超参数或增加训练复杂性。本文提出一种标签弱化方法,在存在标签噪声时构建鲁棒的检索模型。不强制每个查询-文档对采用单一可能错误的标签,而是根据观测监督信号与模型置信度,生成一组合理的候选标签。我们在两种检索模型、一个重排序模型上进行广泛评估,覆盖四个多样化的排名数据集,并通过语义感知的噪声生成技术模拟不同比例的噪声。实验结果表明,该方法在性能上超越10种最先进的损失函数。
原文摘要 · Abstract (English)
Neural Encoders are frequently used in the NLP domain to perform dense retrieval tasks, for instance, to generate the candidate documents for a given query in question-answering tasks. However, sparse annotation and label noise in the training data make it challenging to train or fine-tune such retrieval models. Although existing works have attempted to mitigate these problems by incorporating modified loss functions or data cleaning, these approaches either require some hyperparameters to tune during training or add substantial complexity to the training setup. In this work, we consider a label weakening approach to generate robust retrieval models in the presence of label noise. Instead of enforcing a single, potentially erroneous label for each query document pair, we allow for a set of plausible labels derived from both the observed supervision and the model's confidence scores. We perform an extensive evaluation considering two retrieval models, one re-ranking model, considering four diverse ranking datasets. To this end, we also consider a realistic noisy setting by using a semantic-aware noise generation technique to generate different ratios of noise. Our initial results show that label weakening can improve the performance of the retrieval tasks in comparison to 10 different state-of-the-art loss functions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。