评估关键词预测模型的同质性，发现生成缺失关键词反而可能降低一致性。
Evaluating the Homogeneity of Keyphrase Prediction Models
- 设计新方法衡量关键词预测模型在相同主题下的输出一致性。
- 实验显示生成缺失关键词的能力反而降低模型同质性。
- 适合关注模型稳定性与索引一致性的信息检索研究者。
关键词在多种自然语言处理和信息检索任务中具有重要作用，可通过文本提取或生成模型预测。与提取方法不同，生成式模型能预测原文未出现的关键词（称作'缺失关键词'），从而将文档关联到未明确提及的概念。这暗示：对于同一主题的两篇文档，生成模型更可能产生一致的关键词索引，而提取模型则难以做到。然而，当前基准尚未涵盖关键词预测模型的同质性评估。本文提出一种评估方法，探究缺失关键词生成能力是否提升模型同质性。出乎意料的是，我们发现提取模型在同质性上与生成模型相当，且生成缺失关键词的能力反而会削弱同质性。相关数据、代码与提示已开源于Hugging Face和GitHub。
原文摘要 · Abstract (English)
Keyphrases which are useful in several NLP and IR applications are either extracted from text or predicted by generative models. Contrarily to keyphrase extraction approaches, keyphrase generation models can predict keyphrases that do not appear in a document's text called `absent keyphrases`. This ability means that keyphrase generation models can associate a document to a notion that is not explicitly mentioned in its text. Intuitively, this suggests that for two documents treating the same subjects, a keyphrase generation model is more likely to be homogeneous in their indexing i.e. predict the same keyphrase for both documents, regardless of those keyphrases appearing in their respective text or not; something a keyphrase extraction model would fail to do. Yet, homogeneity of keyphrase prediction models is not covered by current benchmarks. In this work, we introduce a method to evaluate the homogeneity of keyphrase prediction models and study if absent keyphrase generation capabilities actually help the model to be more homogeneous. To our surprise, we show that keyphrase extraction methods are competitive with generative models, and that the ability to generate absent keyphrases can actually have a negative impact on homogeneity. Our data, code and prompts are available on huggingface and github.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。