arXiv:2411.12235cs.IRcs.CL2024-11EMNLP被引 7

测试密集检索对布尔逻辑的理解能力,发现现有系统仍不足。

BoolQuestions: Does Dense Retrieval Understand Boolean Logic in Language?

  • 构建布尔逻辑检索任务与数据集BoolQuestions
  • 实验证明当前检索系统未完全理解布尔逻辑
  • 提出对比持续训练法作为新基准,适合语言模型研究者

密集检索通过将任意文本编码为稠密向量表示,已成为文本检索的有效高效范式,并成为多种自然语言处理系统的核心组件。这些系统通常通过关注文本对的相关性来优化嵌入空间,却忽视了语言中固有的布尔逻辑,而这一特性可能无法被现有训练目标捕捉。本文首次探究当前检索系统是否能理解语言中的布尔逻辑。为此,我们提出了布尔密集检索任务,并构建了名为BoolQuestions的基准数据集,涵盖包含基础布尔逻辑的复杂查询及对应的标注段落。在该任务和数据集上的大量实验结果表明,当前密集检索系统并未充分理解语言中的布尔逻辑,仍有很长的路要走。此外,为促进进一步研究以提升语言模型对布尔逻辑的理解,我们探索了分解查询上的布尔操作,并提出一种对比持续训练方法,可作为研究社区的强基线。

原文摘要 · Abstract (English)

Dense retrieval, which aims to encode the semantic information of arbitrary text into dense vector representations or embeddings, has emerged as an effective and efficient paradigm for text retrieval, consequently becoming an essential component in various natural language processing systems. These systems typically focus on optimizing the embedding space by attending to the relevance of text pairs, while overlooking the Boolean logic inherent in language, which may not be captured by current training objectives. In this work, we first investigate whether current retrieval systems can comprehend the Boolean logic implied in language. To answer this question, we formulate the task of Boolean Dense Retrieval and collect a benchmark dataset, BoolQuestions, which covers complex queries containing basic Boolean logic and corresponding annotated passages. Through extensive experimental results on the proposed task and benchmark dataset, we draw the conclusion that current dense retrieval systems do not fully understand Boolean logic in language, and there is a long way to go to improve our dense retrieval systems. Furthermore, to promote further research on enhancing the understanding of Boolean logic for language models, we explore Boolean operation on decomposed query and propose a contrastive continual training method that serves as a strong baseline for the research community.

稠密检索布尔逻辑语言理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。