提出新基准QO-Bench,诊断文本中事件查询的运算符保真度。
QO-Bench: Diagnosing Query-Operator-Preserving Retrieval over Typed Event Tuples
- 构建基于类型事件元组的查询运算诊断框架
- 785个问题覆盖18种模板,答案通过精确匹配评估
- 揭示检索系统在连接、交集等操作上严重失效
许多商业、法律和科学领域的自然语言问题本质上是文本中隐含记录的数据库式查询。现有检索增强生成(RAG)系统主要优化语义相关性,但检索到合理段落并不保证查询正确执行。我们提出QO-Bench,一个针对类型化事件元组上查询-运算符问答的诊断基准。该基准涵盖22,984篇新闻文章和614个企业事件,基于18种查询模板,评估785个问题。每个标准答案由类型化事件元组确定性计算得出,通过召回率评分,并以精确匹配方式与黄金元组对齐,而非依赖大模型判断。此设计支持运算符级诊断,如连接和交集。我们在相同条件下评估RAG、ReAct RAG、GraphRAG及信息抽取转SQL方法,并引入长上下文理想模型作为上限,以隔离检索失败。提出两轴框架——索引时保留性与查询时执行性——预测各范式失效点,结果验证其有效性:系统虽能检索相关文本,却丢失运算符所需类型化值;可部署范式排名在不同运算符间反转,相似度检索在过滤/投影上领先,而抽取转SQL在交集和计数任务上更优。即使提供黄金证据,长上下文理想模型仍未饱和,说明运算符执行本身是核心瓶颈,更强答案模型无法解决。QO-Bench将目标从段落相关性重新定义为查询-运算符保真检索。
原文摘要 · Abstract (English)
Many real-world questions over business, legal, and scientific corpora are natural-language versions of database-style queries over records latent in text. Existing retrieval-augmented generation (RAG) systems are optimized primarily for semantic relevance, but retrieving plausible passages does not guarantee correct query execution. We introduce QO-Bench, a diagnostic benchmark for query-operator question answering over typed event tuples. The benchmark covers 22,984 news articles and 614 corporate events across 18 query templates, evaluated on 785 questions. Each gold answer is deterministically computed from typed event tuples and scored by recall, with answers matched to the gold tuples by exact match rather than an LLM judge. This design enables operator-level diagnosis such as joins and intersection. We evaluate RAG, ReAct RAG, GraphRAG, and information-extraction-to-SQL under matched conditions, with a long-context oracle ceiling to isolate retrieval failure. A two-axis framework -- index-time preservation versus query-time execution -- predicts where each paradigm fails, and the results bear it out: systems retrieve relevant text but discard the typed values operators need, and the deployable paradigm ranking inverts across operators, with similarity retrieval leading on filter/project and extraction-to-SQL on intersection and counting. Even given the gold evidence, a long-context oracle stays far from saturated, so operator execution -- not retrieval alone -- is a core bottleneck that a stronger answer model does not remove. QO-Bench reframes the goal from passage relevance to query-operator-preserving retrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。