用GPT评估搜索结果相关性，发现常见失败模式
Semantic Search Evaluation
- 构建黄金查询集，用GPT 3.5判断结果是否相关
- 提出“在题率”指标，量化相关结果占比
- 适合优化搜索系统相关性的研究人员使用
我们提出一种新型内容搜索系统评估方法，通过衡量查询与返回结果之间的语义匹配度来评估性能。引入“在题率”指标，用于衡量结果中与查询相关的比例。为此，设计了一套流程：定义黄金查询集，为每个查询获取前K个结果，并通过格式化提示词调用GPT 3.5进行判断。该语义评估流程有助于识别常见失败模式，并为提升相关性提供改进目标。
原文摘要 · Abstract (English)
We propose a novel method for evaluating the performance of a content search system that measures the semantic match between a query and the results returned by the search system. We introduce a metric called "on-topic rate" to measure the percentage of results that are relevant to the query. To achieve this, we design a pipeline that defines a golden query set, retrieves the top K results for each query, and sends calls to GPT 3.5 with formulated prompts. Our semantic evaluation pipeline helps identify common failure patterns and goals against the metric for relevance improvements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。