提出更贴近真实搜索的评估框架,解决智能搜索评估不准、过程忽略的问题。
RAVine: Reality-Aligned Evaluation for Agentic Search
- 设计多点查询与长文本回答,更贴合用户实际需求
- 引入可追溯的真值构建策略,提升细粒度评估准确性
- 评估模型迭代搜索过程与效率,适合研究智能搜索系统者
智能搜索作为更具自主性与适应性的检索增强范式,正推动智能搜索系统的发展。然而,现有评估框架难以契合智能搜索的目标。首先,当前基准普遍使用复杂查询,偏离真实用户搜索场景;其次,以往方法在端到端评估中提取真值时易引入噪声,导致细粒度评估失真;第三,多数框架仅关注最终答案质量,忽视智能搜索固有的迭代过程。为此,我们提出RAVine——面向智能语言模型搜索的现实对齐评估框架。RAVine聚焦多点查询与长文本回答,更真实反映用户意图,并提出可追溯的真值构建策略,提升细粒度评估精度。同时,该框架考察模型在迭代过程中与搜索工具的交互行为,并纳入效率因素考量。我们基于RAVine对一系列模型进行评测,获得若干关键洞察,期望推动智能搜索系统的发展。代码与数据集已开源:https://github.com/SwordFaith/RAVine。
原文摘要 · Abstract (English)
Agentic search, as a more autonomous and adaptive paradigm of retrieval augmentation, is driving the evolution of intelligent search systems. However, existing evaluation frameworks fail to align well with the goals of agentic search. First, the complex queries commonly used in current benchmarks often deviate from realistic user search scenarios. Second, prior approaches tend to introduce noise when extracting ground truth for end-to-end evaluations, leading to distorted assessments at a fine-grained level. Third, most current frameworks focus solely on the quality of final answers, neglecting the evaluation of the iterative process inherent to agentic search. To address these limitations, we propose RAVine -- a Reality-Aligned eValuation framework for agentic LLMs with search. RAVine targets multi-point queries and long-form answers that better reflect user intents, and introduces an attributable ground truth construction strategy to enhance the accuracy of fine-grained evaluation. Moreover, RAVine examines model's interaction with search tools throughout the iterative process, and accounts for factors of efficiency. We benchmark a series of models using RAVine and derive several insights, which we hope will contribute to advancing the development of agentic search systems. The code and datasets are available at https://github.com/SwordFaith/RAVine.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。