评测大模型在智能检索生成中的中间推理能力，发现慢思考模型更优。
RAGCap-Bench: Benchmarking Capabilities of LLMs in Agentic Retrieval Augmented Generation Systems
- 构建细粒度基准，评估智能检索生成中的中间任务表现。
- 慢思考模型在中间能力上表现更好，端到端结果更优。
- 适合研究大模型推理机制与检索增强系统优化者参考。
检索增强生成（RAG）通过动态获取外部信息，缓解大语言模型（LLM）的事实错误、知识过时和幻觉等问题。近期工作将该范式扩展为智能体式RAG系统，让LLM作为智能体迭代规划、检索并推理复杂查询。然而，这些系统在多跳问题上仍存在困难，其内部推理能力尚未充分探索。为此，我们提出RAGCap-Bench，一个面向中间任务的细粒度能力评估基准。通过分析先进系统输出，识别常见任务与所需核心能力，并构建典型LLM错误分类，设计针对性评测问题。实验表明，具备更强RAGCap能力的‘慢思考’模型能获得更好的端到端效果，验证了基准的有效性及提升中间能力的重要性。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) mitigates key limitations of Large Language Models (LLMs)-such as factual errors, outdated knowledge, and hallucinations-by dynamically retrieving external information. Recent work extends this paradigm through agentic RAG systems, where LLMs act as agents to iteratively plan, retrieve, and reason over complex queries. However, these systems still struggle with challenging multi-hop questions, and their intermediate reasoning capabilities remain underexplored. To address this, we propose RAGCap-Bench, a capability-oriented benchmark for fine-grained evaluation of intermediate tasks in agentic RAG workflows. We analyze outputs from state-of-the-art systems to identify common tasks and the core capabilities required for their execution, then construct a taxonomy of typical LLM errors to design targeted evaluation questions. Experiments show that "slow-thinking" models with stronger RAGCap performance achieve better end-to-end results, underscoring the benchmark's validity and the importance of enhancing these intermediate capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。