提出评估RAG系统时该用多细粒度的基准，方法可自适应调整。
How Fine-Grained Should a RAG Benchmark Be? A Hierarchical Framework for Synthetic Question Generation
- 构建分层框架HieraRAG，按判别力确定最优粒度
- 复杂度维度需细粒度（判别力0.053），类型和语言变化中等即可
- 引入一致性比率指标，帮判断细分是否合理，适合做评测的工程师
评估检索增强生成（RAG）系统需要能捕捉多样化问题特征的基准，但实践者缺乏在何种维度及粒度上进行变化的实证指导。我们提出HieraRAG，一个用于研究RAG基准构建粒度的分层框架，将最优粒度定义为在给定RAG配置下使生成质量在各类别间标准差最大化的水平。作为案例研究，我们在FineWeb-10BT上生成5,872个合成问答对，涵盖三个维度（问题复杂度、答案类型、语言变体）在三种粒度级别（2、4、8类）下的组合。使用BM25+Falcon-3-10B管道，发现不同维度的最优粒度不同：复杂度宜细粒度（判别力0.053），而答案类型与语言变体在中等粒度时达到峰值。我们引入一致比率（Coherence Ratio）量化细粒度划分是否清晰分割父类别，揭示各维度结构差异（问题复杂度：0.40，答案类型：1.44）。对110个分层抽样问答对的人工评估确认了合成质量。尽管这些发现基于单一配置，但HieraRAG提供了一种可迁移的流程与验证指标，供从业者在自身RAG设置中确定评估粒度。
原文摘要 · Abstract (English)
Evaluating retrieval-augmented generation (RAG) systems requires benchmarks that capture diverse question characteristics, yet practitioners lack empirical guidance on which dimensions to vary and at what granularity. We present HieraRAG, a hierarchical framework for studying granularity in RAG benchmark construction, defining optimal granularity as the level that maximizes discriminative power (the standard deviation of generation quality across categories) within a given RAG configuration. As a case study, we generate 5,872 synthetic question-answer (QA) pairs from FineWeb-10BT across 3 dimensions (Question Complexity, Answer Type, Linguistic Variation) at 3 granularity levels (2, 4, and 8 categories). With a BM25+Falcon-3-10B pipeline, optimal granularity varies by dimension: complexity benefits from fine-grained distinctions (discriminative power: 0.053) while answer type and linguistic variation peak at medium granularity. We introduce a Coherence Ratio metric to quantify whether fine-grained splits cleanly subdivide parent categories, revealing structural differences across dimensions (Question Complexity: 0.40 vs. Answer Type: 1.44). Human evaluation of 110 stratified QA pairs confirms synthetic quality. While these specific findings reflect a single configuration, HieraRAG provides a portable procedure and validation metric for practitioners to determine evaluation granularity within their own RAG settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。