针对孕产及妇幼健康领域，构建首个公开的医疗问答与检索评估基准。
mamabench and mamaretrieval: Benchmarks for Evaluating Medical Retrieval-Augmented Generation in Maternal, Neonatal, and Reproductive Health
- 从7个专家来源整合2.6万条题，按临床场景筛选并多维度评分
- 建立3185个临床查询与63650段指南的细粒度相关性标注数据集
- 公开标注边界与可靠性检验，助力模型评估可信度
现有医疗问答评估基准极少涵盖助产士常问的孕产、新生儿及生殖健康问题，且尚无公开的孕产指南片段级相关性标注基准。本文发布两个新基准：mamabench 是一个包含25,949条题目、源自7个专家撰写来源的多类型问答集，涵盖选择题、简答题和评分题；为辅助评分模型校准，将HealthBench的医师标注元评估重新聚焦至本领域。mamaretrieval 将3,185个临床查询与63,650段孕产指南内容配对，使用0-6分制细粒度相关性标签，并通过分解标准区分真正回答问题的段落与仅涉及主题的段落。两个基准均采用专家来源筛选而非自行编题、采用非二分类相关性评分、并公开披露标签局限性——包括范围分类器一致性、前沿评判者核查及池完整性审计，而非视为绝对真理。配套论文已用其评估部署在设备端的助手系统，全部数据开放供研究使用。
原文摘要 · Abstract (English)
Medical question-answering benchmarks rarely cover the maternal, neonatal, child, and reproductive-health questions a nurse-midwife asks, and, to our knowledge, no public chunk-level relevance benchmark exists for maternal-health guideline retrieval. We release two benchmarks that fill these gaps. mamabench is a scope-filtered QA set of 25,949 items assembled from seven existing expert-authored sources across multiple-choice, short-answer, and rubric-graded tracks; to help users calibrate the LLM judge that scores the rubric track, we re-scope HealthBench's physician-labelled meta-evaluation to the domain. mamaretrieval pairs 3,185 clinical queries with graded (0-6) relevance labels over a 63,650-chunk maternal-health guideline corpus, using a decomposed rubric that distinguishes a chunk that answers a query from one merely on its topic. Three decisions shape both: assemble and filter expert sources rather than author questions, grade relevance rather than binarise it, and measure and disclose the limits of the labels -- scope-classifier agreement, a frontier-judge check, and a pooling-completeness audit -- rather than treat them as an oracle. A companion paper uses the benchmarks to evaluate a deployed on-device assistant; both are released openly for research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。