构建中文医学报告理解基准,提升医疗图文信息提取准确性
MedRepBench: A Comprehensive Benchmark for Medical Report Interpretation
- 设计端到端视觉语言模型评估框架,聚焦报告字段结构化提取
- 涵盖1925张真实病历图像,覆盖多科室与多种成像格式
- 对比OCR+LLM方案,验证端到端模型在布局错误下的鲁棒性优势
从真实病历图像中理解医学报告对生成患者可读说明和实现临床系统结构化信息交换至关重要。现有视觉语言模型(VLM)和大语言模型(LLM)在文档理解方面表现良好,但医学报告的结构化理解仍缺乏充分评测。为此,我们推出MedRepBench,一个包含1,925张去标识化的中文医学报告图像的基准数据集,覆盖多个科室、患者人群及采集格式。该基准专注于报告基础的结构化解释,而非诊断推理或病史整合。解释任务定义为:提取报告字段(如项目、数值、单位、参考范围、异常标志)并生成严格基于报告内容的患者可读说明。评估框架包括两种互补协议:(1) 客观协议,衡量字段级召回率;(2) 自动化主观协议,使用基于LLM的评判器在固定提示下评分事实性、可解释性和推理质量。以客观指标为奖励信号,我们提供轻量级GRPO对齐基线,使中等规模VLM的字段召回率提升最高达6%。此外,分析表明OCR+LLM流程存在布局相关错误和额外延迟等实际限制,凸显端到端视觉理解的必要性。数据集与评估资源已公开于https://huggingface.co/datasets/MedRepBench/MedRepBench。
原文摘要 · Abstract (English)
Medical report understanding from real-world document images is essential for generating patient-facing explanations and enabling structured information exchange in clinical systems. Existing VLMs and LLMs have shown strong performance on document understanding, but structured understanding of medical reports remains insufficiently benchmarked. Therefore, we introduce MedRepBench, a benchmark with 1,925 de-identified Chinese medical report images spanning diverse departments, patient demographics, and acquisition formats. In MedRepBench, we mainly focus on report-grounded interpretation rather than evaluating diagnostic reasoning, treatment recommendation, or the integration of patient history. The interpretation is defined as structured extraction of report fields (e.g., item, value, unit, reference range, abnormal flag) plus a patient-facing explanation grounded strictly in the report content. The benchmark primarily evaluates end-to-end VLMs, and also includes a controlled text-only setting (high-quality OCR + LLM) to approximate an upper bound when character recognition errors are minimized. Our evaluation framework provides two complementary protocols: (1) an objective protocol measuring field-level recall of structured items, and (2) an automated subjective protocol that uses an LLM-based judge to score factuality, interpretability, and reasoning quality under a fixed prompt. Using the objective metric as a reward signal, we also provide a lightweight GRPO-based alignment baseline for a mid-sized VLM, which improves field-level recall by up to 6%. Finally, we analyze practical limitations of OCR+LLM pipelines, including layout-related errors and additional system latency, showing the need for robust end-to-end vision-based medical report understanding. The dataset and evaluation resources are publicly available on https://huggingface.co/datasets/MedRepBench/MedRepBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。