arXiv:2608.03397cs.AI2026-08

修复长文档问答基准的标注错误,提升评测公平性

MMLongBench-Doc-V2: A Corrected-Annotation, Semantics-Aware Revision of MMLongBench-Doc

  • 修正106处错误标注,每处附页码与计算依据
  • 改用大模型判断答案语义是否匹配,避免字符串误差
  • 清理错误文件名和重复问题,保留更可靠的1071题

MMLongBench-Doc 是一个包含1,082个问题、覆盖135份PDF的长文档问答基准。其原始评测存在两大缺陷:参考答案对比采用字符串匹配,导致'1,358,000'被误判为低于'1358000';且大量真实答案标注错误、模糊或不完整,尤其集中在系统能正确回答的问题上。MMLongBench-Doc-V2 修正了106处标注,每条均附带页码与算式说明,并将字符串匹配替换为由大模型执行的语义判断,以判定响应是否与参考答案语义一致。移除了10个因文件名错误而被错配的问题,以及1个重复问题,最终得到1,071个问题,对应134份文档。核心贡献是提出一套决策流程,用于判断空集键是否可扩展,应用于全部208行数据后,成功扩展了14行。V2版本分数不可与V1直接比较。修正后的数据集、每项修正记录及评估工具已公开于 https://github.com/VectifyAI/MMLongBench-Doc-V2。

原文摘要 · Abstract (English)

MMLongBench-Doc is a long-document QA benchmark of 1,082 questions over 135 PDFs. Two properties of it push measured scores away from the quantity they are meant to capture: the reference metric compares extracted answers, so 1,358,000 loses to 1358000; and a non-trivial share of ground-truth annotations are wrong, ambiguous, or incomplete --- concentrated, because of how they were found, in exactly the questions capable systems answer correctly. MMLongBench-Doc-V2 corrects 106 annotations, each published with the page and arithmetic that settle it, and replaces the string metric with a pinned LLM judge asked whether a response means the reference. Ten questions whose document ships under the wrong filename are removed rather than counted wrong, along with one duplicated question, leaving 1,071 questions over 134 documents. The most reusable contribution is a decision procedure for when an empty set key may be widened and when widening would destroy a deliberate negative sample; applied to all 208 rows, it widened 14. V2 scores are not comparable with published V1 numbers. The corrected corpus, the per-entry correction record and the evaluation harness are available at https://github.com/VectifyAI/MMLongBench-Doc-V2.

评测基准标注修正语义评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。