评测大模型在动态文档中的时间版本理解能力，发现准确率不足70%。
Time Present and Time Past: Benchmarking Large Language Models on Temporally Evolving Document Understanding
- 构建覆盖1969-2025年的孟加拉国海关文件问答数据集，支持多任务与双历法
- 大模型在版本定位上最高准确率仅68.5%，隐含日期理解低至59.7%
- 模型更倾向相信参数化答案而非官方文本，错误拒绝率仅26.7%
动态文档（如法律、税法、软件文档）随时间修订、替换甚至回滚，同一问题在不同日期有不同正确答案。与仅覆盖事实更新的百科知识不同，修订本身是正式文本，说明所替代内容及生效时间，旧版本在有效期内仍正确。核心挑战在于版本解析，即确定查询日期时在效力的版本。现有时间问答数据集仅将时间作为标注，未测试版本解析能力。本文提出TIDE，一个由专家验证的基准，包含3,050个问答对，覆盖1969至2025年间孟加拉国政府发布的644份官方海关文件，涵盖八类任务，文档高度混码，布局异构，且标注双历法日期。我们以统一协议评估九个近期大模型，在参数化、黄金上下文和检索三种访问方式下，由三名评委组成的LLM委员会评分，并设置硬性日期门限区分语义正确与时间正确。最佳宏观平均准确率为68.5%。从隐含日期中解析版本的准确率为59.7%，判断所提供版本不适用于查询的准确率仅为26.7%。模型更擅长找到正确版本，但难以识别错误版本；倾向于信任自信的参数化回答，而非提供的权威文本。所有代码与数据见https://github.com/icsetepa44/TIDE。
原文摘要 · Abstract (English)
Evolving documents, such as laws, tax codes, and software documentation, are amended, replaced, and sometimes reverted over time, so a question has different correct answers at different dates. In contrast to encyclopedic knowledge, where an old fact is simply overwritten, an amendment is itself an official text that states what it replaces and when it takes effect, and the earlier version stays correct for its validity period. The central challenge is therefore version resolution, that is, identifying the version in force on the queried date. Existing temporal QA datasets treat time only as an annotation, so version resolution stays untested. We present TIDE, an expert-verified benchmark of 3,050 QA pairs over 644 official customs instruments issued between 1969 and 2025 by the Government of Bangladesh, covering eight task types over deeply code-mixed documents that are heterogeneous in layout and dated in two calendars. In addition, we evaluate nine recent LLMs under a single protocol across parametric, gold-context, and retrieval access, scored by a three-judge LLM council with a hard date gate separating correct meaning from correct time. The best macro-averaged accuracy is only 68.5%. Resolving a version from an implicit date reaches 59.7%, and detecting that the supplied version does not govern the query reaches only 26.7%. Models are more likely to find correct versions than to reject incorrect ones, and they tend to follow a confident parametric answer over the supplied authoritative text. All code and data are available at https://github.com/icsetepa44/TIDE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。