构建自动化框架，评估大模型对长文本的多层级理解能力
Towards a Holistic and Automated Evaluation Framework for Multi-Level Comprehension of LLMs in Book-Length Contexts
- 按根、枝、叶三级结构组织文本关键事实
- 自动评估显示模型在细粒度信息上表现差，且受位置影响
- 适合研究长文本理解的学者与模型开发者
我们提出 HAMLET，一个全面且自动化的框架，用于评估大语言模型（LLMs）在书籍级长文本中的多层级理解能力。HAMLET 将源文本构建为根、枝、叶三个层次的关键事实层级，并采用聚焦查询的摘要方法，评估模型在各层级上的信息回忆与忠实度表现。为验证全自动流程的可靠性，我们开展系统性人工研究，结果显示自动评估与专家人工判断的吻合度超过90%，同时成本降低至原来的1/25。HAMLET 发现，模型在细粒度理解（尤其叶级）方面表现不佳，且对位置效应（如中间丢失）敏感；分析类查询比叙事类更具挑战性，且开源与专有模型之间、不同模型规模间均存在持续性能差距。代码与数据集已公开于 https://github.com/DISL-Lab/HAMLET。
原文摘要 · Abstract (English)
We introduce HAMLET, a holistic and automated framework for evaluating the long-context comprehension of large language models (LLMs). HAMLET structures source texts into a three-level key-fact hierarchy at root-, branch-, and leaf-levels, and employs query-focused summarization to evaluate how well models recall and faithfully represent information at each level. To validate the reliability of our fully automated pipeline, we conduct a systematic human study, showing that our automatic evaluation achieves over 90% agreement with expert human judgments, while reducing the cost by up to 25 times. HAMLET reveals that LLMs struggle with fine-grained comprehension, especially at the leaf level, and are sensitive to positional effects like the lost-in-the-middle. Analytical queries pose greater challenges than narrative ones, and consistent performance gaps emerge between open-source and proprietary models, as well as across model scales. Our code and dataset are publicly available at https://github.com/DISL-Lab/HAMLET.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。