提出新评估标准，检测大模型能否用全篇长文本信息。
ETHIC: Evaluating Large Language Models on Long-Context Tasks with High Information Coverage
- 用信息覆盖率衡量模型实际使用上下文的程度。
- 现有基准仅覆盖少量有效信息，模型表现大幅下降。
- 适合关注长文本理解的开发者与研究者使用。
当前大语言模型处理超长文本的能力迅速提升，亟需专用评估基准来检验其长上下文能力。然而，现有方法（如针在 haystack 测试）无法有效评估模型是否充分利用了上下文信息，引发对评估可靠性担忧。为此，我们引入信息覆盖率（IC）这一新指标，量化回答问题所需输入上下文的比例。研究发现，现有基准的 IC 值普遍偏低：尽管输入上下文冗长，但实际可用信息有限。为解决此问题，我们提出 ETHIC 基准，涵盖书籍、辩论、医学和法律四个领域共 1,986 个测试实例，所有任务均具备高信息覆盖率。评估显示，主流大模型在该基准上性能显著下降，暴露出长上下文管理的关键挑战。相关数据集已开源：https://github.com/dmis-lab/ETHIC。
原文摘要 · Abstract (English)
Recent advancements in large language models (LLM) capable of processing extremely long texts highlight the need for a dedicated evaluation benchmark to assess their long-context capabilities. However, existing methods, like the needle-in-a-haystack test, do not effectively assess whether these models fully utilize contextual information, raising concerns about the reliability of current evaluation techniques. To thoroughly examine the effectiveness of existing benchmarks, we introduce a new metric called information coverage (IC), which quantifies the proportion of the input context necessary for answering queries. Our findings indicate that current benchmarks exhibit low IC; although the input context may be extensive, the actual usable context is often limited. To address this, we present ETHIC, a novel benchmark designed to assess LLMs' ability to leverage the entire context. Our benchmark comprises 1,986 test instances spanning four long-context tasks with high IC scores in the domains of books, debates, medicine, and law. Our evaluations reveal significant performance drops in contemporary LLMs, highlighting a critical challenge in managing long contexts. Our benchmark is available at https://github.com/dmis-lab/ETHIC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。