构建新基准评估大模型长文本指代能力，发现顶级模型仍存明显短板。
Ref-Long: Benchmarking the Long-context Referencing Capability of Long-context Language Models
- 设计新任务：让模型定位长文本中引用特定关键词的文档索引。
- 13个模型测试显示，连GPT-4o在复杂指代任务中准确率不足60%。
- 适合研究长文本理解、模型指代能力的开发者与评测人员。
长上下文语言模型（LCLMs）在长文本理解任务中表现出色，但其中关键的长上下文指代任务——即要求模型将关注项准确关联到长文本中具体部分——仍缺乏系统评估。为此，本文提出针对长上下文语言模型的指代评估基准Ref-Long，该基准要求模型识别出引用特定关键词的文档索引，强调关键词与文档之间的上下文关系，而非简单检索。基于此任务设计，我们构建了从合成到真实场景的三个子集，形成完整的Ref-Long基准。对13个LCLMs的实验结果揭示显著缺陷，即使先进模型如GPT-4o在指代任务中表现也未达预期。为进一步探究挑战，我们开展人类评估、任务格式调整、微调实验及错误分析，获得若干关键洞察。数据与代码已开源于https://github.com/wujunjie1998/Ref-Long。
原文摘要 · Abstract (English)
Long-context language models (LCLMs) have exhibited impressive capabilities in long-context understanding tasks. Among these, long-context referencing -- a crucial task that requires LCLMs to attribute items of interest to specific parts of long-context data -- remains underexplored. To bridge this gap, this paper proposes Referencing Evaluation for Long-context Language Models (Ref-Long), a novel benchmark designed to assess the long-context referencing capability of LCLMs. Specifically, Ref-Long requires LCLMs to identify the indexes of documents that reference a specific key, emphasizing contextual relationships between the key and the documents over simple retrieval. Based on the task design, we construct three subsets ranging from synthetic to realistic scenarios to form the Ref-Long benchmark. Experimental results of 13 LCLMs reveal significant shortcomings in long-context referencing, even among advanced models like GPT-4o. To further investigate these challenges, we conduct comprehensive analyses, including human evaluations, task format adjustments, fine-tuning experiments, and error analyses, leading to several key insights. Our data and code can be found in https://github. com/wujunjie1998/Ref-Long.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。