探究大模型在长文本中找关键信息的瓶颈，发现数据类型和大小影响远超上下文长度。
DENIAHL: In-Context Features Influence LLM Needle-In-A-Haystack Abilities
- 构建合成基准DENIAHL，系统测试数据类型、大小、模式对记忆的影响
- GPT-3.5与LLaMA 2-7B在不同数据特征下表现差异显著，项大小增加导致召回率下降
- 揭示非数量因素（如数据类型）对长上下文记忆能力的关键作用，适合模型评估者参考
针在草堆中（NIAH）测试是评估语言模型从长输入上下文中检索特定信息能力的通用任务。然而，该框架未能提供分析除上下文长度外哪些因素影响模型表现的方法。为此，我们开发了一个名为DENIAHL（面向大模型的NIAH数据评估）的合成基准，系统评估数据类型、大小和模式等特征对模型NIAH能力的影响。研究发现，GPT-3.5与LLaMA 2-7B在不同特征下的表现存在显著差异，当项大小增加时召回性能明显下降，且在数据类型由数字转为字母时也出现部分性能退化。这表明，对于越来越大的上下文模型，除了项目数量外，其他特征同样显著影响其定位能力。
原文摘要 · Abstract (English)
The Needle-in-a-haystack (NIAH) test is a general task used to assess language models' (LMs') abilities to recall particular information from long input context. This framework however does not provide a means of analyzing what factors, beyond context length, contribute to LMs' abilities or inabilities to separate and recall needles from their haystacks. To provide a systematic means of assessing what features contribute to LMs' NIAH capabilities, we developed a synthetic benchmark called DENIAHL (Data-oriented Evaluation of NIAH for LLM's). Our work expands on previous NIAH studies by ablating NIAH features beyond typical context length including data type, size, and patterns. We find stark differences between GPT-3.5 and LLaMA 2-7B's performance on DENIAHL, and drops in recall performance when features like item size are increased, and to some degree when data type is changed from numbers to letters. This has implications for increasingly large context models, demonstrating factors beyond item-number impact NIAH capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。