研究大模型在长文本中性能如何随上下文变长而下降
Context Discipline and Performance Correlation: Analyzing LLM Performance and Quality Degradation Under Varying Context Lengths
- 分析密集Transformer模型在大量无关上下文下的表现变化
- 发现KV缓存增长导致性能非线性下降，尤其在长上下文时
- 揭示MoE架构在超长上下文下被硬件瓶颈掩盖优势
大型语言模型（LLMs）的扩展趋势倾向于增大最大上下文窗口，以支持复杂推理和长文档分析。然而，管理扩展后的上下文会带来严重的计算开销。本文研究了在面对大量无关和干扰性上下文时，密集Transformer架构（特别是Llama-3.1-70B和Qwen1.5-14B）在系统性能与模型质量之间的关键权衡。研究发现，随着键值（KV）缓存的增长，性能出现非线性退化。此外，对混合专家（MoE）架构的扩展分析揭示了不同上下文规模下的独特行为异常，表明在高令牌量下，架构优势可能被基础设施瓶颈所掩盖。
原文摘要 · Abstract (English)
The scaling trend in Large Language Models (LLMs) has prioritized increasing the maximum context window to facilitate complex, long-form reasoning and document analysis. However, managing this expanded context introduces severe computational overhead. This paper investigates the critical trade-off between system performance and model quality when dense transformer architectures--specifically Llama-3.1-70B and Qwen1.5-14B--are exposed to large volumes of irrelevant and distracting context. The research identifies a non-linear performance degradation tied to the growth of the Key-Value (KV) cache. Furthermore, an extended analysis of the Mixture-of-Experts (MoE) architecture reveals unique behavioral anomalies at varying context scales, suggesting that architectural benefits may be masked by infrastructure bottlenecks at high token volumes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。