揭示上下文如何影响大模型推理误差，提出可解释的理论框架。
When Does Context Help? Error Dynamics of Contextual Information in Large Language Models
- 用误差动态分析上下文对大模型的影响，拆解为基准误差与修正项。
- 上下文修正项需与负误差方向对齐且满足范数约束，才能有效降错。
- 理论指导上下文选择，实验提升性能0.6%，适合模型优化研究者。
推理时的上下文信息（如示范、检索知识或交互历史）可在不更新参数的情况下显著提升大语言模型性能，但其理论机制仍不清楚，尤其在提示学习等特定场景之外。本文提出统一的理论框架，用于分析基于Transformer的大模型中任意上下文信息的影响。通过输出误差动态分析，我们发现单层Transformer中，上下文条件误差向量可分解为基线误差向量与上下文修正向量之和。该分解揭示了误差降低的几何必要条件：上下文修正向量必须与负基线误差对齐，并满足范数约束。进一步证明，上下文修正范数存在由上下文-查询相关性与互补性决定的显式上界。这些结果可推广至多上下文和多层Transformer。在提示学习、检索增强生成及记忆演化任务中的实验验证了理论预测，并启发了一种有原则的上下文选择策略，使性能提升0.6%。
原文摘要 · Abstract (English)
Contextual information at inference time, such as demonstrations, retrieved knowledge, or interaction history, can substantially improve large language models (LLMs) without parameter updates, yet its theoretical role remains poorly understood beyond specific settings such as in-context learning (ICL). We present a unified theoretical framework for analyzing the effect of arbitrary contextual information in Transformer-based LLMs. Our analysis characterizes contextual influence through output error dynamics. In a single-layer Transformer, we prove that the context-conditioned error vector decomposes additively into the baseline error vector and a contextual correction vector. This yields necessary geometric conditions for error reduction: the contextual correction must align with the negative baseline error and satisfy a norm constraint. We further show that the contextual correction norm admits an explicit upper bound determined by context-query relevance and complementarity. These results extend to multi-context and multi-layer Transformers. Experiments across ICL, retrieval-augmented generation, and memory evolution validate our theory and motivate a principled context selection strategy that improves performance by $0.6\%$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。