探究大模型如何利用上下文示例进行回归，发现其机制在记忆检索与学习之间动态切换。
Learning vs Retrieval: The Role of In-Context Examples in Regression with Large Language Models
- 通过回归任务设计实验，区分模型是回忆内部知识还是从上下文学习。
- 模型在有先验知识时更依赖记忆，示例丰富时则更多学习新信息。
- 适用于需要优化提示工程的机器学习研究者和实际应用开发者。
生成式大语言模型具备上下文学习能力，但其内在机制仍不明确，现有实验结果也常不一致。本文提出一种评估上下文学习机制的框架，聚焦于回归任务，认为该机制是内部知识检索与从上下文示例中学习的结合。首先证明大模型可解决真实世界回归问题，并设计实验量化模型依赖内部知识或从上下文学习的程度。我们提出这一过程存在于两个极端之间的连续谱上。深入分析了不同因素（如任务先验知识、上下文示例的类型与丰富度）对两种机制触发程度的影响。采用三种大模型和多个数据集验证结论的稳健性。结果为如何根据具体问题设计提示以促进元学习或知识检索提供了指导。
原文摘要 · Abstract (English)
Generative Large Language Models (LLMs) are capable of being in-context learners. However, the underlying mechanism of in-context learning (ICL) is still a major research question, and experimental research results about how models exploit ICL are not always consistent. In this work, we propose a framework for evaluating in-context learning mechanisms, which we claim are a combination of retrieving internal knowledge and learning from in-context examples by focusing on regression tasks. First, we show that LLMs can solve real-world regression problems and then design experiments to measure the extent to which the LLM retrieves its internal knowledge versus learning from in-context examples. We argue that this process lies on a spectrum between these two extremes. We provide an in-depth analysis of the degrees to which these mechanisms are triggered depending on various factors, such as prior knowledge about the tasks and the type and richness of the information provided by the in-context examples. We employ three LLMs and utilize multiple datasets to corroborate the robustness of our findings. Our results shed light on how to engineer prompts to leverage meta-learning from in-context examples and foster knowledge retrieval depending on the problem being addressed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。