解释概念式上下文学习为何有效，揭示模型如何利用少量示例推理。
On Theoretical Interpretations of Concept-Based In-Context Learning
- 基于概念的上下文学习理论分析，揭示其工作机制。
- 量化模型可利用的知识量，提出演示与查询的相似性度量。
- 适用于大模型预训练和提示工程优化，对实际应用有指导意义。
上下文学习（ICL）已成为自然语言处理和大语言模型（LLM）应用的重要范式，但其理论理解仍不充分。本文聚焦一种特定的ICL方法——概念式上下文学习（CB-ICL），通过理论分析揭示其在仅含少量示例的提示中预测查询标签的有效性原理。所提理论不仅解释了CB-ICL在何种条件下表现良好，还量化了大语言模型可借助于提示任务的知识量，并构建了演示样本与查询输入间的相似性度量，为模型预训练和提示工程提供重要启示。此外，理论还探讨了提示示例数量及模型嵌入维度对ICL性能的影响。最后，多个真实数据集实验验证了CB-ICL及其理论的实际有效性。
原文摘要 · Abstract (English)
In-Context Learning (ICL) has emerged as an important new paradigm in natural language processing and large language model (LLM) applications. However, the theoretical understanding of the ICL mechanism remains limited. This paper aims to investigate this issue by studying a particular ICL approach, called concept-based ICL (CB-ICL). In particular, we propose theoretical analyses on applying CB-ICL to ICL tasks, which explains why and when the CB-ICL performs well for predicting query labels in prompts with only a few demonstrations. In addition, the proposed theory quantifies the knowledge that can be leveraged by the LLMs to the prompt tasks, and leads to a similarity measure between the prompt demonstrations and the query input, which provides important insights and guidance for model pre-training and prompt engineering in ICL. Moreover, the impact of the prompt demonstration size and the dimension of the LLM embeddings in ICL are also explored based on the proposed theory. Finally, several real-data experiments are conducted to validate the practical usefulness of CB-ICL and the corresponding theory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。