用几何方法揭示LLM如何在上下文中学习概率密度估计
Density estimation with LLMs: a geometric investigation of in-context learning trajectories
- 通过InPCA可视化LLM在上下文中的学习轨迹
- 发现其轨迹与传统KDE明显不同，具自适应核宽度和形状
- 仅两个参数的自适应核模型即可捕捉多数行为，适合研究推理机制
大型语言模型（LLMs）在各类任务中展现出显著的上下文学习能力，包括时间序列预测。本文研究了LLMs从上下文数据中估计概率密度函数（PDF）的能力，这是许多概率建模问题的基础。我们利用密集主成分分析（InPCA）可视化并分析了LLaMA-2模型的上下文学习动态。主要发现是，这些模型在低维InPCA空间中均遵循相似的学习轨迹，与传统密度估计方法如直方图和高斯核密度估计（KDE）截然不同。我们将LLaMA的上下文密度估计过程解释为一种自适应核宽和形状的KDE。该自定义核模型尽管仅有两个参数，却能捕捉大部分LLaMA行为。我们进一步推测了其核特性与经典算法不同的原因，为理解LLM的上下文概率推理机制提供了洞见。代码库及3D可视化已公开于https://github.com/AntonioLiu97/LLMICL_inPCA。
原文摘要 · Abstract (English)
Large language models (LLMs) demonstrate remarkable emergent abilities to perform in-context learning across various tasks, including time series forecasting. This work investigates LLMs' ability to estimate probability density functions (PDFs) from data observed in-context; such density estimation (DE) is a fundamental task underlying many probabilistic modeling problems. We leverage the Intensive Principal Component Analysis (InPCA) to visualize and analyze the in-context learning dynamics of LLaMA-2 models. Our main finding is that these LLMs all follow similar learning trajectories in a low-dimensional InPCA space, which are distinct from those of traditional density estimation methods like histograms and Gaussian kernel density estimation (KDE). We interpret the LLaMA in-context DE process as a KDE with an adaptive kernel width and shape. This custom kernel model captures a significant portion of LLaMA's behavior despite having only two parameters. We further speculate on why LLaMA's kernel width and shape differs from classical algorithms, providing insights into the mechanism of in-context probabilistic reasoning in LLMs. Our codebase, along with a 3D visualization of an LLM's in-context learning trajectory, is publicly available at https://github.com/AntonioLiu97/LLMICL_inPCA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。