用图信号处理分析大模型如何内部表征数值序列，发现其结构随上下文变长而系统性变化。
A Graph Signal Processing Perspective on Numerical Sequence Representations in LLM In-Context Learning
- 以注意力生成节点权重图，隐藏状态作为节点信号，构建图信号模型
- 简单输入产生全局连通、频谱集中信号，复杂输入则出现局部化、高频能量增强
- 揭示跨模型家族通用的数值上下文学习内部特征，适合研究模型内部机制者
预训练大语言模型（LLM）在将数值序列以文本形式序列化后，展现出上下文学习（ICL）能力。以往研究主要通过输出层面评估如预测误差来刻画这种数值推理，但对模型内部表示中数值信息如何组织仍缺乏理解。本文采用图信号处理视角：注意力机制在标记之间诱导出加权图，而标记的隐藏状态构成图上节点的信号。定量图谱诊断与定性标记-图可视化表明，随着上下文长度增加，表示会根据输入动态复杂度更清晰地区分。简单输入生成具有更强全局连通性的注意力图和更平滑、频谱集中的隐藏状态信号；复杂输入则导致更局部化的图结构和具有更广谱支持及更高频能量的信号。这些发现共同揭示了与数值上下文学习相关的系统性、依赖上下文的内部表征模式，且在不同模型族间具有普适性。
原文摘要 · Abstract (English)
Pretrained large language models (LLMs) have demonstrated in-context learning (ICL) capabilities for numerical inference over sequences serialized as text. Prior work has identified and characterized this form of numerical inference primarily through output-level evaluations such as prediction error. However, how numerical information is organized within LLM representations remains much less understood. To study this internal organization, we adopt a graph signal processing perspective in which attention induces a weighted graph over tokens, while token hidden states define signals on its nodes. Quantitative graph-spectral diagnostics and qualitative token-graph visualizations reveal that representations become more clearly differentiated by input dynamical complexity as context length increases. Simpler inputs produce attention-induced token graphs with stronger global connectivity and smoother, spectrally concentrated hidden-state signals, whereas more complex inputs produce more localized graphs and hidden-state signals with broader spectral support and greater high-frequency energy. Together, these findings point to systematic, context-dependent internal signatures associated with numerical ICL that are conserved across model families.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。