分析代码模型中状态空间模型的学习除了更懂语法语义，还发现微调时会遗忘某些关系。
Towards Understanding What State Space Models Learn About Code
- 用频域方法揭示SMM在微调中趋向短距离依赖
- 预训练时比Transformer更有效捕捉代码结构
- 改进架构后在代码搜索任务上提升6 MRR
状态空间模型（SSMs）作为Transformer的高效替代方案，已在代码理解任务中表现不俗。然而其内部机制仍不清晰。本文首次系统分析了基于SSM的代码模型的学习特性，并与Transformer进行直接对比。结果显示，预训练阶段SSMs比Transformer更有效地捕捉代码的语法和语义结构，但在部分任务的微调过程中会遗忘某些关系。为此，我们提出SSM-Interpret——一种频域分析框架，揭示了微调时出现的短程依赖谱偏移现象。基于此发现，我们设计了架构改进，使基于SSM的代码模型在NLCodeSearch任务上的MRR最高提升6点。这表明我们的分析不仅解释了模型行为，还直接指导了更优模型设计。
原文摘要 · Abstract (English)
State Space Models (SSMs) have emerged as an efficient alternative to the Transformer architecture. Prior work shows that, when trained under comparable conditions, SSMs can match or surpass Transformers on code understanding tasks. However, their internal mechanisms remain a black box. We present the first systematic analysis of what SSM-based code models learn along with the direct comparison between SSM and Transformer models in this domain. Our analysis shows that SSMs capture syntactic and semantic structure more effectively than Transformers during pretraining but forgets certain relations during fine-tuning on some tasks. To investigate this behavior, we introduce SSM-Interpret, a frequency-domain framework that exposes a spectral shift toward short-range dependencies during fine-tuning. Guided by these findings, we propose architectural modifications that significantly improve the performance of SSM-based code model by upto +6 MRR on NLCodeSearch. This demonstrates that our analysis not only explains model behavior but also leads directly to better designs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。