用状态空间模型提升代码理解，更省显存且能处理更长代码。
CodeSSM: Towards State Space Models for Code Understanding
- 基于状态空间模型构建代码理解新架构，替代传统Transformer。
- 在2048上下文长度下显存减少64%，且越长越省。
- 样本效率高，能外推到比预训练更长的代码序列。
尽管变压器模型主导众多代码任务，但仍存在显著局限。本文探索状态空间模型（SSMs）作为代码理解任务（如检索、分类、克隆检测）的可行替代方案。我们提出CodeSSM，首个在代码语料上训练的SSM模型，用于评估其有效性。实验表明，SSMs具有更高的样本效率，且能外推至超出预训练长度的长上下文。大量实验显示，SSMs是Transformer的可行替代，有效缓解其若干缺陷。此外，与Transformer相比，CodeSSM在2048上下文长度下内存占用减少高达64%，且随着上下文增长，节省幅度进一步加大。
原文摘要 · Abstract (English)
Although transformers dominate many code-specific tasks, they have significant limitations. This paper explores State Space Models (SSMs) as a promising alternative for code understanding tasks such as retrieval, classification, and clone detection. We introduce CodeSSM, the first SSM-based model trained on code corpora to assess its effectiveness. Our results demonstrate that SSMs are more sample-efficient and can extrapolate to longer contexts beyond the pretraining length. Extensive experiments show that SSMs offer a viable alternative to transformers, addressing several their limitations. Additionally, CodeSSM reduces memory usage by up to 64\% compared to transformers at a context length of 2048, with greater savings as context length grows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。