用注意力机制精简代码，提速模型训练与推理。
LEANCODE: Understanding Models Better for Code Simplification of Pre-trained Large Language Models
- 根据上下文注意力得分选择性删除冗余代码token。
- 代码搜索任务提速60%，摘要任务提速29%以上。
- 适合需要高效推理的代码生成与理解场景。
用于代码的大语言模型通常计算复杂度高，且随输入代码长度显著增加。我们提出LeanCode方法，通过利用注意力分数表示词元重要性，实现代码简化以降低训练与预测时间。该方法基于上下文感知的平均注意力得分进行词元选择性移除，而非全局平均。在分类任务（如代码搜索）中，使用编码器中`CLS' token的注意力分数；在序列到序列任务（如代码摘要）中，采用编码器-解码器注意力分数判断词元重要性。实验表明，相比SOTA方法DietCode和Slimcode，LeanCode在代码搜索任务上分别提升60%和16%，在代码摘要任务上分别提升29%和27%。
原文摘要 · Abstract (English)
Large Language Models for code often entail significant computational complexity, which grows significantly with the length of the input code sequence. We propose LeanCode for code simplification to reduce training and prediction time, leveraging code contexts in utilizing attention scores to represent the tokens' importance. We advocate for the selective removal of tokens based on the average context-aware attention scores rather than average scores across all inputs. LeanCode uses the attention scores of `CLS' tokens within the encoder for classification tasks, such as code search. It also employs the encoder-decoder attention scores to determine token significance for sequence-to-sequence tasks like code summarization. Our evaluation shows LeanCode's superiority over the SOTAs DietCode and Slimcode, with improvements of 60% and 16% for code search, and 29% and 27% for code summarization, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。