提升代码模型长序列理解能力，突破上下文长度限制
An Evaluation of Context Length Extrapolation in Long Code via Positional Embeddings and Efficient Attention
- 改进位置编码与注意力机制，实现零样本推理扩展
- 在长代码补全任务中验证方法有效性，支持更长上下文处理
- 适合需要处理长代码逻辑的开发者工具研发者
大语言模型（LLMs）在软件工程中的自动化工具应用迅速发展，可执行代码生成、补全和翻译等任务。然而，其性能受限于固定的上下文长度，难以泛化到长且领域特定的代码序列。为解决此问题，本文研究了无需训练、仅在推理阶段进行优化的方法，聚焦于改进位置编码与注意力机制。目标是全面分析现有技术在代码场景下实现上下文长度外推的能力，特别是在长代码补全任务中的表现。
原文摘要 · Abstract (English)
The rapid advancement of large language models (LLMs) has led to a significant increase in automated tools in the software engineering, capable of performing various code-related tasks such as code generation, completion, and translation. Despite these advancements, its effectiveness is constrained by fixed context lengths, limiting its ability to generalize across long, domain-specific code sequences. To address this challenge, we investigate zero-shot, inference-only methods aimed at improving position encodings and optimizing attention mechanisms. Our goal is to provide a thorough analysis of current approaches that facilitate context length extrapolation in code, particularly in the context of long code completion tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。