对比多个大模型在语法敏感代码补全上的表现，揭示准确率与速度的权衡。
Comparative Analysis of Large Language Models for Context-Aware Code Completion using SAFIM Framework
- 使用SAFIM数据集评估多款聊天类大模型的代码补全能力。
- GPT-4o在准确率上领先，但延迟较高；Gemini 1.5 Flash速度快但准确率较低。
- 为代码补全模型选型提供实证参考，适合开发者和研究者使用。
大型语言模型（LLMs）的出现彻底改变了代码补全技术，使其成为现代集成开发环境中的智能化、上下文感知功能。这些进展显著提升了开发者编写高效且无错误代码的能力。本研究采用语法感知的填空中间（SAFIM）数据集，评估了多种基于聊天的大模型（包括Gemini 1.5 Flash、Gemini 1.5 Pro、GPT-4o、GPT-4o-mini和GPT-4 Turbo）的性能。通过余弦相似度与真实补全结果的匹配度以及延迟等指标，衡量模型的准确性与效率。结果表明，各模型在代码补全能力上存在显著差异，揭示了其各自的优势与不足。该研究提供了具有参考价值的对比分析，突显了准确率与速度之间的权衡，为未来基于大模型的代码补全技术发展建立了基准。
原文摘要 · Abstract (English)
The advent of Large Language Models (LLMs) has revolutionized code completion, transforming it into a more intelligent and context-aware feature in modern integrated development environments. These advancements have significantly enhanced developers' ability to write efficient and error-free code. This study evaluates the performance of several chat-based LLMs, including Gemini 1.5 Flash, Gemini 1.5 Pro, GPT-4o, GPT-4o-mini, and GPT-4 Turbo, using the Syntax-Aware Fill-in-the-Middle (SAFIM) dataset. This benchmark is specifically designed to assess models' capabilities in syntax-sensitive code generation. Performance metrics, such as cosine similarity with ground-truth completions and latency, were employed to measure both accuracy and efficiency. The findings reveal substantial differences in the models' code completion abilities, offering valuable insights into their respective strengths and weaknesses. This work provides a comparative analysis that underscores the trade-offs between accuracy and speed, establishing a benchmark for future advancements in LLM-based code completion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。