arXiv:2601.04540cs.SEcs.AI2026-01中稿 · ASE 2025被引 2

首个评估大模型代码片段适配能力的基准,填补了实践应用空白。

AdaptEval: A Benchmark for Evaluating Large Language Models on Code Snippet Adaptation

  • 基于开发者真实场景设计任务,保留上下文信息
  • 支持多粒度标注与细粒度测试,覆盖不同适配需求
  • 揭示大模型指令遵循能力不足,适合研究代码生成与工程应用

大语言模型在软件工程任务中取得进展,但针对代码复用中的关键环节——代码片段适配,尚无专门评估基准,导致其实际效用不明确。为此,我们提出 AdaptEval,一个面向代码片段适配的大模型评估基准。该基准具有三大特性:第一,实用上下文——任务源自 Stack Overflow 与 GitHub 社区的实际开发行为,保留丰富上下文;第二,多粒度标注——每个任务同时标注任务级与适配级需求,支持多样化适配场景评估;第三,细粒度评估——采用两级测试框架,结合适配级与函数级测试,可衡量模型在各类具体适配中的表现。基于 AdaptEval,我们首次对六种指令微调的 LLM 及三种推理型 LLM 进行实证评估。结果表明,AdaptEval 能从多角度评估模型适配能力,并揭示其在遵循显式指令方面的显著短板。我们希望该基准能推动大模型在代码适配能力上的进一步研究与提升,助力实际应用。

原文摘要 · Abstract (English)

Recent advancements in large language models (LLMs) have automated various software engineering tasks, with benchmarks emerging to evaluate their capabilities. However, for adaptation, a critical activity during code reuse, there is no benchmark to assess LLMs' performance, leaving their practical utility in this area unclear. To fill this gap, we propose AdaptEval, a benchmark designed to evaluate LLMs on code snippet adaptation. Unlike existing benchmarks, AdaptEval incorporates the following three distinctive features: First, Practical Context. Tasks in AdaptEval are derived from developers' practices, preserving rich contextual information from Stack Overflow and GitHub communities. Second, Multi-granularity Annotation. Each task is annotated with requirements at both task and adaptation levels, supporting the evaluation of LLMs across diverse adaptation scenarios. Third, Fine-grained Evaluation. AdaptEval includes a two-tier testing framework combining adaptation-level and function-level tests, which enables evaluating LLMs' performance across various individual adaptations. Based on AdaptEval, we conduct the first empirical study to evaluate six instruction-tuned LLMs and especially three reasoning LLMs on code snippet adaptation. Experimental results demonstrate that AdaptEval enables the assessment of LLMs' adaptation capabilities from various perspectives. It also provides critical insights into their current limitations, particularly their struggle to follow explicit instructions. We hope AdaptEval can facilitate further investigation and enhancement of LLMs' capabilities in code snippet adaptation, supporting their real-world applications.

代码生成大模型评估软件工程

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。