arXiv:2502.01860cs.SEcs.LG2025-02被引 1

SWE-Arena让大模型在真实开发场景中接受多轮交互测试,评估更精准。

SWE-Arena: An Interactive Platform for Evaluating Foundation Models in Software Engineering

  • 构建可交互的评测平台,支持多轮对话与代码任务闭环
  • 提出一致性得分与对话效率指数,量化模型稳定性和沟通成本
  • 集成仓库上下文(RepoChat),贴近真实开发流程

基础模型(特别是大语言模型)在代码生成、调试和需求细化等软件工程任务中展现出巨大潜力。然而,现有评估框架难以有效衡量模型在迭代性、上下文丰富的开发工作流中的表现。为此,我们提出SWE-Arena——一个面向软件工程任务的交互式评测平台。该平台提供透明开源排行榜,支持多轮对话流程,实现端到端模型对比。引入两项新指标:模型一致性得分(通过自对战评估输出稳定性),以及对话效率指数(考虑达成结论所需交互轮数)。此外,平台新增RepoChat功能,自动注入仓库上下文(如问题、提交、拉取请求),使评估更贴近真实开发场景。本文阐述SWE-Arena的设计与能力,强调其在推动基础模型在软件工程中评估与应用方面的潜力。

原文摘要 · Abstract (English)

Foundation models (FMs), particularly large language models (LLMs), have shown significant promise in various software engineering (SE) tasks, including code generation, debugging, and requirement refinement. Despite these advances, existing evaluation frameworks are insufficient for assessing model performance in iterative, context-rich workflows characteristic of SE activities. To address this limitation, we introduce \emph{SWE-Arena}, an interactive platform designed to evaluate FMs in SE tasks. SWE-Arena provides a transparent, open-source leaderboard, supports multi-round conversational workflows, and enables end-to-end model comparisons. The platform introduces novel metrics, including \emph{model consistency score} that measures the consistency of model outputs through self-play matches, and \emph{conversation efficiency index} that evaluates model performance while accounting for the number of interaction rounds required to reach conclusions. Moreover, SWE-Arena incorporates a new feature called \emph{RepoChat}, which automatically injects repository-related context (e.g., issues, commits, pull requests) into the conversation, further aligning evaluations with real-world development processes. This paper outlines the design and capabilities of SWE-Arena, emphasizing its potential to advance the evaluation and practical application of FMs in software engineering.

模型评估代码生成交互评测LLM

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。