针对大模型写Swift代码能力，构建首个专业评测集并发现小模型严重掉分。
SwiftEval: Developing a Language-Specific Benchmark for LLM-generated Code Evaluation
- 手工设计28道高精度Swift编程题，拒绝自动翻译
- 44个代码大模型在新基准上表现普遍下降，尤其小模型
- 适合评估多语言编程模型在特定语言上的真实能力
近年来，大语言模型在代码生成方面取得显著进展。然而，多数评测基准以Python为主，难以高质量评估其他语言如Swift的表现。通过分析HumanEval-XL和MultiPL-E等多语言基准，我们发现其Swift部分存在关键问题，导致评估结果不准确或无关。与现有依赖大模型自动翻译Python基准以快速扩展的方法不同，我们采用质量优先策略，提出首个面向Swift的评测集SwiftEval，包含28道精心设计的题目，并对44个主流代码大模型进行评测。结果表明，在涉及语言特性的任务中，模型得分明显下降，尤以小规模模型最为显著。
原文摘要 · Abstract (English)
In recent years, large language models (LLMs) have showcased significant advancements in code generation. However, most evaluation benchmarks are primarily oriented towards Python, making it difficult to evaluate other programming languages, such as Swift, with high quality. By examining widely established multilingual benchmarks like HumanEval-XL and MultiPL-E, we identified critical issues specific to their Swift components, making them insufficient or even irrelevant for assessing LLM coding capabilities on Swift. Unlike these existing approaches, which prioritize rapid scaling and generalization by automatically translating Python-centric benchmarks with LLMs, we adopt a quality-over-quantity methodology. We present SwiftEval, the first Swift-oriented benchmark consisting of 28 carefully hand-crafted problems, and evaluate 44 popular Code LLMs on it. Our results show significant LLM scores drop for problems requiring language-specific features, most noticeable in the models of smaller sizes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。