评测大模型在函数式编程中的表现，发现代码质量仍有显著提升空间。
Perish or Flourish? A Holistic Evaluation of Large Language Models for Code Generation in Functional Programming
- 构建涵盖三类主流函数式语言的721题基准测试集，评估代码正确性与风格。
- 模型在纯函数式语言中错误率远高于混合或命令式语言，且生成代码不规范。
- 通过静态分析反馈与人工指令，模型可部分自修复代码缺陷，提升可维护性。
函数式编程为构建可靠安全的软件系统提供坚实基础，但因其学习曲线陡峭而未广泛采用。大语言模型（LLMs）在代码生成上的进展为此提供了新契机。然而，现有评估主要聚焦于命令式编程语言，对函数式语言（FP）的能力仍缺乏深入探索。为此，我们提出FPEval，一个基于新基准FPBench的综合性评估框架，涵盖三种主流函数式语言（Haskell、Ocaml、Scala）共721个任务，分三个难度层级。FPEval整合了完备测试套件与静态分析工具，可评估代码功能正确性、风格及可维护性。我们评估了GPT-3.5、GPT-4o和GPT-5在函数式语言及Java（命令式基线）中的表现。结果表明，随着模型演进，其在函数式编程中的表现显著提升；但在纯函数式语言（如Haskell、OCaml）中错误率仍远高于混合型（Scala）或命令式（Java）语言。此外，模型常生成不符合函数式范式的代码，表现出命令式模式，影响代码风格与长期可维护性。最后，我们证明当提供静态分析反馈和针对常见问题的手写指令时，模型可部分实现自我修复，改善正确性与质量。
原文摘要 · Abstract (English)
Functional programming provides strong foundations for developing reliable and secure software systems, yet its adoption remains not widespread due to the steep learning curve. Recent advances in Large Language Models (LLMs) for code generation present new opportunities to lower these barriers. However, extensive evaluations of LLMs largely focus on imperative programming languages, and their capabilities in functional programming languages (FP) remain underexplored. To address this gap, we introduce FPEval, a holistic evaluation framework built on FPBench, a new benchmark of 721 programming tasks across three difficulty levels on three mainstream FP languages: Haskell, Ocaml and Scala. FPEval provides compehensive evaluation infrastructures with both test validations with comprehensive test suites and static analysis tools to assess both functional correctness and code style and maintainability. Using this framework, we evaluate state-of-the-art LLMs, including GPT-3.5, GPT-4o, and GPT-5, for code generation in functional programming languages and Java as an imperative baseline. Our results demonstrate that LLM performance in functional programming improves substantially with model advancement; however, error rates remain significantly higher in purely functional languages (Haskell and OCaml) than in hybrid (Scala) or imperative (Java) languages. Moreover, LLMs frequently generate non-idiomatic functional code that follows imperative patterns, raising concerns about code style and long-term maintainability. Finally, we show that LLMs can partially self-repair both correctness and quality issues when provided with static analysis feedback and hand-crafted instructions for common types of issues.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。