arXiv:2603.14501cs.SEcs.AI2026-03中稿 · ESEM 2026

评测大模型在低资源通用编程语言上的表现,发现语法约束生成最划算。

CangjieBench: Benchmarking LLMs on a Low-Resource General-Purpose Programming Language

  • 构建无污染的CangjieBench基准,涵盖248个手工翻译样本。
  • 语法约束生成在准确率与成本间取得最佳平衡,优于直接生成。
  • 代码转代码任务表现差于文本转代码,存在负迁移现象。

大型语言模型在高资源编程语言上表现优异,但在低资源语言上能力不足。现有研究多聚焦领域专用语言(DSL),而对数据稀缺的通用编程语言关注不足。为此,我们提出CangjieBench,一个针对代表性低资源通用语言Cangjie的无污染基准。该基准包含248个从HumanEval和ClassEval人工翻译的高质量样本,覆盖文本转代码与代码转代码两类任务。我们在四种设置下系统评估了多种LLM:直接生成、语法约束生成、检索增强生成(RAG)和智能体(Agent)。实验表明,直接生成表现差,语法约束生成在准确率与计算成本间平衡最优;智能体达当前最佳准确率但消耗大量令牌。此外,代码转代码任务常劣于文本转代码生成,暗示模型可能过度拟合源语言模式,导致负迁移。我们希望本工作为大模型泛化至未见且低资源编程语言提供重要洞见。代码与数据已公开于https://github.com/cjhCoder7/CangjieBench。

原文摘要 · Abstract (English)

Large Language Models excel in high-resource programming languages but struggle with low-resource ones. Existing research related to low-resource programming languages primarily focuses on Domain-Specific Languages (DSLs), leaving general-purpose languages that suffer from data scarcity underexplored. To address this gap, we introduce CangjieBench, a contamination-free benchmark for Cangjie, a representative low-resource general-purpose language. The benchmark comprises 248 high-quality samples manually translated from HumanEval and ClassEval, covering both Text-to-Code and Code-to-Code tasks. We conduct a systematic evaluation of diverse LLMs under four settings: Direct Generation, Syntax-Constrained Generation, Retrieval-Augmented Generation (RAG), and Agent. Experiments reveal that Direct Generation performs poorly, whereas Syntax-Constrained Generation offers the best trade-off between accuracy and computational cost. Agent achieve state-of-the-art accuracy but incur high token consumption. Furthermore, we observe that Code-to-Code translation often underperforms Text-to-Code generation, suggesting a negative transfer phenomenon where models overfit to the source language patterns. We hope that our work will offer valuable insights into LLM generalization to unseen and low-resource programming languages. Our code and data are available at https://github.com/cjhCoder7/CangjieBench.

大模型编程语言低资源基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。