arXiv:2512.23214cs.CLcs.LG2025-12被引 2

用专用语言Anka让大模型写代码更准,多步任务准确率提升40个百分点。

Anka: A Domain-Specific Language for Reliable LLM Code Generation

  • 设计约束性语法的专用语言Anka,减少代码歧义。
  • 在100个任务上,多步管道任务准确率达100%,比Python高40%。
  • 适合需要高可靠性的自动化数据处理场景。

大型语言模型(LLMs)在代码生成方面表现卓越,但在复杂的多步骤编程任务中仍存在系统性错误。我们假设这些错误源于通用编程语言的灵活性,导致多种有效解法和隐式状态管理需求。为此,我们提出Anka,一种专为数据转换管道设计的领域特定语言(DSL),采用显式且受限的语法以降低代码生成中的歧义。尽管从未在Anka上进行过训练,Claude 3.5 Haiku在100个基准问题上实现了99.9%的解析成功率和95.8%的整体任务准确率。关键是在多步流水线任务中,Anka实现100%准确率,相较Python的60%提升40个百分点;使用GPT-4o-mini验证也获得+26.7个百分点的提升。结果表明:(1) LLM可仅通过上下文提示学习新DSL并达到接近原生水平的准确率;(2) 受限语法显著降低复杂任务错误率;(3) 为LLM生成专门设计的领域特定语言,可超越其长期训练的通用语言。我们开源了完整语言实现、基准测试集与评估框架,推动后续研究。

原文摘要 · Abstract (English)

Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, yet they exhibit systematic errors on complex, multi-step programming tasks. We hypothesize that these errors stem from the flexibility of general-purpose languages, which permits multiple valid approaches and requires implicit state management. To test this hypothesis, we introduce Anka, a domain-specific language (DSL) for data transformation pipelines designed with explicit, constrained syntax that reduces ambiguity in code generation. Despite having zero prior training exposure to Anka, Claude 3.5 Haiku achieves 99.9% parse success and 95.8% overall task accuracy across 100 benchmark problems. Critically, Anka demonstrates a 40 percentage point accuracy advantage over Python on multi-step pipeline tasks (100% vs. 60%), where Python's flexible syntax leads to frequent errors in operation sequencing and variable management. Cross-model validation with GPT-4o-mini confirms this advantage (+26.7 percentage points on multi-step tasks). Our results demonstrate that: (1) LLMs can learn novel DSLs entirely from in-context prompts, achieving near-native accuracy; (2) constrained syntax significantly reduces errors on complex tasks; and (3) domain-specific languages purposefully designed for LLM generation can outperform general-purpose languages on which the LLM has extensive training. We release the complete language implementation, benchmark suite, and evaluation framework to facilitate further research.

代码生成领域语言大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。