大模型能理解代码逻辑却不会写新语言，暴露了算法与实现间的鸿沟。
Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Language
- 用全新编程语言PyLang测试模型，发现模型仅学得语法而不懂语义。
- 在352道题中，模型在新语言上表现比Python低19%，且无法通过调优弥补。
- 模型80%能想出正确算法，但无法转化为可运行的代码，适合研究模型泛化能力者看。
大语言模型在代码生成基准上表现优异，但其能否迁移到预训练中未出现的语言仍不清楚。我们引入了从未出现在任何预训练语料中的极简命令式语言PyLang，对前沿模型Qwen3（4B、8B、32B）进行零样本和微调评估，共涵盖352个问题。结果表明，微调虽能快速掌握语法，但无法实现语义迁移：在所有配置下，模型在PyLang上的表现比Python最高低19%，且多任务学习、偏好微调、代码补全或隐空间目标等干预手段均未能缩小差距。一个LLM裁判显示，模型80%的时间会选择与Python相同的算法，却无法将其转换为有效的PyLang实现。CKA分析证实，微调后模型在内部表示上趋于高度一致（CKA > 0.97），但在输出阶段仍明显分化。我们称此为‘实现保真度缺口’：模型具备跨语言的算法理解能力，却无法在陌生语言中表达出来。研究强调需发展将推理与语言特异性实现解耦的训练方法。
原文摘要 · Abstract (English)
Large language models (LLMs) achieve high pass rates on code generation benchmarks, yet whether they can transfer this ability to languages absent from pretraining remains poorly understood. We introduce PyLang, a minimal imperative language absent from all pretraining corpora, and evaluate frontier models zero-shot and fine-tuned Qwen3 (4B, 8B, 32B) on 352 problems. We find that fine-tuning quickly teaches syntax but fails to transfer semantic competence: Python outperforms PyLang by up to 19% across all configurations, and no intervention (multi-task learning, preference tuning, code infilling, or latent-space objectives) closes the gap. An LLM judge reveals that frontier models select an identical algorithm to Python 80% of the time, yet cannot translate it into a working PyLang implementation., and CKA analysis confirms that fine-tuned models converge to nearly identical internal representations across languages (CKA > 0.97) while diverging at the output stage. We term this the implementation fidelity gap: models possess language-agnostic algorithmic understanding but cannot express it in an unfamiliar language. Our findings highlight the need for training methods that decouple reasoning from language-specific realization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。