用编译器反馈引导，GPT-5在低资源语言Idris中解题率从22%提升至96%。
Compiler-Guided Inference-Time Adaptation: Improving GPT-5 Programming Performance in Idris
- 通过编译错误反馈构建迭代优化循环，指导模型逐步改进代码。
- 在Idris上解题成功率从22/56提升至54/56，接近全对。
- 适合关注LLM在冷门语言中应用的开发者与研究者。
GPT-5在主流编程语言如Python、C++和Java中表现优异，但在低资源或小众语言中的能力尚不明确。本文研究其在功能型语言Idris中的适应性，基于Exercism平台评估。零样本提示下，模型仅解决22/56道习题，远低于Python（45/50）和Erlang（35/47）。测试多种优化策略：基于平台反馈的迭代提示、加入文档与错误分类指南、以及结合本地编译错误和失败测试用例的迭代提示。其中，使用编译错误反馈的策略效果最佳，使解题率提升至54/56。结果表明，尽管大模型初始在低资源语言中表现不佳，但结构化编译器级反馈可显著激活其潜力。
原文摘要 · Abstract (English)
GPT-5, a state of the art large language model from OpenAI, demonstrates strong performance in widely used programming languages such as Python, C++, and Java; however, its ability to operate in low resource or less commonly used languages remains underexplored. This work investigates whether GPT-5 can effectively acquire proficiency in an unfamiliar functional programming language, Idris, through iterative, feedback driven prompting. We first establish a baseline showing that with zero shot prompting the model solves only 22 out of 56 Idris exercises using the platform Exercism, substantially underperforming relative to higher resource languages (45 out of 50 in Python and 35 out of 47 in Erlang). We then evaluate several refinement strategies, including iterative prompting based on platform feedback, augmenting prompts with documentation and error classification guides, and iterative prompting using local compilation errors and failed test cases. Among these approaches, incorporating local compilation errors yields the most substantial improvements. Using this structured, error guided refinement loop, GPT-5 performance increased to an impressive 54 solved problems out of 56. These results suggest that while large language models may initially struggle in low resource settings, structured compiler level feedback can play a critical role in unlocking their capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。