代码翻译中，例子越多越差，好例子比多例子更重要
When Many-Shot Prompting Fails: An Empirical Study of LLM Code Translation
- 用不同数量示例测试大模型代码翻译能力
- 5到25个优质示例时准确率最高，625个反而下降
- 适合软件工程领域用大模型做代码转换的开发者
大型语言模型（LLMs）凭借庞大的上下文窗口，为上下文学习（ICL）提供了新路径，通常认为提供更多示例（多示例提示）能提升性能。我们针对复杂的代码翻译任务开展大规模实证研究，系统评估了从零样本到最多625个示例的多种提示配置，提示长度覆盖约10万至80万词元。研究发现存在“多示例悖论”：尽管静态相似性指标随示例增加略有提升，但功能正确率在少样本提示（5–25个示例）时达到峰值，提供大量示例反而会降低关键的功能性能。结果表明，在代码翻译中，少数精心选择的示例优于大量冗余示例，挑战了‘越多越好’的通用假设，强调了提示策略的任务依赖性。研究对有效利用大模型进行软件工程应用具有重要意义。
原文摘要 · Abstract (English)
Large Language Models (LLMs) with vast context windows offer new avenues for in-context learning (ICL), where providing many examples ("many-shot" prompting) is often assumed to enhance performance. We investigate this assumption for the complex task of code translation. Through a large-scale empirical study of over 90,000 translations, we systematically evaluate the impact of scaling in-context examples from zero-shot to many-shot configurations of up to 625 examples, with prompts spanning from approximately 100,000 to 800,000 tokens. Our findings reveal a "many-shot paradox": while static similarity metrics may modestly improve with more examples, functional correctness consistently peaks with few-shot prompting (5-25 examples). Providing substantially more examples often degrades this crucial functional performance. This study highlights that for code translation, the quality of a few well-chosen examples outweighs sheer quantity, challenging the universal efficacy of "more is better" for ICL and underscoring the task-dependent nature of optimal prompting strategies. Our results have significant implications for effectively leveraging LLMs in software engineering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。