研究大模型选库选语言的偏好，发现它们偏爱热门工具而非最优选择。
A Study of LLMs' Preferences for Libraries and Programming Languages
- 通过实证分析8个大模型的代码生成偏好，考察其对编程语言和库的选择。
- 45%情况下过度使用NumPy等流行库，58%场景下错误坚持用Python而非更优语言。
- 揭示大模型倾向熟悉度而非任务适配性，提示需改进训练与评估标准。
尽管大语言模型在代码生成方面进展迅速，现有评估仍聚焦于功能正确性或语法有效性，忽略了模型在关键设计决策（如选择何种库或编程语言）上的表现。为此，我们首次对大模型在生成代码时对库和编程语言的偏好进行了实证研究，涵盖八种不同模型。结果发现，模型普遍存在过度使用广泛采用的库（如NumPy）的现象，在多达45%的情况下并非必要，且偏离真实解法。此外，所有模型均显著偏好使用Python作为默认语言。在高性能项目初始化任务中，当Python并非最优选择时，它仍被用于58%的案例，而Rust则一次未被使用。这些结果表明，大模型更倾向于选择熟悉和流行的工具，而非任务特定的最优解，凸显了针对性微调、数据多样性提升以及明确衡量语言与库选择准确性的评估基准的重要性。
原文摘要 · Abstract (English)
Despite the rapid progress of large language models (LLMs) in code generation, existing evaluations focus on functional correctness or syntactic validity, overlooking how LLMs make critical design choices such as which library or programming language to use. To fill this gap, we perform the first empirical study of LLMs' preferences for libraries and programming languages when generating code, covering eight diverse LLMs. We observe a strong tendency to overuse widely adopted libraries such as NumPy; in up to 45% of cases, this usage is not required and deviates from the ground-truth solutions. The LLMs we study also show a significant preference toward Python as their default language. For high-performance project initialisation tasks where Python is not the optimal language, it remains the dominant choice in 58% of cases, and Rust is not used once. These results highlight how LLMs prioritise familiarity and popularity over suitability and task-specific optimality; underscoring the need for targeted fine-tuning, data diversification, and evaluation benchmarks that explicitly measure language and library selection fidelity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。