测试大模型编程语言选择偏好，发现多数模型盲目选Python
LangChoiceBench: Measuring and Explaining Programming-Language Choice in LLMs
- 构建项目级代码生成评测集，衡量语言偏好与一致性
- 25个模型中90%过度选择Python，小模型更严重
- 发现模型常凭直觉选语言，甚至编造理由支持错误选择
大型语言模型在生成项目级代码时表现出强烈的Python偏好，但目前缺乏系统性测量方法。为此，我们提出LangChoiceBench，一个用于评估Python偏好、推荐-实现一致性及语言多样性的项目级代码生成基准。该基准涵盖7个软件领域中的28个真实项目，其中Python常非最优选择。我们评估了25个不同模型，发现Python仍被严重高选，推荐与实现的一致性较低，且小型开源模型普遍表现出更强的Python偏好和更低的语言多样性。进一步分析9,826条推理轨迹发现，多数Python选择是自动或基于便捷性，而非项目需求；少数情况下，模型会虚构上下文支持选择Python（称为‘幻觉证据’），或生成与自身推理矛盾的代码。
原文摘要 · Abstract (English)
Large language models (LLMs) have been shown to exhibit strong Python preferences when generating project-level code, but there is currently no systematic way to measure this behaviour across new models. To bridge this gap, we introduce LangChoiceBench, a project-level code-generation benchmark for measuring Python preference, recommendation-implementation consistency, and language diversity. LangChoiceBench covers 28 projects across seven software areas where Python is often a poor default. We evaluate 25 diverse LLMs and find that Python remains heavily over-selected, recommendation-implementation consistency is low, and smaller open-weight models generally show stronger Python preference and lower language diversity. We further analyse 9,826 reasoning traces and find that most Python choices are automatic or driven primarily by ease, rather than explicit consideration of project requirements. In a smaller but important set of cases, models fabricate contextual support for choosing Python - a failure mode we call phantom evidence - or produce code that contradicts the language selected in their own reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。