检测大模型代码生成中对厂商生态的偏好，发现倾向性最高达39.2个百分点。
Do LLMs Favor Their Providers? Measuring Vertical Integration Bias in Code Generation
- 构建基准测试工具VIBench，评估代码生成对厂商生态的偏好。
- 直接生成时偏好最高+18.8个百分点，代理工作流下升至+39.2个百分点。
- 早期选择会持续影响后续文件，持久率达90.3%，适合关注模型公平性的开发者参考。
大型语言模型（LLMs）已深度融入软件开发，尤其在具备代理能力后更为显著。然而，多数前沿模型由特定厂商关联。这引发疑问：生成代码是否偏向其所属厂商生态，从而限制开发者选择并加剧对单一厂商的依赖？本文定义此现象为垂直整合偏差（VIB），提出VIBench基准，用于衡量20种可选软件集成场景下直接与代理式代码生成中的VIB。评估10个厂商关联模型与3个非关联对照模型，结果表明直接生成存在正向VIB，其中6个关联模型显著偏好，最高达+18.8个百分点；代理工作流进一步放大该偏差，最高达+39.2个百分点。此外，代理流程中早期选择的厂商生态偏好可在概念解耦的下游文件中持续存在，最高持久率达90.3%。这些发现强调在代码生成中测量和应对VIB的必要性，尤其是在代理能力日益普及的背景下。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have become an integral part of software development, especially with the advent of agentic capabilities. Yet, many frontier LLMs are affiliated with specific providers. This raises the question of whether generated code favors the provider's own ecosystem over comparable alternatives, potentially constraining developers' choices and increasing dependence on a single provider. We define this behavior as Vertical Integration Bias (VIB) and introduce \textsc{VIBench}, a benchmark for measuring VIB in direct and agentic code generation across $20$ provider-selectable software-integration scenarios. Evaluating $10$ frontier provider-affiliated models against $3$ non-affiliated controls, we find positive VIB in direct generation, with six of ten affiliated models showing statistically significant effects up to $+18.8$ percentage points (pp). Agentic workflows further amplify VIB, reaching $+39.2$ pp. Moreover, early affiliated-ecosystem choices in agentic workflows can persist into conceptually decoupled downstream files, with persistence as high as $90.3\%$. These findings underscore the need to measure and account for VIB in code generation, especially as agentic capabilities become more prevalent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。