研究配置如何影响智能编程工具的自研或调用决策。
The Impact of Configuring Agentic AI Coding Tools on Build-vs-Buy Decisions: A Study Protocol
- 通过控制配置参数测试两个工具的自研与调用行为。
- 发现权限控制和技能指令最显著影响库选择结果。
- 适合关注AI编程安全与合规的开发者和研究者。
自主性不断增强的智能编程工具在生成代码时,需决定何时引入外部库或自行实现功能,这一“自研还是调用”(build-vs-buy)决策直接影响软件安全性、许可合规性、性能及长期可维护性。然而,目前尚无受控实验研究这些决策的形成机制。配置机制(如上下文文件、显式禁止、技能指令、MCP库发现工具、权限控制等)是开发者影响工具行为的主要手段,但其有效性尚不明确。本研究提出预注册协议,通过基准化分阶段项目任务,在Claude Code与OpenAI Codex中系统性地操控配置,测量工具选择的库、是否披露新引入库及其披露的完整性和准确性。九项预注册假设指导实验设计。最终将发布可复用的基准数据集与分析流程,供评估智能编程工具的build-vs-buy行为。
原文摘要 · Abstract (English)
Agentic AI coding tools write code with increasing autonomy and in doing so decide when to import a library and when to implement functionality from scratch. These decisions, whether to build functionality from scratch or buy into an external library, hereafter build-versus-buy, carry direct consequences for software security, licensing compliance, performance, and long-term maintainability. Yet no controlled experimental study has examined what governs build-versus-buy decisions in agentic AI coding tools. Configuration mechanisms, i.e., the means by which developers tailor agentic AI coding tool behavior to a project or workflow, are one of the primary means by which practitioners can influence these decisions. However, it is unclear which configuration mechanisms influence build-versus-buy decisions most effectively. We present a pre-registered protocol to study how configuration mechanisms alter build-versus-buy behavior in two popular agentic AI coding tools: Claude Code and OpenAI Codex. We will execute controlled programming tasks drawn from a benchmark of staged projects, each constructed around identifiable build-versus-buy points, and will manipulate the configuration supplied to each tool, ranging from no configuration, through context files with soft preferences and explicit prohibitions, to Skills (instructions that can be autonomously discovered), MCP-enabled library discovery tools, and permission controls, measuring which libraries the tool selects, whether it discloses newly introduced libraries, and whether those disclosures are complete and accurate. Nine pre-registered hypotheses structure the protocol. The resulting benchmark dataset and analysis pipeline will be released as a reusable artifact for evaluating build-versus-buy behavior in agentic AI coding tools.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。