测试450个AI生成代码，发现近半数存在隐蔽安全漏洞。
Is Vibe Coding the Future? An Empirical Assessment of LLM Generated Codes for Construction Safety
- 用自然语言指令让大模型写工地安全代码，评估其可靠性。
- 45%代码编译通过但逻辑错误，GPT-4o-Mini有56%数学出错。
- 非正式提示易诱发虚构安全变量，需严格管控和人工审核。
Vibe coding 指非技术人员用自然语言指令让大模型生成可执行代码，为建筑行业带来便利但也潜藏风险。本研究实证评估了三款前沿模型（Claude 3.5 Haiku、GPT-4o-Mini、Gemini 2.5 Flash）生成的450个工地安全相关Python脚本，采用150条角色驱动的提示语和双阶段评估流程（隔离动态沙箱+大模型判官）。结果表明，用户角色显著影响数据幻觉：非正式提示使模型更易虚构缺失的安全变量。尽管约85%代码能正常运行，但其中高达~45%存在隐蔽失败（silent failures），GPT-4o-Mini在成功执行的代码中数学错误率高达~56%。当前大模型缺乏独立支撑安全工程所需的确定性，必须通过确定性AI封装与严格治理机制才能用于网络物理系统部署。
原文摘要 · Abstract (English)
The emergence of vibe coding, a paradigm where non-technical users instruct Large Language Models (LLMs) to generate executable codes via natural language, presents both significant opportunities and severe risks for the construction industry. While empowering construction personnel such as the safety managers, foremen, and workers to develop tools and software, the probabilistic nature of LLMs introduces the threat of silent failures, wherein generated code compiles perfectly but executes flawed mathematical safety logic. This study empirically evaluates the reliability, software architecture, and domain-specific safety fidelity of 450 vibe-coded Python scripts generated by three frontier models, Claude 3.5 Haiku, GPT-4o-Mini, and Gemini 2.5 Flash. Utilizing a persona-driven prompt dataset (n=150) and a bifurcated evaluation pipeline comprising isolated dynamic sandboxing and an LLM-as-a-Judge, the research quantifies the severe limits of zero-shot vibe codes for construction safety. The findings reveal a highly significant relationship between user persona and data hallucination, demonstrating that less formal prompts drastically increase the AI's propensity to invent missing safety variables. Furthermore, while the models demonstrated high foundational execution viability (~85%), this syntactic reliability actively masked logic deficits and a severe lack of defensive programming. Among successfully executed scripts, the study identified an alarming ~45% overall Silent Failure Rate, with GPT-4o-Mini generating mathematically inaccurate outputs in ~56% of its functional code. The results demonstrate that current LLMs lack the deterministic rigor required for standalone safety engineering, necessitating the adoption of deterministic AI wrappers and strict governance for cyber-physical deployments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。