arXiv:2508.11824cs.SEcs.AI2025-08被引 8

为AI编程引入安全框架,防止代码生成出错与失控。

Rethinking Autonomy: Preventing Failures in AI-Driven Software Engineering

  • 提出SAFE-AI框架,通过沙箱和实时验证防范风险。
  • 发现大模型易生成漏洞代码,且缺乏回滚与审计机制。
  • 适合关注AI安全与合规的开发者与技术管理者。

大型语言模型(LLMs)在软件工程中的应用虽大幅提升编码效率,但也带来安全隐患:如生成不安全代码、虚构输出、不可逆操作及透明度不足。Replit数据库删除事件凸显了建立可靠安全与治理机制的紧迫性。本文系统分析了LLM辅助编码的固有挑战,包括漏洞继承、过度信任、误解及缺乏标准化验证与回滚协议。为此,提出SAFE-AI框架,强调安全、可审计性、反馈与可解释性,集成防护机制、沙箱环境、运行时验证、风险感知日志、人机协同系统及可解释性技术,以降低风险并增强信任。提出新型行为分类法,将AI行为分为提示型、生成型、自主型与破坏型,助力风险评估与监管。同时指出当前缺乏针对代码幻觉与自主性水平的标准基准,并呼吁开展混合验证、语义防护与主动治理工具的研究。通过对比自主控制、提示工程、可解释性与治理框架,本文为符合欧盟《人工智能法案》与加拿大AIDA等法规的负责任AI开发提供路线图。

原文摘要 · Abstract (English)

The integration of Large Language Models (LLMs) into software engineering has revolutionized code generation, enabling unprecedented productivity through promptware and autonomous AI agents. However, this transformation introduces significant risks, including insecure code generation, hallucinated outputs, irreversible actions, and a lack of transparency and accountability. Incidents like the Replit database deletion underscore the urgent need for robust safety and governance mechanisms. This paper comprehensively analyzes the inherent challenges of LLM-assisted code generation, such as vulnerability inheritance, overtrust, misinterpretation, and the absence of standardized validation and rollback protocols. To address these, we propose the SAFE-AI Framework, a holistic approach emphasizing Safety, Auditability, Feedback, and Explainability. The framework integrates guardrails, sandboxing, runtime verification, risk-aware logging, human-in-the-loop systems, and explainable AI techniques to mitigate risks while fostering trust and compliance. We introduce a novel taxonomy of AI behaviors categorizing suggestive, generative, autonomous, and destructive actions to guide risk assessment and oversight. Additionally, we identify open problems, including the lack of standardized benchmarks for code specific hallucinations and autonomy levels, and propose future research directions for hybrid verification, semantic guardrails, and proactive governance tools. Through detailed comparisons of autonomy control, prompt engineering, explainability, and governance frameworks, this paper provides a roadmap for responsible AI integration in software engineering, aligning with emerging regulations like the EU AI Act and Canada's AIDA to ensure safe, transparent, and accountable AI-driven development.

AI安全代码生成治理框架

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。