用多模型检测大模型编程代理提示词中的干扰问题
Arbiter: Detecting Interference in LLM Agent System Prompts
- 结合形式化规则与多模型扫描，自动发现提示词缺陷
- 在三个主流编码代理中发现152个异常模式，21个明确干扰点
- 揭示提示结构影响故障类型，适合安全审计与模型优化者
基于大语言模型的编程代理依赖系统提示词来控制行为，但这些提示词缺乏传统软件的测试机制。本文提出Arbiter框架，融合形式化评估规则与多模型大语言模型扫描，检测系统提示词中的干扰模式。针对Claude Code（Anthropic）、Codex CLI（OpenAI）和Gemini CLI（Google）三款主流编码代理系统提示词进行分析，在无监督扫描阶段发现152项异常，在有监督定向分析中确认21个手工标注的干扰模式。结果显示，提示词架构（单一、扁平、模块化）与故障类别强相关，但不影响严重程度；多模型评估比单模型能发现截然不同的漏洞类别。其中一项扫描发现的Gemini CLI记忆系统结构数据丢失问题，与谷歌已提交并修复的漏洞一致，但其补丁仅解决症状，未触及扫描器识别出的模式层面根本原因。跨厂商分析总成本仅为0.27美元。
原文摘要 · Abstract (English)
System prompts for LLM-based coding agents are software artifacts that govern agent behavior, yet lack the testing infrastructure applied to conventional software. We present Arbiter, a framework combining formal evaluation rules with multi-model LLM scouring to detect interference patterns in system prompts. Applied to three major coding agent system prompts: Claude Code (Anthropic), Codex CLI (OpenAI), and Gemini CLI (Google), we identify 152 findings across the undirected scouring phase and 21 hand-labeled interference patterns in directed analysis of one vendor. We show that prompt architecture (monolithic, flat, modular) strongly correlates with observed failure class but not with severity, and that multi-model evaluation discovers categorically different vulnerability classes than single-model analysis. One scourer finding was structural data loss in Gemini CLI's memory system was consistent with an issue filed and patched by Google, which addressed the symptom without addressing the schema-level root cause identified by the scourer. Total cost of cross-vendor analysis: \$0.27 USD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。