arXiv:2604.11088cs.AIcs.CL2026-04被引 2

随机规则与专家规则效果相当,关键在禁止错误行为而非规定正确做法。

Guardrails Beat Guidance: A Large-Scale Study of Rules, Skills, and Persistent Configuration for Coding Agents

  • 用规则约束不能做什么比规定应该做什么更有效
  • 随机规则与专家规则在任务通过率上均提升13.8个百分点
  • 适合构建安全可靠的代码生成代理,尤其关注规则配置风险

通过首次大规模受控研究代理规则文件(包括CLAUDE.md、.cursorrules及各类技能、插件声明和角色定义),我们从GitHub爬取679个规则文件(共25,532条规则),并使用Claude Opus 4.6在SWE-bench Verified上进行超过5,000次代理运行。结果发现:(i)规则正负极性可清晰区分有益与有害规则;(ii)性能提升基本与内容无关,随机、打乱、跨域或格式不匹配的规则文件表现与定制规则相当,暗示上下文预热机制;(iii)单条规则常显负面,但在组合中累积影响不显著,通过率在0至50条规则间保持稳定。这些发现揭示了社区规则生态中的潜在可靠性风险,提出核心原则:应限制代理不应执行的行为,而非强制其遵循指令。

原文摘要 · Abstract (English)

Random rules improve a coding agent's task performance as much as expert-curated ones (both $+13.8$pp on a discriminative subset of SWE-bench Verified), and in our data every individually beneficial rule is a negative constraint ("do not refactor unrelated code"), while every individually harmful one is a positive directive ("follow code style"). We arrive at these findings through the first large-scale controlled study of agent rule files (\texttt{CLAUDE.md}, \texttt{.cursorrules}, and the broader family of agent skills, plugin manifests, and persona definitions): we scrape 679 rule files (25{,}532 rules) from GitHub and conduct over 5{,}000 agent runs of Claude Code with Claude Opus 4.6 on SWE-bench Verified. Three patterns emerge. (i) Rule polarity cleanly separates beneficial from harmful rules; we read this through the lens of potential-based reward shaping (PBRS). (ii) Performance gains are largely content-independent: random, shuffled, mismatched-domain, and unconverted-format rule files all match curated rules, pointing to a context priming mechanism. (iii) Individual rules often appear harmful in isolation yet do not visibly accumulate damage in ensemble: pass rates remain stable across rule counts from 0 to 50. These findings expose a hidden reliability risk in the rapidly growing ecosystem of community-authored rules and skills, and they yield a clear principle for safer agent configuration: constrain what agents must not do, rather than prescribing what they should.

代码生成规则系统代理安全

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。