让AI模型自动满足复杂约束,提升部署可靠性。
CAPE: Capability Achievement via Policy Execution
- 用可执行规范替代人工标注,通过四步循环训练模型。
- 在6个领域10万+样本中,违规率降低81%,误差小于0.3%。
- 适合需高可靠性的AI系统,如医疗、金融等关键场景。
现代AI系统缺乏表达和强制执行需求的机制。预训练生成智能,后训练优化偏好,但二者均无法确保模型可靠满足显式、上下文相关的约束。这一缺失抽象解释了为何高智能模型在部署时常失败,尽管基准表现优异。我们提出能力工程,即系统性地将需求转化为可执行规范,并训练模型默认满足这些规范。通过CAPE(Capability Achievement via Policy Execution)协议实现该实践,其包含指定→验证→修正→训练的循环。CAPE基于两项实证发现:(1) 上下文客观性——主观属性在上下文固定后变为客观(标注者一致性从kappa=0.42升至kappa=0.98);(2) 验证精度随模型规模上升(r=0.94),而偏好一致率在30%~50%间趋于饱和,不受算力影响。在六个领域共109,500个示例上,CAPE相比DPO将违规率降低81%(标准差低于0.3%)。通过用可复用规范替代逐例标注,成本降低5到20倍,周期从月缩短至周。我们开源了CAPE协议、PredicateGraph模式、CPL规范语言及策略包(Apache 2.0许可),并推出CapabilityBench,一个由社区贡献政策驱动的公开评估注册表,推动评估从智能基准转向能力度量。
原文摘要 · Abstract (English)
Modern AI systems lack a way to express and enforce requirements. Pre-training produces intelligence, and post-training optimizes preferences, but neither guarantees that models reliably satisfy explicit, context-dependent constraints. This missing abstraction explains why highly intelligent models routinely fail in deployment despite strong benchmark performance. We introduce Capability Engineering, the systematic practice of converting requirements into executable specifications and training models to satisfy them by default. We operationalize this practice through CAPE (Capability Achievement via Policy Execution), a protocol implementing a Specify -> Verify -> Correct -> Train loop. CAPE is grounded in two empirical findings: (1) contextual objectivity, where properties appearing subjective become objective once context is fixed (inter-annotator agreement rises from kappa = 0.42 to kappa = 0.98), and (2) verification-fidelity scaling, where verification accuracy improves with model scale (r = 0.94), unlike preference agreement which plateaus at 30 to 50 percent disagreement regardless of compute. Across 109,500 examples in six domains, CAPE reduces violation rates by 81 percent relative to DPO (standard deviation less than 0.3 percent). By replacing per-example annotation with reusable specifications, CAPE reduces costs by 5 to 20 times and shortens timelines from months to weeks. We release the CAPE protocol, PredicateGraph schema, CPL specification language, and policy packs under Apache 2.0. We also launch CapabilityBench, a public registry of model evaluations against community-contributed policies, shifting evaluation from intelligence benchmarks toward capability measurement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。