为AI代理动作设计可自进化信任层，区分语义与语法威胁并动态优化判断。
AgentTrust: A Self-Improving Trust Layer for AI-Agent Actions
- 按威胁类型分治：语法威胁用规则，语义威胁用大模型判断。
- 自学习使规则准确率从48%提升至85%，且零误拦良性操作。
- 适合需要高安全性和低误判的自动化系统部署者使用。
AI代理执行关键操作（如命令行、云服务调用、工具调用），需可信层判断是否允许、警告、阻断或升级。我们提出按威胁类型分类的思路：固定签名的语法威胁可用确定性规则解决；而依赖意图的语义威胁（表面相同但行为不同）无法通过规则识别。实验证明，人工编写的云规则包在未见数据上准确率仅48%~56%，语义类别无提升（data_db 29→29，observability 59→59，supply_chain 50→50），而强语言模型判断器可精准处理这些类别。引入自学习机制后，在以语义攻击为主的语料上，规则准确率升至83.6%~85.2%，误拦率接近零，跨两家模型提供商均有效。构建双存储自进化系统：模型将语法威胁提炼为低成本规则，语义威胁通过带校验的RAG记忆缓存处理（防表面孪生问题，准确率从70%提至84%）。端到端在线回放显示，判断调用率从50%降至44%，领域准确率从71%升至80%，45,000次操作中无一次良性操作被硬阻断。
原文摘要 · Abstract (English)
AI agents increasingly take consequential actions -- shell commands, cloud operations, and arbitrary tool-calls -- so a trust layer must decide, per action, whether to allow, warn, block, or escalate. We argue that the right way to reason about such a layer is by threat type. Lexical (fixed-signature) threats, where danger lives in a stable token, are decidable by deterministic rules; semantic (intent-dependent) threats, where a benign and a malicious action share the same surface, are out of reach for rules by construction. We make this concrete with a negative proof: a determined, hand-authored cloud rule pack lifts held-out accuracy only 48 to 56% overall and moves the semantic categories by 0pp (data_db 29 to 29, observability 59 to 59, supply_chain 50 to 50), while a strong LLM judge carries exactly those categories. We give the judge a self-learning capability: on a corpus that is mainly semantic attacks it nearly doubles rule accuracy (48% to 83.6-85.2%) with near-zero false-blocks, and this holds across two model providers. We turn this into a self-improving dual-store system: the judge distills a growing deterministic rule floor on lexical threats (cheaper over time) and feeds a guarded RAG memory on semantic threats (a verdict-cache fails -- surface-twins collapse to ~58% -- so a corroboration guard lifts semantic accuracy +13pp, 70 to 84). The result is what sets AgentTrust v2 apart from its static v1 predecessor: a trust layer that self-evolves from its own stream of decisions -- cheaper on the lexical class (it distils its own rules) and smarter on the semantic class (it accrues guarded precedent), while never hard-blocking a benign action. An end-to-end online replay shows the judge-call rate falling (50% to 44%) and judge-domain accuracy rising (71% to 80%), with 0 benign hard-blocks across 45,000 actions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。