检测大模型在长期使用中性格偏移，无需访问模型权重。
Nautilus Compass: Black-box Persona Drift Detection for Production LLM Agents
- 通过用户提示与行为锚点的文本相似度，实现黑盒检测。
- 在真实会话数据上达到0.83的ROC AUC，检索准确率达56.6%。
- 适用于闭源API（如Claude、GPT-4），适合生产环境部署。
生产环境中的大模型编程代理在长时间会话中会出现性格漂移：忘记用户设定约束、重复已标记错误、虚构先前协议。白盒方法如角色向量需访问模型权重，无法用于大多数用户使用的闭源API（如Claude、GPT-4）。我们提出Nautilus Compass，一种黑盒性格漂移检测器与代理记忆层，完全运行于提示文本层面：使用BGE-m3嵌入对用户提示与行为锚点文本进行余弦相似度计算，并通过加权top-k均值聚合。该系统是目前已知唯一不依赖索引时调用LLM提取事实或构建图谱的公开代理记忆层（对比Mem0、Letta、Cognee、Zep、MemOS、smrti，截至2026年5月验证）。原始对话文本直接嵌入，系统以Claude Code插件、MCP 2024-11-05 A2A服务器（支持Cursor、Cline、Hermes）、CLI和单个守护进程的REST API形式提供，并采用梅尔克链审计日志确保锚点更新可追溯。在基于真实Claude Code会话轨迹构建的保留测试集上，由独立LLM裁判标注，其漂移检测达ROC AUC 0.83。嵌入式检索管道在LongMemEval-S v0.8上得分56.6%，EverMemBench-Dynamic（n=500）上得44.4%，超越四个已发表基线。56.6%相较近期白盒领先者（90%+）低约30分，视为无提取设计的架构上限。端到端复现成本仅3.50美元（约GPT-4o判别堆栈的1/14）。配套跨厂商行为A/B实验提供初步系统级证据。代码、锚点、冻结测试数据及审计工具开源，许可为MIT，详见github.com/chunxiaoxx/nautilus-compass。
原文摘要 · Abstract (English)
Production LLM coding agents drift over long sessions: they forget user-specified constraints, slip into mistakes the user already flagged, and confabulate prior agreements. White-box approaches such as persona vectors require model weights and so cannot be applied to closed APIs (Claude, GPT-4) that most users actually interact with. We present Nautilus Compass, a black-box persona drift detector and agent memory layer for production coding agents. The method operates entirely at the prompt-text layer: cosine similarity between user prompts and behavioral anchor texts, aggregated by a weighted top-k mean using BGE-m3 embeddings. Compass is, to our knowledge, the only public agent memory layer (among Mem0, Letta, Cognee, Zep, MemOS, smrti verified May 2026) that does not call an LLM at index time to extract facts or build a graph; raw conversation text is embedded directly. The system ships as a Claude Code plugin, an MCP 2024-11-05 A2A server (Cursor, Cline, Hermes), a CLI, and a REST API on one daemon, with a Merkle-chained audit log for tamper-evident anchor updates. On a held-out test set built from real Claude Code session traces and labeled by an independent LLM judge, Compass reaches ROC AUC 0.83 for drift detection. The embedded retrieval pipeline scores 56.6% on LongMemEval-S v0.8 and 44.4% on EverMemBench-Dynamic (n=500), topping the four published EverMemBench Table 4 baselines. LongMemEval-S 56.6% is ~30 points below recent white-box leaders (90+%); we treat that as the architectural ceiling of the no-extraction design. End-to-end reproduction cost is $3.50 (~14x cheaper than GPT-4o-judged stacks). A paired cross-vendor behavior A/B accompanies these numbers as preliminary system-level evidence. Code, anchors, frozen test data, and audit-log tooling are MIT-licensed at github.com/chunxiaoxx/nautilus-compass.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。