提出可评估AI持续性能力的开源框架，解决长期记忆中事实混淆问题。
ATANT: An Evaluation Framework for AI Continuity
- 定义7项连续性属性，设计10阶段无LLM参与的评估流程
- 在250个故事累积模式下实现96%准确率，250故事独立模式达100%
- 适用于验证各类长时记忆系统，适合研究者与工程团队使用
我们提出ATANT（叙事真实性接受度自动化测试）——一个开放的评估框架，用于衡量AI系统的连续性：即跨时间持久、更新、消歧和重建有意义上下文的能力。尽管行业已开发出记忆组件（RAG流水线、向量数据库、长上下文窗口、用户画像层），但尚无公开框架能正式定义或测量这些组件是否真正实现连续性。我们定义连续性为具有7项必要属性的系统特性，引入无需LLM参与的10阶段评估方法，并构建包含250个故事、1,835个验证问题的叙事测试语料库，覆盖6个生活领域。对基准实现进行5轮测试，从旧架构的58%提升至孤立模式下250故事的100%，50故事累积模式100%，250故事累积模式达96%。累积结果是核心指标：当250个不同人生叙事共存于同一数据库时，系统必须不交叉污染地正确检索对应事实。ATANT具有系统无关性、模型无关性，设计为构建与验证连续性系统的序列化方法。框架规范、示例故事及评估协议已发布于https://github.com/Kenotic-Labs/ATANT，完整250故事语料库将逐步释放。
原文摘要 · Abstract (English)
We present ATANT (Automated Test for Acceptance of Narrative Truth), an open evaluation framework for measuring continuity in AI systems: the ability to persist, update, disambiguate, and reconstruct meaningful context across time. While the AI industry has produced memory components (RAG pipelines, vector databases, long context windows, profile layers), no published framework formally defines or measures whether these components produce genuine continuity. We define continuity as a system property with 7 required properties, introduce a 10-checkpoint evaluation methodology that operates without an LLM in the evaluation loop, and present a narrative test corpus of 250 stories comprising 1,835 verification questions across 6 life domains. We evaluate a reference implementation across 5 test suite iterations, progressing from 58% (legacy architecture) to 100% in isolated mode (250 stories) and 100% in 50-story cumulative mode, with 96% at 250-story cumulative scale. The cumulative result is the primary measure: when 250 distinct life narratives coexist in the same database, the system must retrieve the correct fact for the correct context without cross-contamination. ATANT is system-agnostic, model-independent, and designed as a sequenced methodology for building and validating continuity systems. The framework specification, example stories, and evaluation protocol are available at https://github.com/Kenotic-Labs/ATANT. The full 250-story corpus will be released incrementally.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。