测试代码智能体能否理解软件架构，发现模型能力差异大。
Theory of Code Space: Do Code Agents Understand Software Architecture?
- 用动态探索生成代码库，评估模型构建和更新架构认知能力。
- 小模型信念稳定，大模型易崩溃，主动探索效果因模型而异。
- 适合研究大模型代码理解能力、架构推理机制的学者参考。
AI代码智能体在独立任务上表现优异，但在需要理解多文件软件架构的工程任务中仍显不足。本文提出代码空间理论（ToCS）基准，评估智能体在代码库探索过程中构建、维护和更新连贯架构认知的能力。智能体在部分可观测环境下通过预算内打开文件进行探索，并定期以结构化JSON形式外化其信念状态，形成时间序列的架构理解记录。对四个基线与六种前沿LLM的实验揭示三个发现：第一，主动-被动差距具有模型依赖性——某些模型通过主动探索构建更优地图，而另一些模型则相反，表明主动探索本身是复杂能力；第二，将结构化信念保留在上下文中对部分模型起到自我支撑作用，但对其他模型无效；第三，信念维持能力差异显著：较小模型在探测间保持完全稳定的信念，而其更大版本却出现灾难性信念崩溃，遗忘先前发现的组件。代码已开源。
原文摘要 · Abstract (English)
AI code agents excel at isolated tasks yet struggle with multi-file software engineering requiring architectural understanding. We introduce Theory of Code Space (ToCS), a benchmark that evaluates whether agents can construct, maintain, and update coherent architectural beliefs during codebase exploration. Agents explore procedurally generated codebases under partial observability -- opening files under a budget -- and periodically externalize their belief state as structured JSON, producing a time-series of architectural understanding. Three findings emerge from experiments with four baselines and six frontier LLMs. First, the Active-Passive Gap is model-dependent: one model builds better maps through active exploration than from seeing all files at once, while another shows the opposite -- revealing that active exploration is itself a non-trivial capability absent from some models. Second, retaining structured belief maps in context acts as self-scaffolding for some models but not others, showing that the mechanism is model-dependent. Third, belief state maintenance varies dramatically: a smaller model maintains perfectly stable beliefs across probes while its larger sibling suffers catastrophic belief collapse -- forgetting previously-discovered components between probes. We release ToCS as open-source software. Code: https://github.com/che-shr-cat/tocs
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。