让大模型代理的思考过程可被观察，提升自主系统透明度。
Watson: A Cognitive Observability Framework for the Reasoning of LLM-Powered Agents
- 通过提示词归因技术逆向推断代理的隐含推理路径。
- 在MMLU和SWE-bench-lite上验证了推理洞察的有效性。
- 适合关注大模型代理可信性与可调试性的研发人员。
大型语言模型（LLMs）正被越来越多地集成到自主系统中，催生出一类新型软件——Agentware，即由大模型驱动的智能体在软件工程、客户服务和数据分析等开放领域执行复杂任务。然而，其高度自治性与黑箱推理过程给传统软件可观测性方法带来挑战。为此，我们提出认知可观测性概念——即恢复并审视智能体决策背后的隐含推理。本文介绍Watson，一个通用框架，可在不改变智能体行为的前提下，对快速思考型大模型智能体的推理过程进行观测。Watson采用提示词归因技术，事后推断推理轨迹。我们在MMLU基准以及AutoCodeRover和OpenHands智能体在SWE-bench-lite数据集上的手动调试与自动修正场景中评估了Watson。无论静态还是动态环境下，Watson均能揭示可操作的推理洞见，并支持精准干预，证明其在提升Agentware系统透明性与可靠性方面的实用价值。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly integrated into autonomous systems, giving rise to a new class of software known as Agentware, where LLM-powered agents perform complex, open-ended tasks in domains such as software engineering, customer service, and data analysis. However, their high autonomy and opaque reasoning processes pose significant challenges for traditional software observability methods. To address this, we introduce the concept of cognitive observability - the ability to recover and inspect the implicit reasoning behind agent decisions. We present Watson, a general-purpose framework for observing the reasoning processes of fast-thinking LLM agents without altering their behavior. Watson retroactively infers reasoning traces using prompt attribution techniques. We evaluate Watson in both manual debugging and automated correction scenarios across the MMLU benchmark and the AutoCodeRover and OpenHands agents on the SWE-bench-lite dataset. In both static and dynamic settings, Watson surfaces actionable reasoning insights and supports targeted interventions, demonstrating its practical utility for improving transparency and reliability in Agentware systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。