用可编辑画布让AI推理更高效,错误修改不用重来。
Canvas-of-Thought: Grounding Reasoning via Mutable Structured States
- 用网页画布作为外部思维载体,支持精准修改和状态追踪。
- 在多个评测中表现优于现有方法,最高提升18.7个百分点。
- 适合需要反复修改的复杂视觉推理任务,如几何设计与代码生成。
尽管思维链(CoT)提示显著提升了多模态大模型的推理能力,但仅依赖线性文本序列仍制约复杂任务表现。我们发现,即使引入视觉元素,它们常被视为一维无结构链中的静态快照。此类方法将推理历史视为不可变流,修正局部错误需冗长下游修正或重生成全部上下文,大幅增加令牌消耗与认知负担。这一限制在高维领域(如几何与SVG设计)尤为突出,因文本表达缺乏显式视觉引导,进一步削弱推理精度。为此,我们提出 extbf{Canvas-of-Thought (Canvas-CoT)}:通过HTML Canvas作为外部推理基底,使模型可执行原子级的DOM增删改查操作。该架构支持原位状态更新而不扰动上下文,实现‘真实状态’的显式维护。此外,集成渲染反馈机制作为硬约束验证器,提供明确视觉反馈以解决纯文本难以描述的复杂任务。在VCode、RBench-V与MathVista上的大量实验表明,Canvas-CoT显著超越现有基线,确立了新的上下文高效多模态推理范式。
原文摘要 · Abstract (English)
While Chain-of-Thought (CoT) prompting has significantly advanced the reasoning capabilities of Multimodal Large Language Models (MLLMs), relying solely on linear text sequences remains a bottleneck for complex tasks. We observe that even when auxiliary visual elements are interleaved, they are often treated as static snapshots within a one-dimensional, unstructured reasoning chain. We argue that such approaches treat reasoning history as an immutable stream: correcting a local error necessitates either generating verbose downstream corrections or regenerating the entire context. This forces the model to implicitly maintain and track state updates, significantly increasing token consumption and cognitive load. This limitation is particularly acute in high-dimensional domains, such as geometry and SVG design, where the textual expression of CoT lacks explicit visual guidance, further constraining the model's reasoning precision. To bridge this gap, we introduce \textbf{Canvas-of-Thought (Canvas-CoT)}. By leveraging a HTML Canvas as an external reasoning substrate, Canvas-CoT empowers the model to perform atomic, DOM-based CRUD operations. This architecture enables in-place state revisions without disrupting the surrounding context, allowing the model to explicitly maintain the "ground truth". Furthermore, we integrate a rendering-based critique loop that serves as a hard constraint validator, providing explicit visual feedback to resolve complex tasks that are difficult to articulate through text alone. Extensive experiments on VCode, RBench-V, and MathVista demonstrate that Canvas-CoT significantly outperforms existing baselines, establishing a new paradigm for context-efficient multimodal reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。