用自调用机制让模型仅靠文字完成复杂视觉推理，效率更高。
Thinking with Images via Self-Calling Agent
- 主代理分解任务，调用共享参数的虚拟子代理并行求解。
- 在HR-Bench 4K上提升1.9%性能，训练耗时减少75%。
- 无需跨模态交错，适合追求高效视觉推理的开发者。
基于图像的思维范式通过将视觉信息作为动态元素融入思维链（CoT），展现出强大的视觉推理能力。然而，利用强化学习优化多模态思维链（iMCoT）仍具挑战性，因其依赖稀缺的高质量推理数据。本文提出自调用思维链（sCoT），将iMCoT重构为纯语言的思维链，采用自调用机制。具体而言，主代理将复杂视觉推理任务分解为原子子任务，并调用其虚拟副本（即参数共享的子代理）在隔离上下文中求解。sCoT具备显著的训练效率与效果，无需显式模态交错。该方法采用组相对策略优化以增强有效推理行为。在HR-Bench 4K上的实验表明，sCoT相比强基线方法性能提升最高达1.9%，且训练消耗约减少75%的GPU小时数。代码已开源。
原文摘要 · Abstract (English)
Thinking-with-images paradigms have showcased remarkable visual reasoning capability by integrating visual information as dynamic elements into the Chain-of-Thought (CoT). However, optimizing interleaved multimodal CoT (iMCoT) through reinforcement learning remains challenging, as it relies on scarce high-quality reasoning data. In this study, we propose Self-Calling Chain-of-Thought (sCoT), a novel visual reasoning paradigm that reformulates iMCoT as a language-only CoT with self-calling. Specifically, a main agent decomposes the complex visual reasoning task to atomic subtasks and invokes its virtual replicas, i.e. parameter-sharing subagents, to solve them in isolated context. sCoT enjoys substantial training effectiveness and efficiency, as it requires no explicit interleaving between modalities. sCoT employs group-relative policy optimization to reinforce effective reasoning behavior to enhance optimization. Experiments on HR-Bench 4K show that sCoT improves the overall reasoning performance by up to $1.9\%$ with $\sim 75\%$ fewer GPU hours compared to strong baseline approaches. Code is available at https://github.com/YWenxi/think-with-images-through-self-calling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。