让大模型像写代码一样自动处理图像并推理,提升复杂任务表现
Thyme: Think Beyond Images
- 用可执行代码自主生成和操作图像,同时进行数学计算
- 在近20个基准上显著提升高分辨率感知与复杂推理能力
- 适合需要强视觉推理和自动化决策的AI研究者
继OpenAI提出‘用图像思考’概念后,现有工作尝试通过视觉信息增强模型在感知与推理任务中的表现。然而,据我们所知,目前尚无开源模型能像专有模型(O3)一样提供丰富多样的图像操作与代码驱动的逻辑推理能力。本文提出Thyme(Think Beyond Images),一种新范式,使多模态大模型超越传统‘用图像思考’方式,通过可执行代码自主生成并执行多样化的图像处理与计算操作。该方法不仅支持实时图像变换(如裁剪、旋转、对比度增强),还能完成数学计算,且在何时何地应用这些操作上保持高度自主性。我们采用两阶段训练策略:首先在50万条精选样本上进行监督微调以学习代码生成,随后通过强化学习优化决策机制。为提升学习难度,手动构建高分辨率问答对,并提出GRPO-ATS算法,分别对文本与代码生成施加不同温度以平衡推理探索与代码执行精度。大量实验与消融分析表明,Thyme在近20个基准上均取得显著且一致的性能提升,尤其在高分辨率感知与复杂推理任务中表现突出。
原文摘要 · Abstract (English)
Following OpenAI's introduction of the ``thinking with images'' concept, recent efforts have explored stimulating the use of visual information in the reasoning process to enhance model performance in perception and reasoning tasks. However, to the best of our knowledge, no open-source work currently offers a feature set as rich as proprietary models (O3), which can perform diverse image manipulations and simultaneously enhance logical reasoning capabilities through code. In this paper, we make a preliminary attempt in this direction by introducing Thyme (Think Beyond Images), a novel paradigm for enabling MLLMs to transcend existing ``think with images'' approaches by autonomously generating and executing diverse image processing and computational operations via executable code. This approach not only facilitates a rich, on-the-fly set of image manipulations (e.g., cropping, rotation, contrast enhancement) but also allows for mathematical computations, all while maintaining high autonomy in deciding when and how to apply these operations. We activate this capability through a two-stage training strategy: an initial SFT on a curated dataset of 500K samples to teach code generation, followed by a RL phase to refine decision-making. For the RL stage, we manually collect and design high-resolution question-answer pairs to increase the learning difficulty, and we propose GRPO-ATS (Group Relative Policy Optimization with Adaptive Temperature Sampling), an algorithm that applies distinct temperatures to text and code generation to balance reasoning exploration with code execution precision. We conduct extensive experimental analysis and ablation studies. Comprehensive evaluations on nearly 20 benchmarks show that Thyme yields significant and consistent performance gains, particularly in challenging high-resolution perception and complex reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。