一个模型搞定图像和视频的多种理解任务,实现跨模态通用推理。
OneThinker: All-in-one Reasoning Model for Image and Video
- 统一图像与视频的10类基础视觉任务,用单一模型处理
- 在31个基准上表现优异,支持任务间知识迁移
- 适合需要多模态通用推理能力的研究与应用
强化学习(RL)近期在多模态大语言模型(MLLM)中取得了显著进展,用于激发视觉推理能力。然而,现有方法通常为不同任务训练独立模型,将图像与视频推理视为分离领域,限制了向通用多模态推理模型的扩展,阻碍了任务与模态间的知识共享。为此,我们提出 OneThinker,一个涵盖图像与视频理解的全任务统一推理模型,覆盖问答、描述生成、空间与时间定位、追踪、分割等10类基础视觉任务。我们构建了包含600,000样本的OneThinker-600k训练语料库,并使用商用模型进行思维链(CoT)标注,得到340,000样本的OneThinker-SFT-340k用于监督微调冷启动。此外,我们提出EMA-GRPO算法,通过追踪各任务奖励标准差的移动平均,缓解多任务强化学习中的奖励异质性问题,实现更均衡的优化。在多个视觉基准上的实验表明,OneThinker在31个基准上表现强劲,覆盖10类核心视觉理解任务。同时展现出特定任务间的有效知识迁移及初步零样本泛化能力,标志着向统一多模态推理通用模型迈出关键一步。所有代码、模型与数据均已公开。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has recently achieved remarkable success in eliciting visual reasoning within Multimodal Large Language Models (MLLMs). However, existing approaches typically train separate models for different tasks and treat image and video reasoning as disjoint domains. This results in limited scalability toward a multimodal reasoning generalist, which restricts practical versatility and hinders potential knowledge sharing across tasks and modalities. To this end, we propose OneThinker, an all-in-one reasoning model that unifies image and video understanding across diverse fundamental visual tasks, including question answering, captioning, spatial and temporal grounding, tracking, and segmentation. To achieve this, we construct the OneThinker-600k training corpus covering all these tasks and employ commercial models for CoT annotation, resulting in OneThinker-SFT-340k for SFT cold start. Furthermore, we propose EMA-GRPO to handle reward heterogeneity in multi-task RL by tracking task-wise moving averages of reward standard deviations for balanced optimization. Extensive experiments on diverse visual benchmarks show that OneThinker delivers strong performance on 31 benchmarks, across 10 fundamental visual understanding tasks. Moreover, it exhibits effective knowledge transfer between certain tasks and preliminary zero-shot generalization ability, marking a step toward a unified multimodal reasoning generalist. All code, model, and data are released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。