用强化学习直接优化3D场景理解模型,让效果更贴近实际任务指标。
3D-RFT: Reinforcement Fine-Tuning for Video-based 3D Scene Understanding
- 通过可验证奖励函数,直接以3D IoU、F1等指标指导模型训练
- 4B模型在多个任务上超越更大模型(如VG LLM-8B)
- 适合做视频驱动的3D感知与推理研究者参考
基于视觉语言模型的3D场景理解仍主要依赖监督微调,其损失函数与任务目标存在偏差。本文提出首个面向视频驱动3D场景理解的强化微调框架3D-RFT,将可验证奖励强化学习(RLVR)引入该领域。先通过监督微调激活3D多模态大模型,再采用组相对策略优化(GRPO)进行强化微调,设计基于3D IoU和F1-Score的任务特定奖励函数,直接优化评估指标。实验表明,3D-RFT-4B在3D视频检测、3D视觉定位和空间推理等任务上达到领先性能,显著优于更大规模的VG LLM-8B。同时揭示了该方法在训练策略与数据影响方面的良好特性,为未来3D场景理解提供可靠范式。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards ( RLVR ) has emerged as a transformative paradigm for enhancing the reasoning capabilities of Large Language Models ( LLMs), yet its potential in 3D scene understanding remains under-explored. Existing approaches largely rely on Supervised Fine-Tuning ( SFT), where the token-level cross-entropy loss acts as an indirect proxy for optimization, leading to a misalignment between training objectives and task performances. To bridge this gap, we present Reinforcement Fine-Tuning for Video-based 3D Scene Understanding (3D-RFT ), the first framework to extend RLVR to video-based 3D perception and reasoning. 3D-RFT shifts the paradigm by directly optimizing the model towards evaluation metrics. 3D-RFT first activates 3D-aware Multi-modal Large Language Models ( MLLM s) via SFT, followed by reinforcement fine-tuning using Group Relative Policy Optimization ( GRPO) with strictly verifiable reward functions. We design task-specific reward functions directly from metrics like 3D IoU and F1-Score to provide more effective signals to guide model training. Extensive experiments demonstrate that 3D-RFT-4B achieves state-of-the-art performance on various video-based 3D scene understanding tasks. Notably, 3D-RFT-4B significantly outperforms larger models (e.g., VG LLM-8B) on 3D video detection, 3D visual grounding, and spatial reasoning benchmarks. We further reveal good properties of 3D-RFT such as robust efficacy, and valuable insights into training strategies and data impact. We hope 3D-RFT can serve as a robust and promising paradigm for future development of 3D scene understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。