提出轻量级推理分割模型,实现边缘设备实时运行。
Fast Reasoning Segmentation for Images and Videos
- 用数字孪生解耦感知与推理,支持多步逻辑迁移
- 0.6B参数模型性能超20倍参数大模型,达7.79帧/秒
- 适合嵌入式智能体、无人机等资源受限场景
推理分割通过隐式文本查询实现开放集目标分割,是自主智能体在真实环境运行的基础。然而,现有方法依赖数十亿参数的多模态大模型,超出边缘设备计算能力。虽然蒸馏可压缩模型,但现有方法未能传递推理分割所需的多步推理能力,因仅关注输出预测和中间特征匹配。我们提出FastReasonSeg,利用数字孪生表征重新定义问题,将感知与推理解耦,实现更有效的蒸馏。首先在教师生成的推理链上进行监督微调;随后通过联合奖励(评估分割准确率与推理质量一致性)进行强化微调。在两个视频基准(JiTBench, RVTBench)和两个图像基准(ReasonSeg, LLM-Seg40K)上的实验表明,FastReasonSeg达到当前最优性能。其0.6B参数版本在仅2.1GB内存下实现7.79 FPS吞吐量,性能优于参数量20倍更大的模型,满足资源受限环境下的实时推理分割需求。
原文摘要 · Abstract (English)
Reasoning segmentation enables open-set object segmentation via implicit text queries, therefore serving as a foundation for embodied agents that should operate autonomously in real-world environments. However, existing methods for reasoning segmentation require multimodal large language models with billions of parameters that exceed the computational capabilities of edge devices that typically deploy the embodied AI systems. Distillation offers a pathway to compress these models while preserving their capabilities. Yet, existing distillation approaches fail to transfer the multi-step reasoning capabilities that reasoning segmentation demands, as they focus on matching output predictions and intermediate features rather than preserving reasoning chains. The emerging paradigm of reasoning over digital twin representations presents an opportunity for more effective distillation by re-framing the problem. Consequently, we propose FastReasonSeg, which employs digital twin representations that decouple perception from reasoning to enable more effective distillation. Our distillation scheme first relies on supervised fine-tuning on teacher-generated reasoning chains. Then it is followed by reinforcement fine-tuning with joint rewards evaluating both segmentation accuracy and reasoning quality alignment. Experiments on two video (JiTBench, RVTBench) and two image benchmarks (ReasonSeg, LLM-Seg40K) demonstrate that our FastReasonSeg achieves state-of-the-art reasoning segmentation performance. Moreover, the distilled 0.6B variant outperforms models with 20 times more parameters while achieving 7.79 FPS throughput with only 2.1GB memory consumption. This efficiency enables deployment in resource-constrained environments to enable real-time reasoning segmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。