arXiv:2602.12936cs.CV2026-02被引 1

用大模型统一跨模态重识别,让边缘设备也能高效运行。

Unleashing MLLMs on the Edge: A Unified Framework for Cross-Modal ReID via Adaptive SVD Distillation

  • 将大模型适配为统一嵌入空间,支持多模态输入。
  • 边缘模型在多个基准上达到顶尖性能,精度超现有方法。
  • 适合需要多模态识别的边缘计算场景,如安防、机器人。

跨模态重识别(CM-ReID)在云-边部署中面临专用云端模型碎片化的问题。尽管多模态大语言模型(MLLM)具备统一潜力,但现有方法难以将其整合为端到端骨干网络,且缺乏有效的知识蒸馏策略用于边缘部署。为此,我们提出 MLLMEmbed-ReID,一种基于云-边架构的统一框架。首先,将基础 MLLM 适配为先进云端模型,通过指令提示引导其在 RGB、红外、素描和文本模态间生成统一嵌入空间;采用分层低秩微调(LoRA-SFT)策略,在整体跨模态对齐目标下高效训练。其次,为将知识迁移至轻量级边缘学生模型,我们提出一种基于教师特征空间低秩特性的新蒸馏策略:通过主成分映射损失优先保留关键信息,以特征关系损失保持结构一致性。轻量级边缘模型在多个视觉 CM-ReID 基准上达到最先进性能,而云端版本在所有基准上表现优异。该框架实现了资源受限设备上的统一大模型智能部署。代码与模型即将开源。

原文摘要 · Abstract (English)

Practical cloud-edge deployment of Cross-Modal Re-identification (CM-ReID) faces challenges due to maintaining a fragmented ecosystem of specialized cloud models for diverse modalities. While Multi-Modal Large Language Models (MLLMs) offer strong unification potential, existing approaches fail to adapt them into a single end-to-end backbone and lack effective knowledge distillation strategies for edge deployment. To address these limitations, we propose MLLMEmbed-ReID, a unified framework based on a powerful cloud-edge architecture. First, we adapt a foundational MLLM into a state-of-the-art cloud model. We leverage instruction-based prompting to guide the MLLM in generating a unified embedding space across RGB, infrared, sketch, and text modalities. This model is then trained efficiently with a hierarchical Low-Rank Adaptation finetuning (LoRA-SFT) strategy, optimized under a holistic cross-modal alignment objective. Second, to deploy its knowledge onto an edge-native student, we introduce a novel distillation strategy motivated by the low-rank property in the teacher's feature space. To prioritize essential information, this method employs a Principal Component Mapping loss, while relational structures are preserved via a Feature Relation loss. Our lightweight edge-based model achieves state-of-the-art performance on multiple visual CM-ReID benchmarks, while its cloud-based counterpart excels across all CM-ReID benchmarks. The MLLMEmbed-ReID framework thus presents a complete and effective solution for deploying unified MLLM-level intelligence on resource-constrained devices. The code and models will be open-sourced soon.

跨模态识别大模型蒸馏边缘计算多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。