动态选择模型组合,提升人体识别准确率与效率
FusionAgent: A Multimodal Agent with Dynamic Model Selection for Human Recognition
- 用大语言模型做智能选模,根据样本特点动态决定调用哪些模型
- 在多个数据集上超越现有方法,且减少约40%的模型调用次数
- 适合需要高鲁棒性、可解释性的实际人体识别系统
模型融合是应对非受限场景下人体识别挑战的关键策略,因不同模型在面部、步态、体型等生物特征上具有互补优势。传统分数融合方法多为静态策略,对所有测试样本均调用全部模型,忽视样本质量与模态可靠性。为此,本文提出FusionAgent——一种基于多模态大语言模型(MLLM)的智能代理框架,将各专家模型视为工具,通过基于指标的强化微调(RFT)学习自适应选择最优模型组合。针对得分不一致与嵌入异质性问题,提出锚定置信度的Top-k融合(ACT),以最可靠模型为锚点,融合互补预测结果。在多个全身生物特征基准测试中,FusionAgent显著优于当前最佳方法,并通过更少的模型调用实现更高效率,验证了动态、可解释、鲁棒模型融合在真实识别系统中的关键作用。
原文摘要 · Abstract (English)
Model fusion is a key strategy for robust recognition in unconstrained scenarios, as different models provide complementary strengths. This is especially important for whole-body human recognition, where biometric cues such as face, gait, and body shape vary across samples and are typically integrated via score-fusion. However, existing score-fusion strategies are usually static, invoking all models for every test sample regardless of sample quality or modality reliability. To overcome these limitations, we propose \textbf{FusionAgent}, a novel agentic framework that leverages a Multimodal Large Language Model (MLLM) to perform dynamic, sample-specific model selection. Each expert model is treated as a tool, and through Reinforcement Fine-Tuning (RFT) with a metric-based reward, the agent learns to adaptively determine the optimal model combination for each test input. To address the model score misalignment and embedding heterogeneity, we introduce Anchor-based Confidence Top-k (ACT) score-fusion, which anchors on the most confident model and integrates complementary predictions in a confidence-aware manner. Extensive experiments on multiple whole-body biometric benchmarks demonstrate that FusionAgent significantly outperforms SoTA methods while achieving higher efficiency through fewer model invocations, underscoring the critical role of dynamic, explainable, and robust model fusion in real-world recognition systems. Project page: \href{https://fusionagent.github.io/}{FusionAgent}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。