融合大模型与CLIP,实现教育游戏中的多粒度物体识别
Bridging Coarse and Fine Recognition: A Hybrid Approach for Open-Ended Multi-Granularity Object Recognition in Interactive Educational Games

- 用大模型做粗粒度识别,用CLIP专注细粒度分类
- 在课本图像上识别准确率提升23.2%,细粒度差距仅0.2%
- 适合需要精准物体感知的互动教育游戏开发
多模态大模型虽能实现开放域物体识别,但在细粒度任务上表现不足;而CLIP类模型擅长细粒度识别,却缺乏通用类别覆盖。为此,我们提出HyMOR框架,将大模型与CLIP模型结合:大模型负责开放域粗粒度识别,CLIP专注动物、植物等特定领域细粒度识别。该混合设计实现了跨语义粒度的精准物体理解,为多模态内容生成与互动游戏提供可靠感知基础。为支持教育场景评估,我们构建了TBO数据集,包含20,942张教材图像,标注8,816个物体类别。实验表明,HyMOR在细粒度识别上与CLIP差距仅为0.2%,比基线大模型提升2.5%通用识别能力,整体平均SBert相似度提升23.2%,验证其在多模态游戏生成与互动学习中的有效性。
原文摘要 · Abstract (English)
Recent advances in Multimodal Large Language Models (MLLMs) have enabled open-ended object recognition, yet they struggle with fine-grained tasks. In contrast, CLIP-style models excel at fine-grained recognition but lack broad coverage of general object categories. To bridge this gap, we propose \textbf{HyMOR}, a \textbf{Hy}brid \textbf{M}ulti-granularity open-ended \textbf{O}bject \textbf{R}ecognition framework that integrates an MLLM with a CLIP model. In HyMOR, the MLLM performs open-ended and coarse-grained object recognition, while the CLIP model specializes in fine-grained identification of domain-specific objects such as animals and plants. This hybrid design enables accurate object understanding across multiple semantic granularities, serving as a robust perceptual foundation for downstream multi-modal content generation and interactive gameplay. To support evaluation in content-rich and educational scenarios, we introduce TBO (TextBook Objects), a dataset containing 20,942 images annotated with 8,816 object categories extracted from textbooks. Extensive experiments demonstrate that HyMOR narrows the fine-grained recognition gap with CLIP to 0.2\% while improving general object recognition by 2.5\% over a baseline MLLM, measured by average Sentence-BERT (SBert) similarity. Overall, HyMOR achieves a 23.2\% improvement in average SBert across all evaluated datasets, highlighting its effectiveness in enabling accurate perception for multi-modal game content generation and interactive learning applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。