用大模型隐状态直接增强目标检测，无需人工设计提示词。
LED: LLM Enhanced Open-Vocabulary Object Detection without Human Curated Data Generation
- 将大语言模型中间层隐状态融合到检测器，实现零初始化交叉注意力适配。
- 在OmniLabel上提升GroundingDINO性能3.82%~6.22%，仅增加8.7%计算量。
- 适合追求高精度开放词汇检测且无标注数据的科研与工程场景。
大规模视觉语言预训练模型可通过合成数据提升开放词汇目标检测性能，但人工设计的生成流程常引入偏差并过拟合特定提示。本文提出直接融合大语言模型（LLM）隐藏状态到检测器的新方法，该路径此前极少被探索。我们系统性地利用多模态大模型（MLLM）解码器层实现视觉定位增强，引入零初始化交叉注意力适配器，实现高效知识迁移，称为LED（LLM Enhanced Open-Vocabulary Object Detection）。研究发现，LLM中间层已蕴含丰富空间语义信息，仅适配早期层即可获得主要收益。以Swin-T为视觉编码器，Qwen2-0.5B + LED在OmniLabel上使GroundingDINO提升3.82%，仅增加8.7%额外GFLOPs；使用更大视觉主干网络时，提升达6.22%。对适配器变体、LLM规模及融合深度的广泛消融实验进一步验证了设计有效性。
原文摘要 · Abstract (English)
Large foundation models trained on large-scale vision-language data can boost Open-Vocabulary Object Detection (OVD) via synthetic training data, yet the hand-crafted pipelines often introduce bias and overfit to specific prompts. We sidestep this issue by directly fusing hidden states from Large Language Models (LLMs) into detectors-an avenue surprisingly under-explored. This paper presents a systematic method to enhance visual grounding by utilizing decoder layers of the LLM of an MLLM. We introduce a zero-initialized cross-attention adapter to enable efficient knowledge fusion from LLMs to object detectors, a new approach called LED (LLM Enhanced Open-Vocabulary Object Detection). We find that intermediate LLM layers already encode rich spatial semantics; adapting only the early layers yields most of the gain. With Swin-T as the vision encoder, Qwen2-0.5B + LED lifts GroundingDINO by 3.82 % on OmniLabel at just 8.7 % extra GFLOPs, and a larger vision backbone pushes the improvement to 6.22 %. Extensive ablations on adapter variants, LLM scales and fusion depths further corroborate our design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。