arXiv:2607.19857cs.CVcs.AI2026-07

解决无人机视频中微小目标识别的实时感知难题。

Memory-Augmented Multimodal Large Language Models for Small Object Understanding in Streaming Aerial Videos

论文配图:Memory-Augmented Multimodal Large Language Models for Small Object Understanding in Streaming Aerial Videos
图 1 · 摘自论文原文
  • 设计语义感知的令牌路由机制,高效保留微小目标细节。
  • 提出分层记忆库,在资源受限下持续追踪目标。
  • 构建首个像素级开源数据集,支持细粒度目标理解。

基于语言的空中感知旨在理解复杂无人飞行器(UAV)场景中用户指定的微小目标。在真实部署中,无人机需边飞行边响应,因此感知必须以在线流式方式运行——帧按顺序到达,模型需对每帧即时响应且无法访问未来帧。然而,将现有多模态大语言模型(MLLMs)应用于该场景面临两大挑战:其一,从空中视角看目标往往极小,而现有MLLM的视觉压缩对所有区域一视同仁,丢弃了细粒度信息;其二,理解连续视频流需要历史上下文,但在资源受限的机载硬件上完整保留历史不可行,而完全丢弃又会导致目标漂移或消失。本文从数据与方法双角度应对上述挑战。数据层面,提出首个像素级、开放词汇的参考分割数据集DroneEyes,包含2,140段高清视频和176,623个标注对,涵盖目标描述与指代表达任务,附带每帧密集掩码。方法层面,提出SkyAnchor,一种新型MLLM,包含两项设计:语义感知令牌路由,可在减少视觉令牌数量的同时保留微小目标特征;分层记忆库,确保在流式处理中持续稳定理解目标。

原文摘要 · Abstract (English)

Language-guided aerial perception aims to understand user-specified tiny targets in complex unmanned aerial vehicle (UAV) scenes. In real UAV deployment, the UAV must respond while it flies, so such perception runs in an online streaming manner, where frames arrive sequentially and the model responds to each one without access to future frames. However, applying current Multimodal Large Language Models (MLLMs) to this setting raises two challenges. First, targets viewed from the air are often tiny, yet the visual compression in existing MLLMs treats all regions equally and discards their fine-grained details. Second, understanding a continuous stream requires past-frame context, yet retaining the entire history is infeasible on resource-constrained onboard hardware, whereas discarding it causes the target to drift or disappear. We address the tiny object and streaming challenges from both data and method perspectives. From the data perspective, we present \textbf{DroneEyes}, the \textbf{first} pixel-level and open-vocabulary referring-segmentation dataset for tiny aerial targets, comprising $2,140$ high-definition videos and $176,623$ pairs across Object Description and Referring Expression tasks, with dense per-frame masks. From the method perspective, we propose \textbf{SkyAnchor}, an MLLM with two designs to the above challenges: a Semantics-Aware Token Router that preserves small-target under a reduced visual-token budget, and a Hierarchical Memory Bank that keeps the target consistently understood on streams.

无人机视觉小目标识别多模态模型流式处理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。