arXiv:2603.27365cs.CV2026-03

统一视觉语言模型,用单栈实现感知与任务建模,提升密集预测精度。

Falcon Perception

  • 共享参数的早期融合架构,图像与文本在首层协同处理。
  • 在SA-Co上达到68.0的宏平均F1,优于SAM3的62.3。
  • 适合需要密集输出和复杂提示理解的任务场景。

感知系统通常采用模块化的编码器-解码器结构:视觉主干提取特征,独立解码器或后融合模块进行任务预测。这引发核心问题:这种结构分离是否必要?能否用单一早期融合堆叠实现大规模感知与任务建模?我们提出Falcon Perception,一种统一的稠密Transformer,从第一层开始在共享参数空间中处理图像块与文本标记,采用混合注意力模式(图像标记间双向、预测标记自回归),结合全局视觉上下文与可变长度实例生成。为保持稠密输出实用性,该模型保留轻量级标记接口,并通过专用头解码连续空间输出,支持并行高分辨率掩码预测。设计强调简洁性:仅保留一个可扩展主干,将复杂度移至数据与训练信号,仅在连续密集输出处添加小型头部。在SA-Co数据集上,Falcon Perception掩码质量达68.0宏平均F1,优于SAM3的62.3。我们还引入PBench基准,针对组合式提示(如OCR、空间约束、关系)和稠密长上下文场景,在此任务上模型表现更优。最后,我们将相同早期融合方法扩展至Falcon OCR——一个300M参数的小型模型,在olmOCR上取得80.3%准确率,在OmniDocBench上达88.64分。

原文摘要 · Abstract (English)

Perception-centric systems are typically implemented with a modular encoder-decoder pipeline: a vision backbone for feature extraction and a separate decoder (or late-fusion module) for task prediction. This raises a central question: is this architectural separation essential or can a single early-fusion stack do both perception and task modeling at scale? We introduce Falcon Perception, a unified dense Transformer that processes image patches and text tokens in a shared parameter space from the first layer, using a hybrid attention pattern (bidirectional among image tokens, causal for prediction tokens) to combine global visual context with autoregressive, variable-length instance generation. To keep dense outputs practical, Falcon Perception retains a lightweight token interface and decodes continuous spatial outputs with specialized heads, enabling parallel high-resolution mask prediction. Our design promotes simplicity: we keep a single scalable backbone and shift complexity toward data and training signals, adding only small heads where outputs are continuous and dense. On SA-Co, Falcon Perception improves mask quality to 68.0 Macro-F$_1$ compared to 62.3 of SAM3. We also introduce PBench, a benchmark targeting compositional prompts (OCR, spatial constraints, relations) and dense long-context regimes, where the model shows better gains. Finally, we extend the same early-fusion recipe to Falcon OCR: a compact 300M-parameter model which attains 80.3% on olmOCR and 88.64 on OmniDocBench.

视觉语言稠密预测早期融合Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。