去掉视觉定位解码器中的前馈网络，性能几乎不变且更省资源。
Do Visual Grounding Decoders Need Feed-Forward Networks? A Controlled Study over Frozen Vision-Language Features
- 用纯注意力结构替代解码器中的前馈网络，仅依赖预训练的视觉语言特征。
- 在多个数据集上，纯注意力解码器性能接近甚至略超含前馈网络的版本。
- 减少44.4%可训练参数，降低10.1%缓存延迟，适合轻量化部署场景。
视觉定位解码器中的前馈网络（FFNs）在预训练视觉语言模型已编码图像与语言上下文后，是否仍需额外计算？我们对比了四个块的纯注意力解码器（A4）、匹配的含前馈网络解码器（S4）以及八块的纯注意力参数对照组（A8），均基于冻结的VLM特征。A4在RefCOCOg和Ref-Adv-s上表现与S4相当或稍优。FineCops-Ref显示，A4在[email protected]上略有0.52个百分点劣势（95%置信区间[0.12, 0.95]支持S4），但A8反超，最终领先S4 0.26个百分点。官方FineCops评估未呈现单调差距。A4将可训练解码器参数减少44.4%，缓存解码延迟降低10.1%，但端到端延迟仍由骨干网络主导。结论针对可训练的接地解码器，非完整注意力型VLM。
原文摘要 · Abstract (English)
Do feed-forward networks (FFNs) in visual grounding decoders add essential computation once a pretrained vision-language model has already encoded image and language context? We compare a four-block attention-only decoder (A4), a matched four-block attention-plus-FFN decoder (S4), and an eight-block attention-only parameter control (A8) over frozen VLM features. A4 matches or slightly exceeds S4 on RefCOCOg and Ref-Adv-s. FineCops-Ref reveals a small A4 deficit of 0.52 percentage points at [email protected] (95% CI [0.12, 0.95] in favor of S4), but A8 recovers it and finishes 0.26 points above S4. Official FineCops levels do not show a monotonic increase in the gap. A4 reduces trainable decoder parameters by 44.4% and cached-decoder latency by 10.1%, although end-to-end latency remains backbone-dominated. These results concern the trainable grounding decoder, not a complete attention-only VLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。