让视觉模型适应鱼眼相机的畸变,无需重新训练就能准确感知全景
FishRoPE: Projective Rotary Position Embeddings for Omnidirectional Visual Perception

- 用鱼眼旋转位置编码重构注意力机制,基于角度而非像素距离计算
- 在WoodScape上达54.3 mAP,SynWoodScapes上达65.1 mIoU,性能领先
- 轻量改造冻结模型,适合部署在自动驾驶鱼眼摄像头系统中
视觉基础模型(VFMs)和鸟瞰图(BEV)表示显著提升了视觉感知能力,但其内部空间表征假设了针孔相机的矩形几何。而广泛用于量产自动驾驶车辆的鱼眼相机具有严重径向畸变,导致现有表征在几何上不一致。同时,大规模鱼眼标注数据稀缺,使从头训练基础模型不可行。本文提出 extbf{FishRoPE},一种轻量级框架,通过两个组件将冻结的VFMs适配至鱼眼几何:冻结的DINOv2主干结合低秩适配(LoRA),无须任务特定预训练即可迁移丰富的自监督特征;鱼眼旋转位置编码(FishRoPE)重新参数化球面投影下的注意力机制,使自注意力与交叉注意力均基于角度间隔而非像素距离运行。FishRoPE具有架构无关性,引入可忽略的计算开销,并在针孔几何下自然退化为标准形式。我们在WoodScape 2D检测(54.3 mAP)和SynWoodScapes BEV分割(65.1 mIoU)上进行评估,结果均达到当前最优水平。
原文摘要 · Abstract (English)
Vision foundation models (VFMs) and Bird's Eye View (BEV) representation have advanced visual perception substantially, yet their internal spatial representations assume the rectilinear geometry of pinhole cameras. Fisheye cameras, widely deployed on production autonomous vehicles for their surround-view coverage, exhibit severe radial distortion that renders these representations geometrically inconsistent. At the same time, the scarcity of large-scale fisheye annotations makes retraining foundation models from scratch impractical. We present \ours, a lightweight framework that adapts frozen VFMs to fisheye geometry through two components: a frozen DINOv2 backbone with Low-Rank Adaptation (LoRA) that transfers rich self-supervised features to fisheye without task-specific pretraining, and Fisheye Rotary Position Embedding (FishRoPE), which reparameterizes the attention mechanism in the spherical coordinates of the fisheye projection so that both self-attention and cross-attention operate on angular separation rather than pixel distance. FishRoPE is architecture-agnostic, introduces negligible computational overhead, and naturally reduces to the standard formulation under pinhole geometry. We evaluate \ours on WoodScape 2D detection (54.3 mAP) and SynWoodScapes BEV segmentation (65.1 mIoU), where it achieves state-of-the-art results on both benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。