arXiv:2601.04381cs.CVcs.AI2026-01被引 1

用少量数据让基础模型跨模态翻译,提升红外与雷达目标检测效果。

Few-Shot LoRA Adaptation of a Flow-Matching Foundation Model for Cross-Spectral Object Detection

  • 仅用100对图像,通过低秩适配微调流匹配模型实现跨模态翻译。
  • 合成的红外/雷达图像使行人与基础设施检测准确率显著提升。
  • 50组验证数据的LPIPS值可预判下游检测性能,指导模型选择。

视觉基础模型主要在RGB数据上训练,但许多安全关键应用依赖红外(IR)和合成孔径雷达(SAR)等非可见模态。本文研究是否可通过少量共测样本,将仅在RGB图像上预训练的流匹配基础模型,转化为跨模态翻译器,并利用生成的合成数据增强下游检测。基于FLUX.1 Kontext,插入低秩适配(LoRA)模块,在KAIST(RGB→IR)和M4-SAR(RGB→SAR)数据集上各仅用100对图像进行微调。适应后的模型能生成像素对齐的红外/雷达图像,从而复用原有边界框,在目标模态上纯训练检测模型。在一组LoRA超参数中发现,仅用50组保留样本计算的LPIPS值是下游性能的强代理:较低LPIPS始终预测更高mAP——YOLOv11n在红外和雷达上表现更优,DETR在KAIST IR测试集上也表现更好。使用最佳LPIPS筛选的LoRA适配器,来自外部RGB数据集(LLVIP、FLIR ADAS)的合成红外图像提升了KAIST红外行人检测性能;合成SAR结合有限真实SAR数据后,显著增强M4-SAR上的基础设施检测。结果表明,少数样本下对流匹配基础模型进行LoRA适配,是支持非可见模态的基础模型路径。

原文摘要 · Abstract (English)

Foundation models for vision are predominantly trained on RGB data, while many safety-critical applications rely on non-visible modalities such as infrared (IR) and synthetic aperture radar (SAR). We study whether a single flow-matching foundation model pre-trained primarily on RGB images can be repurposed as a cross-spectral translator using only a few co-measured examples, and whether the resulting synthetic data can enhance downstream detection. Starting from FLUX.1 Kontext, we insert low-rank adaptation (LoRA) modules and fine-tune them on just 100 paired images per domain for two settings: RGB to IR on the KAIST dataset and RGB to SAR on the M4-SAR dataset. The adapted model translates RGB images into pixel-aligned IR/SAR, enabling us to reuse existing bounding boxes and train object detection models purely in the target modality. Across a grid of LoRA hyperparameters, we find that LPIPS computed on only 50 held-out pairs is a strong proxy for downstream performance: lower LPIPS consistently predicts higher mAP for YOLOv11n on both IR and SAR, and for DETR on KAIST IR test data. Using the best LPIPS-selected LoRA adapter, synthetic IR from external RGB datasets (LLVIP, FLIR ADAS) improves KAIST IR pedestrian detection, and synthetic SAR significantly boosts infrastructure detection on M4-SAR when combined with limited real SAR. Our results suggest that few-shot LoRA adaptation of flow-matching foundation models is a promising path toward foundation-style support for non-visible modalities.

跨模态少样本目标检测生成模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。