优化相机传感器滤色阵列，提升自动驾驶分割精度。
Beyond Bayer: Task-Optimal Sensor Co-Design for Robust Autonomous-Driving Segmentation
- 通过可微分管线学习最优彩色滤色阵列权重。
- 在KITTI-360和ACDC数据集上分别提升mIoU 0.017和0.023。
- 适合关注感知前端优化的自动驾驶研究者。
鲁棒感知是自动驾驶的基础，近期进展主要依赖模型规模扩展——如更大骨干网络、基础模型与多智能体协同融合。本文从上游提出不同问题：相机本身应测量什么？基于可微分的RAW到任务流水线，我们分解了传感器自由度对密集预测的影响。学习光谱彩色滤色阵列（CFA）权重是主要调控因素，在KITTI-360和ACDC上分别带来+0.017和+0.023的mIoU提升，优于固定相机。相反，点扩散函数（光学）联合设计导致性能下降（KITTI-360上-0.020 mIoU），这源于数据处理不等式限制了下游模型能恢复的任务信息量。噪声联合优化效果有限，且超出2×2的CFA块会持续恶化性能，因滤色器受限于三秩的sRGB输入。由于干预位于传感器端，收益对模型无关；我们在ACDC的雾、夜、雨、雪场景中验证了鲁棒性，并给出简洁建议：学习2×2 CFA权重，保持点扩散函数为恒等映射。
原文摘要 · Abstract (English)
Robust perception underpins autonomous driving, and most recent progress comes from scaling the model-larger backbones, foundation models, and cooperative multi-agent fusion. We pursue a complementary, upstream question: what should the camera itself measure? Using a differentiable RAW-to-task pipeline, we decompose which sensor degrees of freedom benefit dense prediction. Learning the spectral colour-filter-array (CFA) weights is the dominant lever, improving mIoU by +0.017 (KITTI-360) and +0.023 (ACDC) over a fixed camera. In contrast, point-spread-function (optics) co-design is net-negative (-0.020 mIoU on KITTI-360) - a consequence of the data-processing inequality, which also bounds the task information that any downstream model, however large or cooperative, can recover. Noise co-optimisation is marginal, and counter to intuition enlarging the CFA tile beyond 2x2 consistently hurts, as the filters are confined to the rank three sRGB input. Because the intervention is at the sensor, the gains are model-agnostic; we validate robustness on ACDC's fog, night, rain, and snow, and conclude with a simple recipe: learn the 2x2 CFA weights and keep an identity PSF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。