arXiv:2502.00843cs.CV2025-02被引 8

用视觉语言模型+持续学习,让自动驾驶问答系统不遗忘旧知识。

VLM-Assisted Continual learning for Visual Question Answering in Self-Driving

  • 结合VLM与选择性记忆回放、知识蒸馏,防止新任务覆盖旧知识。
  • 在DriveLM数据集上性能提升20.11%至35.16%,显著减少遗忘。
  • 适合研究自动驾驶多任务学习与模型长期适应性的学者。

本文提出一种融合视觉语言模型(VLM)与持续学习的新方法,用于解决自动驾驶中的视觉问答(VQA)任务。在自动驾驶中,VQA对理解环境至关重要,但传统模型在顺序学习感知、预测、规划等新任务时易出现灾难性遗忘。为此,我们设计了一种新框架,结合选择性记忆回放、知识蒸馏与任务特定投影层正则化。知识蒸馏使预训练模型作为“教师”指导后续任务,减少遗忘;任务特定投影层通过计算特征表示的差异损失,保持学习连续性,降低任务间偏移。在DriveLM数据集上的实验表明,该框架在多个指标上性能提升20.11%至35.16%,验证了其在增强VQA系统鲁棒性与可靠性方面的有效性。代码将公开。

原文摘要 · Abstract (English)

In this paper, we propose a novel approach for solving the Visual Question Answering (VQA) task in autonomous driving by integrating Vision-Language Models (VLMs) with continual learning. In autonomous driving, VQA plays a vital role in enabling the system to understand and reason about its surroundings. However, traditional models often struggle with catastrophic forgetting when sequentially exposed to new driving tasks, such as perception, prediction, and planning, each requiring different forms of knowledge. To address this challenge, we present a novel continual learning framework that combines VLMs with selective memory replay and knowledge distillation, reinforced by task-specific projection layer regularization. The knowledge distillation allows a previously trained model to act as a "teacher" to guide the model through subsequent tasks, minimizing forgetting. Meanwhile, task-specific projection layers calculate the loss based on the divergence of feature representations, ensuring continuity in learning and reducing the shift between tasks. Evaluated on the DriveLM dataset, our framework shows substantial performance improvements, with gains ranging from 20.11% to 35.16% across various metrics. These results highlight the effectiveness of combining continual learning with VLMs in enhancing the resilience and reliability of VQA systems in autonomous driving. We will release our source code.

视觉问答持续学习自动驾驶VLM

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。