用博弈论方法融合多个视觉语言模型,提升自动驾驶多标签识别的准确性和可靠性。
Strategic Fusion of Vision Language Models: Shapley-Credited Context-Aware Dawid-Skene for Multi-Label Tasks in Autonomous Driving
- 基于博弈论分配模型信用,动态评估各模型在不同情境下的可信度。
- 相比单个模型,误报率降低23%,宏平均F1提升55%。
- 适合需要高可靠性的自动驾驶决策系统,可解释性强。
大型视觉语言模型(VLM)在自动驾驶系统中应用日益广泛,但幻觉问题限制了其在安全关键任务中的可靠性。本文提出一种基于博弈论的上下文感知戴维德-斯克恩融合方法(Shapley-credited Context-Aware Dawid-Skene with Agreement),用于对自车视角行车记录仪视频进行多标签理解。该方法从标注历史中学习每个模型、每类标签在特定上下文下的可靠性,并在推理时将各模型输出转换为受共识保护的对数似然比,结合上下文先验与基于谢尔普利值更新的公共声誉状态,生成校准后的可阈值后验概率。结果能有效放大可靠模型间的一致性,保留单一模型的独特正确信号,并适应分布漂移。为适配通用VLM,我们通过自动流程整合硬盘地面真值、车辆运动学数据及YOLOv11 + BoT-SORT追踪信息,构建了包含1000段真实行车记录视频的结构化数据集(含场景描述、操作建议、推理依据),并使用LoRA微调三个异构VLM。评估指标包括汉明距离、微观/宏观F1以及每视频平均延迟。实验表明,该方法相较最优单模型,汉明距离减少23%,宏平均F1提升55%,微平均F1提升47%,验证了VLM融合作为可校准、可解释、鲁棒的决策支持组件在自动驾驶流水线中的有效性。
原文摘要 · Abstract (English)
Large vision-language models (VLMs) are increasingly used in autonomous-vehicle (AV) stacks, but hallucination limits their reliability in safety-critical pipelines. We present Shapley-credited Context-Aware Dawid-Skene with Agreement, a game-theoretic fusion method for multi-label understanding of ego-view dashcam video. It learns per-model, per-label, context-conditioned reliabilities from labelled history and, at inference, converts each model's report into an agreement-guardrailed log-likelihood ratio that is combined with a contextual prior and a public reputation state updated via Shapley-based team credit. The result is calibrated, thresholdable posteriors that (i) amplify agreement among reliable models, (ii) preserve uniquely correct single-model signals, and (iii) adapt to drift. To specialise general VLMs, we curate 1,000 real-world dashcam clips with structured annotations (scene description, manoeuvre recommendation, rationale) via an automatic pipeline that fuses HDD ground truth, vehicle kinematics, and YOLOv11 + BoT-SORT tracking, guided by a three-step chain-of-thought prompt; three heterogeneous VLMs are then fine-tuned with LoRA. We evaluate with Hamming distance, Micro-Macro-F1, and average per-video latency. Empirically, the proposed method achieves a 23% reduction in Hamming distance, 55% improvement in Macro-F1, and 47% improvement in Micro-F1 when comparing with the best single model, supporting VLM fusion as a calibrated, interpretable, and robust decision-support component for AV pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。