arXiv:2411.18428cs.LGcs.AI2024-11KDD被引 29

融合道路与图像多模态信息,提升路径表示的准确性与泛化能力。

MM-Path: Multi-modal, Multi-granularity Path Representation Learning -- Extended Version

  • 通过多粒度对齐策略关联道路节点、路段与图像区域
  • 在两个真实世界数据集上实现优于基线模型的路径表征性能
  • 适合交通智能、地图理解等需要多源信息融合的研究者

在智能交通领域,构建有效的路径表示日益重要。尽管预训练路径表示学习模型已取得进展,但多数仅关注单一模态(如道路网络)的拓扑结构,忽略了与路径相关的图像(如遥感图像)所蕴含的几何和上下文特征。如同人类认知,融合多模态信息可提供更全面视角,提升表示精度与泛化能力。然而,信息粒度差异阻碍了基于道路网络的路径(道路路径)与基于图像的路径(图像路径)之间的语义对齐,且多模态数据异质性给有效融合带来挑战。本文提出一种新型多模态、多粒度路径表示学习框架(MM-Path),通过整合道路路径与图像路径的多模态信息,学习通用路径表示。为增强多模态对齐,设计多粒度对齐策略,系统关联节点、道路子路径与道路路径对应的图像块,确保局部细节与全局上下文同步。为应对多模态异质性,引入基于图的跨模态残差融合组件,实现不同模态与粒度间的信息全面融合。在两个大规模真实世界数据集上,针对两种下游任务开展大量实验,验证了MM-Path的有效性。代码已公开:https://github.com/decisionintelligence/MM-Path。

原文摘要 · Abstract (English)

Developing effective path representations has become increasingly essential across various fields within intelligent transportation. Although pre-trained path representation learning models have shown improved performance, they predominantly focus on the topological structures from single modality data, i.e., road networks, overlooking the geometric and contextual features associated with path-related images, e.g., remote sensing images. Similar to human understanding, integrating information from multiple modalities can provide a more comprehensive view, enhancing both representation accuracy and generalization. However, variations in information granularity impede the semantic alignment of road network-based paths (road paths) and image-based paths (image paths), while the heterogeneity of multi-modal data poses substantial challenges for effective fusion and utilization. In this paper, we propose a novel Multi-modal, Multi-granularity Path Representation Learning Framework (MM-Path), which can learn a generic path representation by integrating modalities from both road paths and image paths. To enhance the alignment of multi-modal data, we develop a multi-granularity alignment strategy that systematically associates nodes, road sub-paths, and road paths with their corresponding image patches, ensuring the synchronization of both detailed local information and broader global contexts. To address the heterogeneity of multi-modal data effectively, we introduce a graph-based cross-modal residual fusion component designed to comprehensively fuse information across different modalities and granularities. Finally, we conduct extensive experiments on two large-scale real-world datasets under two downstream tasks, validating the effectiveness of the proposed MM-Path. The code is available at: https://github.com/decisionintelligence/MM-Path.

路径表示多模态学习交通智能

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。