用视觉线索预测行人过街动作,无需额外感知模块
Unified Vision-Centric Pedestrian Crossing Action Prediction via Adaptive Patch Projection and Proactive Spatial Rectification

- 通过自适应补丁映射实现高效视觉编码
- 在多个数据集上超越纯视觉方法,接近多源融合效果
- 适合关注视觉主导行人行为预测的研究者
视觉线索对行人过街动作预测有重要价值,但在缺乏帧级外部感知的情况下,从视频中获取稳定的目标中心表示仍具挑战。现有方法多依赖额外感知模块或多源信息融合,使纯视觉设置的可靠性存疑。为此,我们提出ViCross框架,基于多模态大语言模型,在仅依赖首帧目标初始化的前提下,实现无外部感知模块的视觉中心推理。尽管多模态大模型具备强大视觉理解能力,直接应用于视觉中心动作预测面临两大挑战:一是高分辨率输入与密集视觉标记化需大量计算,难以全帧编码;二是时空先验缺失导致跨帧推理不一致。ViCross通过可变分辨率补丁映射模块实现高效标记分配,同时保留关键行人细节;并通过空间约束增强策略,捕获历史运动、未来位置和动作语义,实现训练阶段的主动空间修正。大量实验表明,ViCross在视觉中心预测设定下显著提升性能,在部分场景下媲美多源融合方法。代码已开源。
原文摘要 · Abstract (English)
Vision cues are available and informative for pedestrian action prediction, but obtaining stable target-centric representations from video frames remains challenging without frame-level external perception cues. Thus, most methods rely on additional perception modules or multi-source information fusion, leaving the reliability of vision-centric setting an open question. To this end, we propose ViCross, a vision-centric pedestrian crossing action prediction framework powered by multimodal large language models, which maintains target-centric reasoning from video frames without additional perception modules beyond first-frame target initialization. While multimodal large language models exhibit strong visual understanding, applying them directly to vision-centric action prediction faces two challenges. First, accurately perceiving target pedestrians often requires high resolution inputs and dense visual tokenization, making full-frame encoding computationally prohibitive. ViCross tackles this with Variable Resolution Patch Mapping module for efficient token allocation while preserving key pedestrian details. Second, missing spatiotemporal priors hinder consistent cross frame reasoning. ViCross mitigates this with a Spatial Constraint Enhancement Strategy that captures past motion, future locations, and action semantics for training-time proactive spatial rectification. Extensive experiments show that ViCross delivers clear gains in vision-centric prediction settings and is competitive with multi-source fusion approaches in several settings. Code is available at https://github.com/2tianyao1/ViCross.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。