用语言引导的图结构提升人体姿态估计，尤其在遮挡场景下表现更好。
Parse Graph-Based Visual-Language Interaction for Human Pose Estimation
- 构建视觉-语言解析图，分层融合局部与全局信息。
- 在COCO和PoseTrack上平均精度提升1.2%~2.3%，遮挡区域响应更强。
- 适合需要高鲁棒性姿态估计的自动驾驶、人机交互场景。
解析图通过整合上下文和层次结构提升人体姿态估计（HPE）性能，但现有方法多聚焦单模态建模，忽视多模态融合潜力。语言蕴含丰富的先验知识，如空间关系，可帮助推断遮挡区域，但现有全局特征融合方式削弱了遮挡区域响应，导致对齐与定位失败。为此，本文提出基于解析图的视觉-语言交互模型PGVL，核心为引导模块（GM）。PGVL中，低层节点关注局部特征，保持遮挡区域响应；高层节点整合全局特征，推断不可见部位。GM使高语义节点引导低语义节点的特征更新，实现有效信息融合。模型包含自顶向下分解与自底向上重构过程：先构建模态专属解析图，再通过递归双向交叉注意力并经由GM净化。所提网络在主流姿态估计数据集上验证，性能显著优于基线，在COCO和PoseTrack上平均精度提升1.2%~2.3%。代码将公开。
原文摘要 · Abstract (English)
Parse graphs boost human pose estimation (HPE) by integrating context and hierarchies, yet prior work mostly focuses on single modality modeling, ignoring the potential of multimodal fusion. Notably, language offers rich HPE priors like spatial relations for occluded scenes, but existing visual-language fusion via global feature integration weakens occluded region responses and causes alignment and location failures. To address this issue, we propose Parse Graph-based Visual-Language interaction (PGVL) with a core novel Guided Module (GM). In PGVL, low-level nodes focus on local features, maximizing the maintenance of responses in occluded areas and high-level nodes integrate global features to infer occluded or invisible parts. GM enables high semantic nodes to guide the feature update of low semantic nodes that have undergone cross attention. It ensuring effective fusion of diverse information. PGVL includes top-down decomposition and bottom-up composition. In the first stage, modality specific parse graphs are constructed. Next stage. recursive bidirectional cross-attention is used, purified by GM. We also design network based on PGVL. The PGVL and our network is validated on major pose estimation datasets. We will release the code soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。