用非线性连接器提升语言引导人体姿态估计精度
PoseLLM: Enhancing Language-Guided Human Pose Estimation with MLP Alignment
- 用两层GELU激活的MLP替代线性投影，增强视觉与文本特征融合
- 在COCO上达77.8 AP，比LocLLM高0.4 AP，零样本泛化能力强
- 轻量设计适合部署，适合需要精准姿态定位的多场景应用
人体姿态估计传统上依赖编码关键点先验的架构，限制了对新姿态或未见关键点的泛化能力。近期语言引导方法如LocLLM将关键点定位重构为视觉-语言任务，通过文本描述实现零样本泛化。然而，LocLLM的线性投影器无法捕捉复杂的空间-文本交互，影响定位精度。为此，我们提出PoseLLM，首个基于大语言模型（LLM）的姿态估计框架，将线性投影替换为非线性MLP视觉-语言连接器。该轻量级两层MLP结合GELU激活，实现分层跨模态特征转换，有效融合视觉图像块与文本关键点描述。仅在COCO数据上训练，PoseLLM在COCO验证集上取得77.8 AP，优于LocLLM的77.4 AP；同时在Human-Art和MPII数据集上保持强零样本泛化能力。结果表明，简单而强大的非线性连接器可显著提升定位精度而不牺牲泛化性，推动语言引导姿态估计的前沿进展。代码已开源。
原文摘要 · Abstract (English)
Human pose estimation traditionally relies on architectures that encode keypoint priors, limiting their generalization to novel poses or unseen keypoints. Recent language-guided approaches like LocLLM reformulate keypoint localization as a vision-language task, enabling zero-shot generalization through textual descriptions. However, LocLLM's linear projector fails to capture complex spatial-textual interactions critical for high-precision localization. To address this, we propose PoseLLM, the first Large Language Model (LLM)-based pose estimation framework that replaces the linear projector with a nonlinear MLP vision-language connector. This lightweight two-layer MLP with GELU activation enables hierarchical cross-modal feature transformation, enhancing the fusion of visual patches and textual keypoint descriptions. Trained exclusively on COCO data, PoseLLM achieves 77.8 AP on the COCO validation set, outperforming LocLLM by +0.4 AP, while maintaining strong zero-shot generalization on Human-Art and MPII. Our work demonstrates that a simple yet powerful nonlinear connector significantly boosts localization accuracy without sacrificing generalization, advancing the state-of-the-art in language-guided pose estimation. Code is available at https://github.com/Ody-trek/PoseLLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。