用两个智能代理让大模型精准指导骨骼动作识别
SkeletonAgent: An Agentic Interaction Framework for Skeleton-based Action Recognition
- 设计问答与选择双代理，让大模型动态生成关键关节约束
- 在5个数据集上均超越现有方法，最高提升2.1%准确率
- 适合需要精细动作区分的智能监控、人机交互场景
基于骨架的动作识别近年越来越多地利用大语言模型（LLM）的语义先验来增强骨架表征。然而，传统方法中LLM与识别模型孤立运行，缺乏性能反馈，难以提供区分相似动作的关键线索。为此，我们提出SkeletonAgent框架，通过问答代理（Questioner）和选择代理（Selector）实现识别模型与LLM的协同。Questioner识别易混淆类别，并将其作为上下文输入给LLM以获取更精准的引导；Selector解析LLM输出，提取具体的关节级约束并反馈给识别器，实现细粒度跨模态对齐。在NTU RGB+D、NTU RGB+D 120、Kinetics-Skeleton、FineGYM和UAV-Human五个基准上的全面评估表明，SkeletonAgent持续优于现有最先进方法。代码已开源：https://github.com/firework8/SkeletonAgent。
原文摘要 · Abstract (English)
Recent advances in skeleton-based action recognition increasingly leverage semantic priors from Large Language Models (LLMs) to enrich skeletal representations. However, the LLM is typically queried in isolation from the recognition model and receives no performance feedback. As a result, it often fails to deliver the targeted discriminative cues critical to distinguish similar actions. To overcome these limitations, we propose SkeletonAgent, a novel framework that bridges the recognition model and the LLM through two cooperative agents, i.e., Questioner and Selector. Specifically, the Questioner identifies the most frequently confused classes and supplies them to the LLM as context for more targeted guidance. Conversely, the Selector parses the LLM's response to extract precise joint-level constraints and feeds them back to the recognizer, enabling finer-grained cross-modal alignment. Comprehensive evaluations on five benchmarks, including NTU RGB+D, NTU RGB+D 120, Kinetics-Skeleton, FineGYM, and UAV-Human, demonstrate that SkeletonAgent consistently outperforms state-of-the-art benchmark methods. The code is available at https://github.com/firework8/SkeletonAgent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。