解决动作质量评估中传感器缺失问题,提升系统鲁棒性。
BriMA: Bridged Modality Adaptation for Multi-Modal Continual Action Quality Assessment
- 用记忆引导的桥梁模块重建缺失模态数据
- 在三种数据集上平均相关性提升6-8%,误差降低12-15%
- 适合实际部署中模态不稳定场景的持续学习应用
动作质量评估(AQA)旨在评分动作执行质量,广泛应用于体育分析、康复评估和人类技能评价。多模态AQA通过融合视觉与运动学线索取得显著进展,但真实场景中常出现模态非平稳缺失,如传感器故障或标注空缺导致某些模态时断时续。现有持续学习方法忽略此问题,假设所有模态始终完整稳定,限制了实用性。为此,我们提出桥接模态适应(BriMA),一种应对模态缺失条件下的多模态持续AQA方法。BriMA包含一个记忆引导的桥梁插补模块,利用任务无关与任务特定表示重建缺失模态;以及一个模态感知回放机制,根据模态失真与分布漂移优先选择信息量高的样本。在三个代表性多模态AQA数据集(RG、Fis-V、FS1000)上的实验表明,BriMA在不同模态缺失条件下均显著提升性能,平均相关性提高6–8%,误差降低12–15%。结果证明了其向真实部署约束下鲁棒多模态AQA系统迈进的重要一步。
原文摘要 · Abstract (English)
Action Quality Assessment (AQA) aims to score how well an action is performed and is widely used in sports analysis, rehabilitation assessment, and human skill evaluation. Multi-modal AQA has recently achieved strong progress by leveraging complementary visual and kinematic cues, yet real-world deployments often suffer from non-stationary modality imbalance, where certain modalities become missing or intermittently available due to sensor failures or annotation gaps. Existing continual AQA methods overlook this issue and assume that all modalities remain complete and stable throughout training, which restricts their practicality. To address this challenge, we introduce Bridged Modality Adaptation (BriMA), an innovative approach to multi-modal continual AQA under modality-missing conditions. BriMA consists of a memory-guided bridging imputation module that reconstructs missing modalities using both task-agnostic and task-specific representations, and a modality-aware replay mechanism that prioritizes informative samples based on modality distortion and distribution drift. Experiments on three representative multi-modal AQA datasets (RG, Fis-V, and FS1000) show that BriMA consistently improves performance under different modality-missing conditions, achieving 6--8\% higher correlation and 12--15\% lower error on average. These results demonstrate a step toward robust multi-modal AQA systems under real-world deployment constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。