通过控制注意力分布，让Transformer生成与动作同步的文本描述。
Transformer with Controlled Attention for Synchronous Motion Captioning
- 设计注意力掩码与结构化损失，约束模型聚焦关键动作帧。
- 在KIT-ML和HumanML3D数据集上实现更优的时序对齐效果。
- 适合需要可解释性与动作-文本同步的应用场景。
本文解决一项挑战性任务——同步动作描述生成，旨在为人体动作序列生成与之时间对齐的语言描述。该任务在手语转录、无监督动作分割和时序定位等场景中具有重要意义。提出的方法通过调控Transformer的自注意力与交叉注意力分布，提升模型可解释性并实现时序对齐的文本生成。具体采用掩码策略与结构化损失，促使模型仅在对生成动词至关重要的帧上集中注意力，避免注意力图中信息混杂，并确保词元间注意力分布单调。由此，交叉注意力支持逐步生成与动作序列同步的文本。在两个公开基准数据集KIT-ML和HumanML3D上的评估表明，该方法性能优越。为便于可视化评估，代码仓库提供了一系列动态演示：https://github.com/rd20karim/Synch-Transformer。
原文摘要 · Abstract (English)
In this paper, we address a challenging task, synchronous motion captioning, that aim to generate a language description synchronized with human motion sequences. This task pertains to numerous applications, such as aligned sign language transcription, unsupervised action segmentation and temporal grounding. Our method introduces mechanisms to control self- and cross-attention distributions of the Transformer, allowing interpretability and time-aligned text generation. We achieve this through masking strategies and structuring losses that push the model to maximize attention only on the most important frames contributing to the generation of a motion word. These constraints aim to prevent undesired mixing of information in attention maps and to provide a monotonic attention distribution across tokens. Thus, the cross attentions of tokens are used for progressive text generation in synchronization with human motion sequences. We demonstrate the superior performance of our approach through evaluation on the two available benchmark datasets, KIT-ML and HumanML3D. As visual evaluation is essential for this task, we provide a comprehensive set of animated visual illustrations in the code repository: https://github.com/rd20karim/Synch-Transformer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。