RoMAE用旋转位置编码统一处理不规则时序，无需特殊结构即可跨模态高效学习。
Rotary Masked Autoencoders are Versatile Learners
- 基于旋转位置编码实现连续位置插值，无需时序专用结构
- 在ELAsTiCC挑战赛上超越专用时序模型，多模态性能稳定
- 揭示嵌入位置会破坏旋转编码的相对位置特性
将Transformer应用于不规则时序数据通常需要对基础架构进行特殊化设计，导致计算开销增加和方法复杂度上升。我们提出旋转掩码自编码器（RoMAE），利用流行的旋转位置编码（RoPE）处理连续位置信息。RoMAE是掩码自编码器（MAE）的扩展，可在不引入时序特定结构的前提下，实现多维连续位置信息的插值与表示学习。我们在多种模态上验证了其性能，包括不规则多变量时序、图像和音频，结果显示，在DESC ELAsTiCC挑战赛等困难数据集上，RoMAE优于专用时序架构，同时在其他模态上保持了MAE的常规性能。此外，我们研究了RoMAE重构嵌入位置的能力，发现将学习到的位置嵌入加入输入序列会破坏RoPE的相对位置特性。
原文摘要 · Abstract (English)
Applying Transformers to irregular time-series typically requires specializations to their baseline architecture, which can result in additional computational overhead and increased method complexity. We present the Rotary Masked Autoencoder (RoMAE), which utilizes the popular Rotary Positional Embedding (RoPE) method for continuous positions. RoMAE is an extension to the Masked Autoencoder (MAE) that enables interpolation and representation learning with multidimensional continuous positional information while avoiding any time-series-specific architectural specializations. We showcase RoMAE's performance on a variety of modalities including irregular and multivariate time-series, images, and audio, demonstrating that RoMAE surpasses specialized time-series architectures on difficult datasets such as the DESC ELAsTiCC Challenge while maintaining MAE's usual performance across other modalities. In addition, we investigate RoMAE's ability to reconstruct the embedded continuous positions, demonstrating that including learned embeddings in the input sequence breaks RoPE's relative position property.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。