为联邦学习中的异常检测提出注意力层专用聚合方法。
Attention, Anomalies! Handling Attention Layers in Unsupervised Federated Outlier Detection
- 设计引导式聚合策略，专门处理内存增强自编码器的注意力层。
- 在非独立同分布数据下提升模型性能，尤其适用于资源受限场景。
- 使浅层自编码器也能实现高效异常检测，适合边缘设备部署。
注意力层是当前最强大模型的核心组件，依赖其提供的上下文知识，大规模模型得以运行。注意力机制不仅用于大语言模型，也广泛应用于记忆增强自编码器（MemAE），用于无监督表征学习与异常检测任务。已有研究表明，在集中式学习中注意力机制能显著提升模型效果。然而，在联邦学习（FL）中，针对MemAE模型缺乏专门的聚合技术。本文深入分析了MemAE架构特性，提出新型引导式聚合方法，有效解决联邦学习环境下的模型聚合问题。实验表明，该方法在非独立同分布（non-IID）数据和数据不平衡场景下表现更稳健，适用于大量边缘节点部署。即使对于极浅层自编码器，该方法仍可显著提升性能，使其在资源受限环境中具备实用价值。
原文摘要 · Abstract (English)
Attention layers are the backbone of today's most powerful and impactful models. Models with multi-million and billion parameters rely on contextual knowledge provided by attention layers. However, their use goes well beyond just being the core component of large language models. One particularly interesting application is in Memory Augmented Autoencoders (MemAE), specifically for unsupervised representation learning in outlier detection tasks. It was shown that attention helps these models be more effective in centralized learning scenarios. Our work aims to address the lack of specialized aggregation techniques in Federated Learning (FL) when it comes to MemAE models. In this paper we analyze the intricacies of the architecture behind Memory Augmented Autoencoders, and propose novel, guided approaches to effectively aggregate these models in federated scenarios. We demonstrate our approach on non-IID datasets and show that these novel aggregation schemes are more robust when dealing with numerous edge nodes in environments with unbalanced datasets, specifically for unsupervised anomaly detection scenarios. This approach improves the performance of even very shallow autoencoders, allowing them to be used in resource constrained environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。