改进Transformer注意力机制，提升小数据下家电负荷识别精度
Scaled and Inter-token Relation Enhanced Transformer for Sample-restricted Residential NILM
- 移除注意力矩阵对角线，强化跨标记关系捕捉
- 动态温度调节防止注意力过度平滑，提升敏感度
- 在小数据集上比现有方法高10-15%的F1分数
Transformer凭借自注意力机制在多个领域表现优异，但小规模数据训练时，标准注意力易导致注意力分数过度平滑，并过度关注标记内关系，削弱对关键跨标记依赖的捕捉能力，影响非侵入式负荷监测（NILM）任务性能。为此，本文提出一种新型Transformer架构，包含两项创新：跨标记关系增强机制通过移除相似性矩阵的对角线项，提升对跨标记关系的关注；动态温度调制机制作为可学习参数，在训练中自适应调整注意力锐度，防止过度平滑并增强对标记间关系的敏感性。我们在REDD数据集上验证了该方法，结果表明其在多种电器类型上均比原始Transformer和当前最优模型提升10-15%的F1分数，验证了其在小数据场景下的有效性。
原文摘要 · Abstract (English)
Transformers have demonstrated exceptional performance across various domains due to their self-attention mechanism, which captures complex relationships in data. However, training on smaller datasets poses challenges, as standard attention mechanisms can over-smooth attention scores and overly prioritize intra-token relationships, reducing the capture of meaningful inter-token dependencies critical for tasks like Non-Intrusive Load Monitoring (NILM). To address this, we propose a novel transformer architecture with two key innovations: inter-token relation enhancement and dynamic temperature tuning. The inter-token relation enhancement mechanism removes diagonal entries in the similarity matrix to improve attention focus on inter-token relations. The dynamic temperature tuning mechanism, a learnable parameter, adapts attention sharpness during training, preventing over-smoothing and enhancing sensitivity to token relationships. We validate our method on the REDD dataset and show that it outperforms the original transformer and state-of-the-art models by 10-15\% in F1 score across various appliance types, demonstrating its efficacy for training on smaller datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。