用量子启发机制提升语言模型性能，效果显著且推理时间仅略增
Quantum-Inspired Self-Attention in a Large Language Model
- 提出经典量子启发自注意力机制，融入GPT-1完整生成流程
- 在字符错误率上比标准自注意力好15.5倍，交叉熵损失低13倍
- 适合关注高效模型优化与量子启发计算的开发者与研究者
自然语言处理近年主要由基于Transformer的架构推动，其依赖自注意力机制建模序列中词元间关系。与此同时，量子自然语言处理领域也发展出量子自注意力机制，尝试利用量子原理解决语言理解与生成任务。本文提出一种经典量子启发自注意力（QISA）机制，并将其集成至GPT-1的完整自回归语言建模流程中。据我们所知，这是首次将此类机制应用于端到端语言建模，此前量子自注意力多局限于文本分类任务。实验表明，相比标准自注意力，QISA在字符错误率（15.5倍）、词错误率（4.7倍）和交叉熵损失（13倍）上表现更优，仅需2.6倍更长的推理时间。
原文摘要 · Abstract (English)
Recent advances in Natural Language Processing have been predominantly driven by transformer-based architectures, which rely heavily on self-attention mechanisms to model relationships between tokens in a sequence. Similarly, the field of Quantum Natural Language Processing, which seeks to leverage quantum principles to address challenges in language understanding and generation tasks, has seen the recent development of quantum self-attention mechanisms. We propose a classical quantum-inspired self-attention (QISA) mechanism and integrate it into the full autoregressive language modeling pipeline of GPT-1. To the best of our knowledge, this is the first integration of this kind, as previous quantum self-attention mechanisms have been primarily tested on text classification. In our experiments, QISA achieves better performance when compared to standard self-attention on the metrics character error rate ($15.5\times$ better), word error rate ($4.7 \times $) and cross-entropy loss ($13 \times$). This is achieved while only requiring a $ 2.6\times$ longer inference time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。