用量子电路替换经典模型的前馈层,大幅减少参数量且提升数据效率。
QFFN-BERT: An Empirical Study of Depth, Performance, and Data Efficiency in Hybrid Quantum-Classical Transformers
- 将量子电路嵌入BERT的前馈网络,通过旋转与纠缠设计提升表达能力。
- 在全量数据下准确率达基线102.0%,前馈层参数减少超99%。
- 少样本场景表现优异,适合追求高效低参数模型的研究者。
参数化量子电路(PQC)近年被视作增强神经架构表达力的有前景组件。本文提出QFFN-BERT,一种混合量子-经典Transformer模型,将紧凑型BERT中前馈网络(FFN)模块替换为基于PQC的层。该设计源于FFN占标准Transformer编码器块约三分之二参数的事实。不同于以往主要将PQC集成于自注意力模块,本工作聚焦于FFN,系统研究了PQC深度、表达力与可训练性间的权衡。最终架构引入残差连接、$R_Y$和$R_Z$旋转及交替纠缠策略,以保障训练稳定性和高表达力。实验在经典模拟器上进行,基于SST-2与DBpedia基准测试,得出两个关键结论:第一,在全量数据设置下,优化后的QFFN-BERT达到基线102.0%的准确率,超越经典模型同时将FFN相关参数减少超过99%;第二,模型在少样本学习场景中持续保持竞争力,验证其卓越的数据效率。消融实验进一步显示,未优化的PQC无法有效学习,证实当与深度学习原理协同设计时,PQC可作为强大且参数高效的经典FFN替代方案。
原文摘要 · Abstract (English)
Parameterized quantum circuits (PQCs) have recently emerged as promising components for enhancing the expressibility of neural architectures. In this work, we introduce QFFN-BERT, a hybrid quantum-classical transformer where the feedforward network (FFN) modules of a compact BERT variant are replaced by PQC-based layers. This design is motivated by the dominant parameter contribution of FFNs, which account for approximately two-thirds of the parameters within standard Transformer encoder blocks. While prior studies have primarily integrated PQCs into self-attention modules, our work focuses on the FFN and systematically investigates the trade-offs between PQC depth, expressibility, and trainability. Our final PQC architecture incorporates a residual connection, both $R_Y$ and $R_Z$ rotations, and an alternating entanglement strategy to ensure stable training and high expressibility. Our experiments, conducted on a classical simulator, on the SST-2 and DBpedia benchmarks demonstrate two key findings. First, a carefully configured QFFN-BERT achieves up to 102.0% of the baseline accuracy, surpassing its classical counterpart in a full-data setting while reducing FFN-specific parameters by over 99%. Second, our model exhibits a consistent and competitive edge in few-shot learning scenarios, confirming its potential for superior data efficiency. These results, supported by an ablation study on a non-optimized PQC that failed to learn, confirm that PQCs can serve as powerful and parameter-efficient alternatives to classical FFNs when co-designed with foundational deep learning principles.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。