提出正交二次补丁，让视觉Transformer的前馈网络更高效地补充新信息。
Orthogonal Quadratic Complements for Vision Transformer Feed-Forward Networks
- 设计正交投影机制，确保辅助分支只添加主分支未覆盖的信息。
- 在CIFAR-100上提升准确率至65.59，TinyImageNet上达51.88，优于基线。
- 适合关注模型效率与特征解耦的计算机视觉研究者。
近期的双线性前馈替代方法虽能显著提升视觉Transformer的精度，但常混淆强二阶交互与冗余增加两种效应。本文提出正交二次补丁（OQC），通过构建低秩二次辅助分支，并将其显式投影到主分支的正交补空间后再注入，确保仅补充原始表示未涵盖的信息。进一步提出高效低秩实现（OQC-LR）及门控扩展（OQC-static与OQC-dynamic）。在参数匹配的Deep-ViT与CIFAR-100协议下，全量OQC将AFBO基线从64.25 ± 0.22提升至65.59 ± 0.22；OQC-LR达到65.52 ± 0.25，且速度-精度权衡更优。在TinyImageNet上，门控版本OQC-dynamic达51.88 ± 0.32，较基线50.45 ± 0.21提升1.43点，优于所有无门控变体。机制分析显示，投影后辅助-主分支重叠接近零，且表征几何与类别分离度均改善。全系列方法在两个数据集上均具一致泛化能力。
原文摘要 · Abstract (English)
Recent bilinear feed-forward replacements for vision transformers can substantially improve accuracy, but they often conflate two effects: stronger second-order interactions and increased redundancy relative to the main branch. We study a complementary design principle in which auxiliary quadratic features contribute only information not already captured by the dominant hidden representation. To this end, we propose Orthogonal Quadratic Complements (OQC), which construct a low-rank quadratic auxiliary branch and explicitly project it onto the orthogonal complement of the main branch before injection. We further study an efficient low-rank realization (OQC-LR) and gated extensions (OQC-static and OQC-dynamic). Under a parameter-matched Deep-ViT and CIFAR-100 protocol with a fixed penultimate residual readout, full OQC improves an AFBO baseline from 64.25 +/- 0.22 to 65.59 +/- 0.22, while OQC-LR reaches 65.52 +/- 0.25 with a substantially better speed-accuracy tradeoff. On TinyImageNet, the gated extension OQC-dynamic achieves 51.88 +/- 0.32, improving the baseline (50.45 +/- 0.21) by 1.43 points and outperforming all ungated variants. Mechanism analyses show near-zero post-projection auxiliary-main overlap together with improved representation geometry and class separation. The full family, including both ungated and gated variants, generalizes consistently across both datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。