用小波域直接学习推理,不用神经网络也能达到接近Transformer的性能。
Wavelet Logic Machines: Learning and Reasoning in the Spectral Domain Without Neural Networks
- 在小波系数上做可学习的非线性变换,替代传统神经层。
- 3D去噪和GLUE任务上准确率达89.3%,参数少72%,内存降58%。
- 适合追求高效、可解释模型的视觉与语言研究者。
我们提出一个完全基于谱域的学习框架,彻底摒弃传统神经层,仅在小波域内操作。模型对小波系数施加可学习的非线性变换,包括软阈值化和增益-相位调制,并引入可微的小波基选择机制,支持Haar、Daubechies和Biorthogonal等小波族自适应处理。基于PyTorch实现,具备完整的3D支持,全程保持谱域流水线,无需空间卷积或注意力机制。在合成3D去噪和来自GLUE基准的自然语言任务(如SST-2情感分类)中,模型取得89.3%准确率,接近4层Transformer基线(90.1%),同时参数减少72%,峰值内存降低58%。由于谱稀疏性先验,早期收敛更快。相比Transformer的二次复杂度与大规模矩阵乘法,本方法采用线性时间小波变换和逐点非线性,显著降低推理开销。结果表明,基于原理的谱学习在视觉与语言任务中均具可行性,为无过参数化架构提供了新范式。
原文摘要 · Abstract (English)
We introduce a fully spectral learning framework that eliminates traditional neural layers by operating entirely in the wavelet domain. The model applies learnable nonlinear transformations, including soft-thresholding and gain-phase modulation, directly to wavelet coefficients. It also includes a differentiable wavelet basis selection mechanism, enabling adaptive processing using families such as Haar, Daubechies, and Biorthogonal wavelets. Implemented in PyTorch with full 3D support, the model maintains a spectral pipeline without spatial convolutions or attention. On synthetic 3D denoising and natural language tasks from the GLUE benchmark, including SST-2 sentiment classification, the model achieves 89.3 percent accuracy, close to a 4-layer Transformer baseline (90.1 percent), while using 72 percent fewer parameters and 58 percent less peak memory. Faster early convergence is observed due to spectral sparsity priors. In contrast to the quadratic complexity of self-attention and large matrix multiplications in Transformers, our approach uses linear-time wavelet transforms and pointwise nonlinearities, significantly reducing inference cost. This yields a compact, interpretable, and efficient alternative to neural models. Our results support the viability of principled spectral learning in both vision and language tasks, offering new directions for model design without overparameterized architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。