用斐波那契结构替代位置编码，轻松重建高频信号
The Fibonacci Network: A Simple Alternative for Positional Encoding
- 构建分块斐波那契网络，每块输入前两块输出与原始输入
- 训练各块对应信号频率，可重建任意高频率成分
- 无需位置编码，适合对频率敏感的神经网络任务
基于坐标的多层感知机（MLP）在重建训练数据的高频部分时存在困难。传统解决方案是引入位置编码（PE），但其存在高频伪影，并引入额外超参数。本文认为在特定条件下，位置编码并非必需，通过更智能的网络架构设计与训练策略即可达到类似效果。我们发现简单MLP在输入半频和四分之一频信号时，能直接输出对应频率。据此设计分块斐波那契网络：每块输入为前两块输出与原始输入。通过分别训练各块对应信号频率，实验表明斐波那契网络可实现任意高频信号重建。
原文摘要 · Abstract (English)
Coordinate-based Multi-Layer Perceptrons (MLPs) are known to have difficulty reconstructing high frequencies of the training data. A common solution to this problem is Positional Encoding (PE), which has become quite popular. However, PE has drawbacks. It has high-frequency artifacts and adds another hyper-hyperparameter, just like batch normalization and dropout do. We believe that under certain circumstances PE is not necessary, and a smarter construction of the network architecture together with a smart training method is sufficient to achieve similar results. In this paper, we show that very simple MLPs can quite easily output a frequency when given input of the half-frequency and quarter-frequency. Using this, we design a network architecture in blocks, where the input to each block is the output of the two previous blocks along with the original input. We call this a {\it Fibonacci Network}. By training each block on the corresponding frequencies of the signal, we show that Fibonacci Networks can reconstruct arbitrarily high frequencies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。