无需分词和注意力机制，直接用字节处理文本分类，性能超预训练模型。
Kathleen: Oscillator-Based Byte-Level Text Classification Without Tokenization or Attention
- 基于振荡器的频率域处理，实现字节级序列建模，时间空间复杂度均为O(L)。
- 在IMDB、AG News、SST-2上分别达88.5%、92.4%、85.8%准确率，超越预训练基线。
- 仅469K参数，无预训练，适合资源受限场景下的高效文本分类任务。
我们提出Kathleen，一种直接对原始UTF-8字节进行文本分类的架构，无需分词与注意力机制，参数少于47万。其核心包括：(1) 循环振荡器组——具时序记忆的阻尼正弦卷积，实现O(L)序列处理；(2) FFT-Rotate波表编码器，用单个256维可学习向量映射全部256个字节值；(3) 相位谐波——仅含6个可学习相位参数的正弦非线性，提升精度+2.6%，参数占比<0.001%；(4) 内容依赖混响机制，衰减速率由输入内容与位置偏置向量联合决定；(5) 基于音调和谐/不和谐干扰通道的词级模块序列器。从初始73.3万参数基线（Kathleen-Clean）迭代优化至当前版本Kathleen-V9（46.9万参数），完全去除预训练的同时提升性能。Kathleen-V9在IMDB上达88.5%±0.2%，在AG News上达92.4%±0.2%，在SST-2上达85.8%±0.5%（三种子平均），各项指标均匹配或超越预训练基线，参数减少36%。在SST-2上较预训练基线绝对提升2.5%。整个模型以O(L)时间和内存处理序列。
原文摘要 · Abstract (English)
We present Kathleen, a text classification architecture that operates directly on raw UTF-8 bytes using frequency-domain processing -- requiring no tokenizer, no attention mechanism, and under 470K parameters. Kathleen introduces several novel components: (1) RecurrentOscillatorBanks -- damped sinusoid convolutions with temporal memory for O(L) sequence processing; (2) an FFT-Rotate Wavetable Encoder that maps all 256 byte values using a single learnable vector (256 floats); (3) PhaseHarmonics -- a sinusoidal non-linearity with just 6 learnable phase parameters (+2.6% accuracy, <0.001% of model parameters); (4) Content-Dependent Reverb with Positional Decay Modulation -- a temporal memory mechanism whose decay rate is jointly conditioned on input content and a learned position-indexed bias vector; (5) Token-Level Module Sequencer with consonance and dissonance interference channels. Through iterative architecture evolution from an initial 733K-parameter baseline (Kathleen-Clean) to the current Kathleen-V9 (469K parameters), we demonstrate that pretraining can be entirely eliminated while improving accuracy. Kathleen-V9 achieves 88.5% +/- 0.2% on IMDB, 92.4% +/- 0.2% on AG News, and 85.8% +/- 0.5% on SST-2 (3-seed averages) -- matching or exceeding the pretrained baseline on all benchmarks with 36% fewer parameters. On SST-2, the improvement is +2.5% absolute over the pretrained predecessor. Kathleen processes sequences in O(L) time and memory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。