改变训练数据顺序竟能让模型用极少量数据高效学习，且隐藏信息可被精准捕捉。
The Order Is The Message
- 通过控制数据顺序，在极低采样率下实现高精度训练
- 仅用0.3%数据便达99.5%准确率，远超传统理论下限
- 顺序结构生成的傅里叶特征具普适性，可被多轮实验复现
在模运算（p = 9973）的受控实验中，仅改变训练样本顺序而保持其他条件不变，两种固定顺序策略分别在第487和第659轮达到99.5%测试准确率，训练集仅占输入空间的0.3%，远低于该任务在独立同分布（IID）顺序下的样本复杂度下限。相同数据下，IID基线经5,000轮训练后仅达0.30%准确率。对抗性构造的顺序完全抑制学习。通用模型能稳定构建出与顺序结构傅里叶对偶的频域表示，编码了单个训练样本中不存在的信息，且所有种子测试下均出现相同基本频率。
原文摘要 · Abstract (English)
In a controlled experiment on modular arithmetic ($p = 9973$), varying only example ordering while holding all else constant, two fixed-ordering strategies achieve 99.5\% test accuracy by epochs 487 and 659 respectively from a training set comprising 0.3\% of the input space, well below established sample complexity lower bounds for this task under IID ordering. The IID baseline achieves 0.30\% after 5{,}000 epochs from identical data. An adversarially structured ordering suppresses learning entirely. The generalizing model reliably constructs a Fourier representation whose fundamental frequency is the Fourier dual of the ordering structure, encoding information present in no individual training example, with the same fundamental emerging across all seeds tested regardless of initialization or training set composition. We discuss implications for training efficiency, the reinterpretation of grokking, and the safety risks of a channel that evades all content-level auditing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。