HRM模型在无数据增强下对小图像分类表现不佳,不如简单CNN。
Exploring the Hierarchical Reasoning Model for Small Natural-Image Classification Without Augmentation
- 用两阶段Transformer模块和旋转位置编码构建层级推理模型
- 在CIFAR-10上仅达65.0%准确率,远低于基准CNN的77.2%
- 适合研究模型归纳偏置对小图像分类的影响
本文探讨了层级推理模型(HRM)在无数据增强的严格条件下是否可作为实用图像分类器。该模型采用两个Transformer式模块(f_L, f_H)、一步式(DEQ风格)训练、深度监督、旋转位置编码和RMSNorm。在MNIST、CIFAR-10和CIFAR-100上评估,使用相同优化器族(一周期预热后余弦衰减)与标签平滑。HRM在MNIST上表现良好(约98%测试准确率),但在小自然图像上过拟合严重:在CIFAR-10上经25轮训练达65.0%测试准确率,而两阶段卷积基线仅需约1/30时间即可达到77.2%;在CIFAR-100上,尽管训练准确率达91.5%,测试准确率仅29.7%,而相同卷积网络达45.3%测试准确率(训练准确率50.5%)。损失轨迹与错误分析显示优化正常,但模型缺乏足够的图像特定归纳偏置。结论为:当前形式的HRM在无增强的小图像分类中不具竞争力,但修改模型仍有改进空间。
原文摘要 · Abstract (English)
This paper asks whether the Hierarchical Reasoning Model (HRM) with the two Transformer-style modules $(f_L,f_H)$, one step (DEQ-style) training, deep supervision, Rotary Position Embeddings, and RMSNorm can serve as a practical image classifier. It is evaluated on MNIST, CIFAR-10, and CIFAR-100 under a deliberately raw regime: no data augmentation, identical optimizer family with one-epoch warmup then cosine-floor decay, and label smoothing. HRM optimizes stably and performs well on MNIST ($\approx 98\%$ test accuracy), but on small natural images it overfits and generalizes poorly: on CIFAR-10, HRM reaches 65.0\% after 25 epochs, whereas a two-stage Conv--BN--ReLU baseline attains 77.2\% while training $\sim 30\times$ faster per epoch; on CIFAR-100, HRM achieves only 29.7\% test accuracy despite 91.5\% train accuracy, while the same CNN reaches 45.3\% test with 50.5\% train accuracy. Loss traces and error analyses indicate healthy optimization but insufficient image-specific inductive bias for HRM in this regime. It is concluded that, for small-resolution image classification without augmentation, HRM is not competitive with even simple convolutional architectures as the HRM currently exist but this does not exclude possibilities that modifications to the model may allow it to improve greatly.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。