评估模型从零学习时的累计错误，揭示适应能力差异。
Guess-and-Learn (G&L): Measuring the Cumulative Error Cost of Cold-Start Adaptation
- 通过逐步标注数据并记录错误，量化模型冷启动适应过程中的总失误数。
- 实验显示小模型初期错误少，预训练效果因任务而异，当前模型普遍远超理想基准。
- 适合关注模型早期学习可靠性与适应效率的研究者。
机器学习模型的评估通常只关注最终准确率，忽略了适应成本：即从零开始学习过程中产生的累计错误。本文提出的猜-学（G&L）v1.0协议通过测量冷启动适应性——模型在顺序标注未标记数据集时的总错误数——填补了这一空白。每一步中，学习者选择一个样本，预测其标签，获得真实标签后，在在线（单样本）或批量（延迟）模式下更新参数。由此产生的误差轨迹揭示了适应速度、选择质量与偏差等动态特征，这些是终点指标无法捕捉的。G&L定义了四种路径（从零/预训练 × 在线/批量），以分离初始化和更新频率的影响。我们形式化了该协议，将其与经典误判界理论关联，并为MNIST数据集估算了一条启发式‘理想参考带’作为合理性参照。在MNIST和AG News上的基线实验涵盖感知机、k-NN、CNN、ResNet-50以及预训练Transformer（ViT-B/16、BERT-base），结果表明早期适应效率存在系统性差异：小型模型能以更少初始错误适应，而预训练收益随领域变化。跨所有设置，当前模型均显著高于理想参考带，凸显出适应能力差距。通过量化早期学习的错误代价，G&L补充了传统基准，提供了一个可复现的框架，用于开发不仅最终准确，且从首例起就可靠的模型。
原文摘要 · Abstract (English)
Evaluation of machine learning models typically emphasizes final accuracy, overlooking the cost of adaptation: the cumulative errors incurred while learning from scratch. Guess-and- Learn (G&L) v1.0 addresses this gap by measuring cold-start adaptability - the total mistakes a model makes while sequentially labeling an unlabeled dataset. At each step, the learner selects an instance, predicts its label, receives the ground truth, and updates parameters under either online (per-sample) or batch (delayed) mode. The resulting error trajectory exposes adaptation speed, selection quality, and bias - dynamics invisible to endpoint metrics. G&L defines four tracks (Scratch/Pretrained $\times$ Online/Batch) to disentangle the effects of initialization and update frequency. We formalize the protocol, relate it to classical mistake-bound theory, and estimate a heuristic "oracle reference band" for MNIST as a plausibility reference. Baseline experiments on MNIST and AG News, spanning classical methods (Perceptron, k-NN), convolutional architectures (CNN, ResNet-50), and pretrained transformers (ViT-B/16, BERT-base), reveal systematic differences in early-phase efficiency: smaller models can adapt with fewer initial errors, while pretraining benefits vary by domain. Across settings, current models remain well above the oracle band, highlighting an adaptability gap. By quantifying the mistake cost of early learning, G&L complements conventional benchmarks and provides a reproducible framework for developing learners that are not only accurate in the limit but also reliable from the first examples.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。