提出在线纠错恢复评估基准,测试模型在分布漂移下的实时适应能力。
OCRR: A Benchmark for Online Correction Recovery under Distribution Shift

- 设计矫正流评测框架,量化模型在纠错中的新类与旧类准确率变化
- 所提底座系统在银行和客服数据集上同时实现88.7%新类准确率与95.4%旧类保持率
- 适合关注持续学习、在线更新与低存储场景的系统设计者
静态基准仅衡量训练冻结后的模型性能。真实系统面临分布漂移:新类别、改写查询与数据漂移,必须通过用户纠错在线恢复。现有基准无法评估纠错流下的恢复速度。本文提出OCRR(在线纠错恢复率)基准:将语料库流式输入分类系统,对错误预测施加理想或随机纠错,报告新类别准确率与原分布准确率随纠错次数的变化曲线。评估了九种基线算法(五类)及七种受限存储的底座变体,包括标准在线学习(River)、持续学习方法(EWC, A-GEM, LwF)、检索/参数混合(kNN-LM)、15亿参数编码器的参数高效微调(LoRA on DeBERTa-v3-large)以及哈希链追加式底座(Substrate)。在Banking77与CLINC150数据集上,理想与稀疏纠错策略下,底座系统是唯一同时实现新类准确率88.7±2.9%与原分布准确率95.4±0.8%的系统,较次优持续学习基线提升32.6个百分点,较LoRA-on-DeBERTa-v3-large提升84.6个百分点。此外,即使近似最近邻召回@5从0.69降至0.23(语料规模从1万增至1000万),分类准确率仍稳定在99%,表明底座的边界投票机制对检索误差具有鲁棒性,而传统顶K召回指标未能预判此现象。代码与数据见https://github.com/adriangrassi/ocrr-benchmark。
原文摘要 · Abstract (English)
Static benchmarks measure a model frozen at training time. Real systems face distribution shift: new categories, paraphrased queries, drift: and must recover online via user corrections. No existing benchmark measures recovery speed under correction streams. We introduce OCRR (Online Correction Recovery Rate): a benchmark that streams a corpus through a classification system, applies oracle or stochastic corrections to wrong predictions, and reports two curves: novel-class accuracy and original-distribution accuracy versus correction count. We evaluate the substrate alongside nine baseline algorithms from five families plus seven bounded-storage variants of the substrate for the Pareto sweep, including standard online-learning baselines (river), continual-learning methods (EWC, A-GEM, LwF), retrieval/parametric hybrids (kNN-LM), parameter-efficient fine-tuning of a 1.5 B-parameter encoder (LoRA on DeBERTa-v3-large), and a hash-chained append-only substrate (Substrate). On Banking77 and CLINC150, under oracle and sparse correction policies, the substrate is the only system that simultaneously recovers novel-class accuracy (88.7 +/- 2.9 %) and retains original-distribution accuracy (95.4 +/- 0.8 %) beating the next-best published continual-learning baseline by 32.6 percentage points at equal memory budget, and beating LoRA-on-DeBERTa-v3-large by 84.6 percentage points on retention. We further find that classification accuracy remains stable at 99 % even as approximate-nearest-neighbour recall@5 degrades from 0.69 to 0.23 across 10 k to 10 M corpus scales, suggesting the substrate's margin-band majority vote is robust to retrieval imperfection in a way that pure top-k recall metrics do not predict. Code and data are available at https://github.com/adriangrassi/ocrr-benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。