提出学习泛化光谱,量化模型从单个样本中迁移知识的范围。
The Generalization Spectrum: A Chromatographic Approach to Evaluating Learning Algorithms

- 构建多层级测试变体,按迁移距离系统评估模型泛化能力。
- 强化学习比指令微调更高效实现跨任务迁移,提示依赖性强于抽象。
- 发现局部性能提升未必扩大泛化范围,自蒸馏可能削弱远距离迁移。
传统评估仅关注模型在独立同分布测试集上的最终表现,将学习简化为单一得分,掩盖了关键问题:从某个例子中学到的知识能推广到多少其他例子?这种个体样本的泛化能力,类似于人类通过类比学习,却未被标准基准揭示。本文提出泛化光谱(Generalization Spectrum)评估框架,以暴露这一隐藏维度。对每个训练样本,构建一组由近及远的测试变体:从精确回忆、跨语言实现,到完全叙事重构、同类别领域内问题,直至无配对基线。通过追踪性能随迁移距离的变化,不仅判断模型是否学会,还揭示其学习可扩展的范围。我们在编程竞赛任务上实例化该框架,使用近期题目生成管道避免数据污染。首先对比三种典型学习范式在匹配记忆条件下的表现:强化学习比SFT系列基线更高效地将记忆转化为近似迁移;上下文学习(ICL)表现出强但依赖对应关系的迁移能力。随后用光谱诊断同类变体差异:结果表明,局部提升未必扩大泛化半径——抽象与提示主要增强局部迁移,参考型强化学习(RFT)比基准SFT保留更强的远距离迁移尾部,而自蒸馏或提示辅助强化学习即使提升局部表现,也可能削弱远距离迁移。
原文摘要 · Abstract (English)
Traditional evaluations measure a learning algorithm's final performance on an i.i.d. test set, reducing learning to a single aggregate score. This approach obscures a fundamental question: to what extent does learning from a specific example generalize to others? Such per-sample generalization, akin to learning by analogy in human cognition, captures how far the knowledge extracted from one example can transfer, yet remains invisible to standard benchmarks. We introduce the Generalization Spectrum, an evaluation framework designed to expose this hidden dimension. For each training example, we construct a controlled suite of test variants arranged by increasing transfer distance, from exact recall to implementation transfer across languages, context transfer under complete narrative re-framing, category-matched in-domain problems, and an unpaired baseline. By tracking performance across these distances, we reveal not just whether an algorithm learns, but how far that learning extends. We instantiate this framework on competitive programming, using a selection-and-synthesis pipeline seeded with recent problems to mitigate contamination. We first compare three canonical learning paradigms under matched memorization. RL converts memorization into near-transfer more efficiently than SFT-family baselines, while ICL exhibits strong but correspondence-dependent transfer. We then use the Spectrum to diagnose within-family variants. The resulting profiles show that local gains need not expand the generalization radius: abstractions and hints mainly lift local transfer, RFT preserves a stronger far-transfer tail than reference SFT, and self-distillation or hint-assisted RL can reduce far transfer even when local transfer or optimization improves.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。