提出VISTA框架，让模型训练时持续保留关键知识，避免后期遗忘。
VISTA: Validation-Informed Trajectory Adaptation via Self-Distillation
- 通过验证集识别早期优秀模型状态，作为知识锚点
- 在线融合这些锚点，使训练轨迹更稳定、泛化更强
- 轻量设计节省90%存储，适合实际部署
深度学习模型即使在验证精度很高时，仍可能因优化路径偏离而陷入次优解，这种现象称为轨迹偏差。这是因为训练过程中模型会放弃具有良好泛化性的状态，转而适应特定数据子群体，丢弃之前学到的潜在特征，却不会触发传统过拟合信号。为此，我们提出VISTA——一种基于验证信息的在线自蒸馏框架，通过边际覆盖得分识别早期模型状态中的专家锚点，这些锚点在不同数据区域保持专长能力。将这些锚点以覆盖加权的方式在线整合进训练过程，从而规整损失曲面并保留已掌握的知识。在多个基准测试中，VISTA相较于标准训练和已有自蒸馏方法展现出更强的鲁棒性和泛化能力，且轻量实现使存储开销降低90%而性能无损。
原文摘要 · Abstract (English)
Deep learning models may converge to suboptimal solutions despite strong validation accuracy, masking an optimization failure we term Trajectory Deviation. This is because as training proceeds, models can abandon high generalization states for specific data sub-populations, thus discarding previously learned latent features without triggering classical overfitting signals. To address this problem we introduce VISTA, an online self-distillation framework that enforces consistency along the optimization trajectory. Using a validation-informed Marginal Coverage score, VISTA identifies expert anchors, which are earlier model states that retain specialized competence over distinct data regions. A coverage-weighted ensemble of these anchors is integrated online during training, regularizing the loss landscape and preserving mastered knowledge. When evaluated across multiple benchmarks, VISTA demonstrates improved robustness and generalization over standard training and prior self-distillation methods, while a lightweight implementation reduces storage overhead by 90% without performance loss.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。