用语义空间构建优化算法轨迹图,跨模型比较学习动态
Semantic Space Search Trajectory Networks

- 将模型预测向量作为语义空间,聚类生成网络节点
- 真实标签训练的图更密集高效,结构更集中
- 适合研究模型泛化与不同训练策略的差异
搜索轨迹网络(STNs)是一种基于图的工具,用于可视化和描述优化算法的行为。传统STNs依赖于搜索空间的离散化,因此主要适用于低维或组合场景。本文提出在语义空间中构建STNs的方法,语义空间定义为模型对固定样本集预测结果的空间。通过归一化汉明距离下的完全链接凝聚聚类,对语义向量进行离散化并聚合为网络节点。由于任何预测器均可由其语义向量表示,该方法实现了对不同算法族之间学习动态的可比性分析。我们在分类和回归任务中应用语义空间STNs,成功复现了不同算法间的已知定性差异。此外,我们通过对比标准训练与Zhang等人(2017)提出的标签随机化训练,研究神经网络泛化能力。结果显示,真实标签训练生成的图更密集、更高效、更中心化。这些结果表明,语义空间STNs能捕捉学习算法与数据交互产生的功能性训练动态,为跨模型与训练策略的学习动态分析提供有效工具。
原文摘要 · Abstract (English)
Search Trajectory Networks (STNs) are a graph-based tool for visualizing and characterizing the behavior of optimization algorithms. STNs' reliance on discretization of the search space has largely confined them to low-dimensional or combinatorial settings. We introduce a methodology for constructing STNs in semantic spaces, defined as the space of a model's predictions on a fixed sample set. Our approach discretizes semantic vectors and aggregates them into network nodes via agglomerative clustering with complete linkage under a normalized Hamming distance. Since any predictor can be summarized by its semantic vector, this method enables comparison of learning dynamics across otherwise incomparable algorithm families. We apply semantic space STNs to classification and regression tasks solved using different machine learning algorithms, recovering known qualitative differences between them. Additionally, we use semantic space STNs to study neural network generalization by contrasting standard training with the label randomization regime of Zhang et al. (2017). The resulting STNs exhibit consistent structural differences, training on real labels produces denser, more efficient and more centralized graphs than training on shuffled labels. Together, our results show that semantic space STNs capture functional training dynamics arising from the interaction between learning algorithms and data, providing a tool for analyzing and comparing learning dynamics across machine learning models and training regimes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。