细粒度训练不一定更好,关键看数据结构与标签层级的匹配度。
The Advantage of Fine-Grained Training
- 通过分析数据几何结构与标签层级关系,揭示细粒度训练的有效性条件。
- 发现决策边界对齐程度(边界冗余)决定训练效果优劣。
- 模型容量和数据量影响策略选择,存在性能转折点。
在分类问题中,模型基于输入特征预测类别标签。然而,许多数据集的标签具有层次结构。尽管分类任务通常定义在某一特定层级,但训练可利用更细粒度的标签。实证表明,细粒度训练能提升性能。本文通过真实与合成数据集研究该现象的普遍性及其成因。结果表明,细粒度训练并非始终有效,其优势取决于数据的几何结构与标签层级的关系。具体而言,细粒度与粗粒度任务所需决策边界之间的对齐程度——即边界冗余——是关键因素。此外,数据集规模和模型容量显著影响细粒度训练是否带来收益。我们识别出一个转变点,其位置主要由模型过参数化程度决定,该点将细粒度训练有益与直接粗粒度训练更优的区域分隔开。
原文摘要 · Abstract (English)
In classification problems, models are trained to predict a class label based on the input data features. However, class labels are organized hierarchically in many datasets. While a classification task is often defined at a specific level of this hierarchy, training can utilize a finer granularity of labels. Empirical evidence suggests that such fine-grained training can enhance performance. In this work, we investigate the generality of this observation and explore its underlying causes using both real and synthetic datasets. We show that training on fine-grained labels does not universally improve classification accuracy. Instead, the effectiveness of this strategy depends on the geometric structure of the data and its relations with the label hierarchy. Specifically, we show that the advantage of fine-grained training crucially depends on the degree of alignment between the decision boundaries required for the fine- and coarse-grained tasks, a property that we term boundary redundancy. Additionally, factors such as dataset size and model capacity significantly influence whether fine-grained labels provide a performance benefit. Indeed, we identify a transition, whose location is largely controlled by the degree of overparameterization, separating regimes where fine-grained training improves performance from those where direct coarse-grained training is preferable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。