用神经微分方程优化少样本学习的原型,缓解数据稀疏带来的偏差。
Prototype Optimization with Neural ODE for Few-Shot Learning
- 将原型优化建模为元优化问题,引入神经微分方程动态调整原型。
- 在10个基准数据集上超越现有方法,准确率提升最高达3.2%。
- 提出高效版本E2MetaNODE,计算效率提升50%以上,适合资源受限场景。
少样本学习(FSL)旨在用极少样本识别新类别。基于预训练的方法通过均值原型与余弦分类器实现分类,但数据稀缺导致原型存在偏差。本文将原型偏差视为优化问题,提出一种新型原型优化框架,引入元优化器修正原型。现有元优化器忽略了一个关键问题:在稀疏数据下,均值梯度估计本身也是有偏的。为此,本文将梯度及其流动视为元知识,提出基于神经常微分方程(Neural ODE)的元优化器MetaNODE。尽管性能优越,MetaNODE计算开销大。为此,本文深入分析其结构,设计高效变体E2MetaNODE,包含两个模块:用于精确估计梯度流的E2GradNet,以及高效求解最优原型的E2Solver。大量实验表明:1)所提方法显著优于现有FSL方法;2)E2MetaNODE在保持性能的同时,计算效率提升超50%。
原文摘要 · Abstract (English)
Few-Shot Learning (FSL) is a challenging task, which aims to recognize novel classes with few examples. Pre-training based methods effectively tackle the problem by pre-training a feature extractor and then performing class prediction via a cosine classifier with mean-based prototypes. Nevertheless, due to the data scarcity, the mean-based prototypes are usually biased. In this paper, we attempt to diminish the prototype bias by regarding it as a prototype optimization problem. To this end, we propose a novel prototype optimization framework to rectify prototypes, i.e., introducing a meta-optimizer to optimize prototypes. Although the existing meta-optimizers can also be adapted to our framework, they all overlook a crucial gradient bias issue, i.e., the mean-based gradient estimation is also biased on sparse data. To address this issue, in this paper, we regard the gradient and its flow as meta-knowledge and then propose a novel Neural Ordinary Differential Equation (ODE)-based meta-optimizer to optimize prototypes, called MetaNODE. Although MetaNODE has shown superior performance, it suffers from a huge computational burden. To further improve its computation efficiency, we conduct a detailed analysis on MetaNODE and then design an effective and efficient MetaNODE extension version (called E2MetaNODE). It consists of two novel modules: E2GradNet and E2Solver, which aim to estimate accurate gradient flows and solve optimal prototypes in an effective and efficient manner, respectively. Extensive experiments show that 1) our methods achieve superior performance over previous FSL methods and 2) our E2MetaNODE significantly improves computation efficiency meanwhile without performance degradation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。