用关键令牌构建数据摘要，大幅降低记忆占用仍保持模型性能。
Core Tokensets for Data-efficient Sequential Training of Transformers
- 在令牌层面挑选信息量最大的数据特征，而非整条样本。
- 仅用1%数据的令牌集，效果相当于传统方法2~10倍大小的采样。
- 适合需要持续学习且内存受限的视觉任务场景。
深度网络常需在新任务上调整，并持续从数据流中学习。此类序列训练要求整合新旧信息，通常通过保留最重要数据点（即核心集）来实现。传统核心集基于完整样本（如图像或句子），但现代Transformer以令牌为单位处理数据，而并非所有令牌都同等重要。为此，我们提出在令牌层级构建更深层的数据摘要——核心令牌集，不仅筛选最具信息量的样本，还利用特征归因仅存储其最相关特征。实验表明，核心令牌集在增量图像分类、开放式视觉问答和持续图像描述生成任务中均能显著保持性能，同时大幅减少内存消耗。实际验证发现，仅1%数据规模的核心令牌集表现相当于传统核心集2至10倍大小的水平。
原文摘要 · Abstract (English)
Deep networks are frequently tuned to novel tasks and continue learning from ongoing data streams. Such sequential training requires consolidation of new and past information, a challenge predominantly addressed by retaining the most important data points - formally known as coresets. Traditionally, these coresets consist of entire samples, such as images or sentences. However, recent transformer architectures operate on tokens, leading to the famous assertion that an image is worth 16x16 words. Intuitively, not all of these tokens are equally informative or memorable. Going beyond coresets, we thus propose to construct a deeper-level data summary on the level of tokens. Our respectively named core tokensets both select the most informative data points and leverage feature attribution to store only their most relevant features. We demonstrate that core tokensets yield significant performance retention in incremental image classification, open-ended visual question answering, and continual image captioning with significantly reduced memory. In fact, we empirically find that a core tokenset of 1\% of the data performs comparably to at least a twice as large and up to 10 times larger coreset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。