提出一种以软标签压缩为核心的图像数据浓缩方法,显著降低存储开销。
SCORE: Soft Label Compression-Centric Dataset Condensation via Coding Rate Optimization
- 从信息论出发,优化编码率平衡数据的语义信息、区分度和可压缩性。
- 在ImageNet-1K上,软标签压缩30倍后性能仅下降5.5%(IPC=10)。
- 适合关注大规模数据浓缩且对存储敏感的研究与应用。
数据浓缩(DC)旨在生成一个紧凑的数据集,使在该数据集上训练的模型性能可媲美全量数据训练的结果。近期方法通过将知识编码到带软标签的真实图像中,实现对ImageNet规模数据集的可扩展性和强跨域泛化能力,但带来了显著的存储成本,可能远超原始数据。本文认为缓解性能-存储矛盾的关键是数据的语义信息量、区分度和可压缩性。为此,提出基于编码率优化的软标签压缩中心框架SCORE。SCORE将数据浓缩建模为一个极小极大优化问题,从信息论角度平衡上述三要素。理论证明其编码率启发的目标函数具有子模性,优化过程自然促使每个浓缩数据对应的软标签集呈现低秩结构。在ImageNet-1K和Tiny-ImageNet等大规模数据集上的实验表明,SCORE在多数情况下优于现有方法。即使软标签压缩30倍,在IPC=10和50时,ImageNet-1K上性能仅下降5.5%和2.7%。代码将在论文接受后发布。
原文摘要 · Abstract (English)
Dataset Condensation (DC) aims to obtain a condensed dataset that allows models trained on the condensed dataset to achieve performance comparable to those trained on the full dataset. Recent DC approaches increasingly focus on encoding knowledge into realistic images with soft labeling, for their scalability to ImageNet-scale datasets and strong capability of cross-domain generalization. However, this strong performance comes at a substantial storage cost which could significantly exceed the storage cost of the original dataset. We argue that the three key properties to alleviate this performance-storage dilemma are informativeness, discriminativeness, and compressibility of the condensed data. Towards this end, this paper proposes a \textbf{S}oft label compression-centric dataset condensation framework using \textbf{CO}ding \textbf{R}at\textbf{E} (SCORE). SCORE formulates dataset condensation as a min-max optimization problem, which aims to balance the three key properties from an information-theoretic perspective. In particular, we theoretically demonstrate that our coding rate-inspired objective function is submodular, and its optimization naturally enforces low-rank structure in the soft label set corresponding to each condensed data. Extensive experiments on large-scale datasets, including ImageNet-1K and Tiny-ImageNet, demonstrate that SCORE outperforms existing methods in most cases. Even with 30$\times$ compression of soft labels, performance decreases by only 5.5\% and 2.7\% for ImageNet-1K with IPC 10 and 50, respectively. Code will be released upon paper acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。