SAGE用紧凑梯度快照筛选代表性数据，省时省内存。
SAGE: Streaming Agreement-Driven Gradient Sketches for Representative Subset Selection
- 用频次方向法压缩梯度信息，仅需O(ℓD)内存
- 保留少量样本训练，精度接近全量数据
- 适合资源受限场景的高效训练
在大规模数据上训练现代神经网络计算和能耗高昂。我们提出SAGE，一种流式数据子集选择方法，以O(ℓD)内存维护梯度几何的频繁方向（FD）快照，优先选择其梯度与共识方向对齐的样本。该方法避免了N×N的成对相似性计算和显式的N×ℓ梯度存储，实现简单、两遍、适合GPU的流水线。利用FD的确定性近似保证，我们分析了对齐评分如何在主快照子空间中保持梯度能量。在多个基准测试中，SAGE在小保留率预算下训练，精度仍可媲美全数据训练及近期子集选择基线，且显著降低端到端计算量与峰值内存。总体而言，SAGE提供了一种实用的恒定内存替代方案，可与剪枝和模型压缩协同用于高效训练。
原文摘要 · Abstract (English)
Training modern neural networks on large datasets is computationally and energy intensive. We present SAGE, a streaming data-subset selection method that maintains a compact Frequent Directions (FD) sketch of gradient geometry in $O(\ell D)$ memory and prioritizes examples whose sketched gradients align with a consensus direction. The approach eliminates $N \times N$ pairwise similarities and explicit $N \times \ell$ gradient stores, yielding a simple two-pass, GPU-friendly pipeline. Leveraging FD's deterministic approximation guarantees, we analyze how agreement scoring preserves gradient energy within the principal sketched subspace. Across multiple benchmarks, SAGE trains with small kept-rate budgets while retaining competitive accuracy relative to full-data training and recent subset-selection baselines, and reduces end-to-end compute and peak memory. Overall, SAGE offers a practical, constant-memory alternative that complements pruning and model compression for efficient training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。