提出快速元素选择算法,无需乘法即可降维。
Fast Swap-Based Element Selection for Multiplication-Free Dimension Reduction

- 基于矩阵求逆引理,通过交换元素优化选择方案。
- 在MNIST上实现低误差降维,速度远超传统方法。
- 适合资源受限设备的高效降维,尤其适合嵌入式场景。
本文提出一种快速元素选择算法,这是一种无需乘法的降维方法,通过从输入中直接选取部分元素生成降维向量。降维是减少模型参数、缓解过拟合和加速训练推理的基础技术。标准方法如主成分分析(PCA)依赖矩阵乘法,在资源受限系统中乘法次数可能成为瓶颈。元素选择消除该开销,其关键挑战在于确定保留哪些元素。我们通过最小均方误差的线性回归来评估候选子集的性能,目标可为分类任务中的一热标签向量;当无显式目标时,可用输入自身作为目标,得到基于重建的准则。该优化问题为组合优化,穷举搜索不可行。为此,我们利用矩阵求逆引理推导出交换一个已选与未选元素所导致的目标函数变化公式,并采用基于交换的局部搜索,反复执行使目标下降的交换,直至无法改进。在手写数字数据集MNIST上的实验验证了该方法的有效性。
原文摘要 · Abstract (English)
In this paper, we propose a fast algorithm for element selection, a multiplication-free form of dimension reduction that produces a dimension-reduced vector by simply selecting a subset of elements from the input. Dimension reduction is a fundamental technique for reducing unnecessary model parameters, mitigating overfitting, and accelerating training and inference. A standard approach is principal component analysis (PCA), but PCA relies on matrix multiplications; on resource-constrained systems, the multiplication count itself can become a bottleneck. Element selection eliminates this cost because the reduction consists only of selecting elements, and thus the key challenge is to determine which elements should be retained. We evaluate a candidate subset through the minimum mean-squared error of linear regression that predicts a target vector from the selected elements, where the target may be, for example, a one-hot label vector in classification. When an explicit target is unavailable, the input itself can be used as the target, yielding a reconstruction-based criterion. The resulting optimization is combinatorial, and exhaustive search is impractical. To address this, we derive an efficient formula for the objective change caused by swapping a selected and an unselected element, using the matrix inversion lemma, and we perform a swap-based local search that repeatedly applies objective-decreasing swaps until no further improvement is possible. Experiments on MNIST handwritten-digit images demonstrate the effectiveness of the proposed method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。