GIST通过子空间对齐实现高效指令微调数据筛选,大幅降低资源消耗。
GIST: Targeted Data Selection for Instruction Tuning via Coupled Optimization Geometry
- 用SVD从验证梯度中提取任务相关子空间,实现参数耦合下的精准评分
- 在仅0.29%存储和25%计算时间下达到或超越当前最佳效果
- 特别适合低资源场景下的参数高效微调(如LoRA)
目标数据选择已成为高效指令微调的关键范式,旨在为特定任务挑选少量但具有影响力的训练样本。现有方法通常通过示例对参数更新的影响来衡量其重要性,并利用优化器状态(如Adam)作为更新几何的轴对齐近似,隐含假设参数间相互独立。然而,在参数高效微调(PEFT)方法(如LoRA)中,这种假设失效:优化几何呈现强参数耦合与非平凡的非对角交互,且任务相关更新方向被限制在低维子空间。为此,我们提出GIST(Gradient Isometric Subspace Transformation),一种简洁而原理清晰的替代方案,将轴对齐缩放替换为鲁棒的子空间对齐。GIST通过奇异值分解(SVD)从验证梯度中恢复任务特定子空间,将训练梯度投影至该耦合子空间,并根据其与目标方向的对齐度评分。大量实验表明,GIST在相同选择预算下,仅需0.29%的存储和25%的计算时间,即可匹配或超越当前最优基线。
原文摘要 · Abstract (English)
Targeted data selection has emerged as a crucial paradigm for efficient instruction tuning, aiming to identify a small yet influential subset of training examples for a specific target task. In practice, influence is often measured through the effect of an example on parameter updates. To make selection scalable, many approaches leverage optimizer statistics (e.g., Adam states) as an axis-aligned surrogate for update geometry (i.e., diagonal precondition), implicitly treating parameters as coordinate-wise independent. We show that this assumption breaks down in parameter-efficient fine-tuning (PEFT) methods such as LoRA. In this setting, the induced optimization geometry exhibits strong cross-parameter coupling with non-trivial off-diagonal interactions, while the task-relevant update directions are confined to a low-dimensional subspace. Motivated by this mismatch, we propose GIST (Gradient Isometric Subspace Transformation), a simple yet principled alternative that replaces axis-aligned scaling with robust subspace alignment. GIST recovers a task-specific subspace from validation gradients via singular value decomposition (SVD), projects training gradients into this coupled subspace, and scores examples by their alignment with target directions. Extensive experiments have demonstrated that GIST matches or outperforms the state-of-the-art baseline with only 0.29% of the storage and 25% of the computational time under the same selection budget.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。