用训练轨迹信息选关键文本，压缩数据集至原大小0.1%仍保持效果
TAKE: Trajectory-Aware Knowledge Estimation for Text Dataset Distillation
- 基于影响函数与训练轨迹，计算每条数据的知识贡献得分
- 在0.1%压缩比下，分类和推理任务准确率接近全量数据
- 适合追求数据高效、模型轻量化的研究者和工程师
大规模文本语料已成为现代NLP的隐性瓶颈，不仅占用存储空间，还带来训练、微调和持续学习的累积成本。本文提出一种文本数据集蒸馏框架，可将语料压缩至原始大小的0.1%，同时保持下游任务性能。方法基于影响函数，量化每个样本对下游目标的贡献，提供合理的筛选依据。提出轨迹感知知识估计（TAKE），通过卷积训练轨迹上的知识影响，生成单样本知识得分，捕捉有信息量的样本。该得分作为离散最优传输问题中的样本权重，指导从合成候选池中选取原型数据。在文本分类和自然语言推理任务上，极端压缩比（0.1%或20样本/类）下仍保持高下游准确率，证明了数据效率的可行性。方法理论基础扎实，对核心集构建和以数据为中心的AI具有广泛意义。代码已开源：https://github.com/votrinhan88/take。
原文摘要 · Abstract (English)
Large-scale text corpora have become a quiet bottleneck in modern NLP, not just in storage, but in the accumulated cost of training, fine-tuning, and continual learning. We propose a text dataset distillation framework that reduces corpora to as little as 0.1% of their original size while preserving downstream task fidelity. We approach distillation through the lens of influence functions, which quantify each sample's contribution to the downstream objective, a natural and principled basis for selection. We introduce Trajectory-Aware Knowledge Estimation (TAKE), which convolves the knowledge-based influence along the training trajectory into a single per-sample knowledge score, capturing informative samples. These scores serve as sample weights within a discrete Optimal Transport objective, guiding prototype selection from a synthetically generated candidate pool. We evaluate TAKE on downstream accuracy across text classification and natural language inference tasks at extreme compression (0.1% or 20 samples/class), showing that data efficiency is achievable without sacrificing task fidelity. The approach is theoretically grounded, with broader implications for coreset construction and data-centric AI. We release our source code at https://github.com/votrinhan88/take.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。