通过感知数据质量的分词技术，让嘈杂真实数据更好用于大模型预训练。
Unlocking Noisy Real-World Corpora for Foundation Model Pre-Training via Quality-Aware Tokenization
- 将数据可靠性融入词表构建，用双层优化联合优化词表与下游性能。
- 在基因组和金融数据上分别提升6.7% F1值和30%夏普比率，路径检测94.53 MCC。
- 适合处理噪声多的真实世界数据，如基因组和金融时序，无推理开销。
现有分词方法未考虑信号质量，难以有效处理嘈杂的真实世界语料。本文提出QA-Token（质量感知分词），将数据可靠性直接纳入词表构建。主要贡献包括：(i) 双层优化框架，联合优化词表构造与下游性能；(ii) 基于强化学习的合并策略学习方法，使用质量感知奖励并具备收敛性保证；(iii) 通过Gumbel-Softmax松弛实现端到端参数自适应学习。实验表明，该方法在基因组数据上使变异检测的F1值提升6.7个百分点（超过BPE），在金融领域实现30%的夏普比率提升。在基础模型规模下，对包含1.7万亿碱基对的预训练语料进行分词，实现了94.53 MCC的最优病原体检测效果，同时减少15%的分词数量。该方法成功解锁了涵盖拍基组基因序列和千兆字节金融时间序列的噪声真实语料，可用于大模型训练且不增加推理开销。
原文摘要 · Abstract (English)
Current tokenization methods process sequential data without accounting for signal quality, limiting their effectiveness on noisy real-world corpora. We present QA-Token (Quality-Aware Tokenization), which incorporates data reliability directly into vocabulary construction. We make three key contributions: (i) a bilevel optimization formulation that jointly optimizes vocabulary construction and downstream performance, (ii) a reinforcement learning approach that learns merge policies through quality-aware rewards with convergence guarantees, and (iii) an adaptive parameter learning mechanism via Gumbel-Softmax relaxation for end-to-end optimization. Our experimental evaluation demonstrates consistent improvements: genomics (6.7 percentage point F1 gain in variant calling over BPE), finance (30% Sharpe ratio improvement). At foundation scale, we tokenize a pretraining corpus comprising 1.7 trillion base-pairs and achieve state-of-the-art pathogen detection (94.53 MCC) while reducing token count by 15%. We unlock noisy real-world corpora, spanning petabases of genomic sequences and terabytes of financial time series, for foundation model training with zero inference overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。