arXiv:2607.02266cs.LGcs.AI2026-07

提出可动态调节粒度的标签体系,让预训练数据混合更灵活高效。

HERMES: A Multi-Granularity Labeling Substrate for Pre-training Data Mixtures

论文配图:HERMES: A Multi-Granularity Labeling Substrate for Pre-training Data Mixtures
图 1 · 摘自论文原文
  • 基于层次化向量量化构建可变粒度的文档标签体系
  • 在10亿参数、250亿词元训练中提升多任务性能0.0253
  • 适合需要精细控制数据混合策略的研究者使用

现有数据混合方法依赖预先划分好的分组,而标签体系决定了混合能力的边界。当前标签如来源、主题或格式分类,以及平坦的嵌入聚类,仅支持单一语义轴和固定粒度,调整分辨率需重新标注。本文认为瓶颈在于标签系统而非混合器,提出一种数据驱动的层次化标签体系——HERMES:通过学习语义变换与三阶段残差向量量化,一次性将每个文档编码为粗到细的代码,其前缀长度可控制粒度,最多达约13万单元。在粗粒度下,HERMES在标准聚类指标上与KMeans方法持平,核心贡献在于标签底座。在10亿参数、250亿词元的预训练中,该层级揭示了固定粒度无法测试的交互效应:在某一前缀长度下,结合阶段二规则的对比策略(等量子桶覆盖优于按大小比例分配),使16项任务宏平均提升0.0253;而在更细粒度下,候选池缩小约5倍时,该优势消失。HERMES将数据混合设计从选择固定标签集转变为导航可复用的数据驱动粒度层级。

原文摘要 · Abstract (English)

Most data-mixing methods assume the corpus has already been partitioned into groups, and the choice of those groups determines what a mixer can express. Existing labels, including provenance, topic or format taxonomies, and flat embedding clusters, commit to one semantic axis at one granularity; changing the resolution rebuilds the labels. We argue the bottleneck is the label system, not the mixer, and provide a hierarchical one. HERMES is a data-derived labeling substrate: a Learned Semantic Transform followed by 3-stage residual vector quantization annotates each document once into a coarse-to-fine code whose prefix length controls granularity up to approximately 130k cells. At coarse granularity HERMES sits at a plateau with KMeans-family methods on standard clustering metrics, so the contribution is the substrate, not the clusterer. On 1B-parameter, 25B-token pre-training, the hierarchy exposes an interaction fixed-granularity pipelines cannot test: at one prefix length, a combined Stage-2 rule contrast, equal-subbucket coverage versus size-proportional within-bucket quality top-30%, lifts a 16-task capability macro-average by +0.0253; at the next finer level, the same rule loses its measurable edge as candidate pools contract approximately 5x. HERMES reframes data mixture design from choosing among fixed label sets to navigating a reusable, data-derived granularity hierarchy.

数据混合标签体系层次化预训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。