平衡数据域的词元分布能提升小样本领域表现，优于单纯增加数据量。
Token Distribution versus Data Volume: Domain Balancing in Multi-Domain Meeting Summarisation
- 通过控制词元总量，对比均衡与自然比例混合，分离数据量与分布的影响。
- 在2-32M词元预算下，均衡分配使数据少的领域质量显著提升，代价极小。
- 按词元平衡比按样本平衡更有效，且可删减15%低价值对话内容而不降质。
在规模差异巨大的多领域会议摘要语料上联合微调大语言模型时，现有研究未厘清：领域均衡训练混合带来的增益是源于词元在各领域的分布均衡，还是仅仅因为看到的总数据量更大？我们通过在五个英文会议语料上构建相同词元预算（2-3200万）下的均衡与自然比例（原生比例）词元混合，使用Mistral-7B与QLoRA进行微调，并按领域评估性能。结果表明，均衡分配能有效重分配性能，显著提升数据稀疏的少数领域表现，对数据丰富的多数领域影响极小。当少数领域重要时，均衡策略更优：其占比在自然比例下始终固定在1-2%，要达到均衡质量需远超总数据量。此外发现，剔除低价值对话行可移除约15%的词元而无明显性能损失，且按词元平衡不等同于按样本平衡。基于741条标注事实的双标注员研究验证了我们的事实级评估方法。这些结果为从业者提供了决定是否及如何平衡非均衡多领域数据集的依据。
原文摘要 · Abstract (English)
Jointly fine-tuning an LLM on meeting-summarisation corpora of widely varying size raises a question that prior work leaves confounded: when a domain-balanced training mixture helps, is the gain due to the distribution of tokens across domains, or merely to the volume of data seen? We disentangle these factors by constructing balanced and natural (native-proportional) token mixtures at matched token budgets (2-32M) over five English meeting corpora, fine-tuning Mistral-7B with QLoRA, and evaluating per domain. Balancing redistributes quality, improving the data-scarce minority domains at a low cost to the data-rich ones. The trade favours balancing whenever the minority domains matter: their share under proportional allocation is fixed at 1-2% regardless of budget, so matching balanced quality on those domains requires far more total data. We further find that pruning low-value transcript lines removes ~15% of tokens from the conversational corpora at no measurable cost, and that balancing by tokens is not the same as balancing by examples. A two-annotator study of 741 judge-labelled facts validates our fact-level evaluation. Together these results give practitioners a basis for deciding when to balance an imbalanced multi-domain mixture, and on what unit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。