让大模型动态生成词汇，突破固定词表限制。
DVAGen: Dynamic Vocabulary Augmented Generation
- 设计模块化框架，支持动态词表训练与评估
- 实现批量推理，推理吞吐量显著提升
- 开源提供命令行和网页界面，便于实时查看结果
使用固定词表的语言模型在处理新词或未登录词时泛化能力受限，影响其对多样化词组的适应性。现有动态词表方法存在代码库分散、不支持现代大模型、推理扩展性差等问题。为此，我们提出DVAGen，一个完全开源的统一框架，用于训练、评估和可视化动态词表增强的语言模型。该框架模块化设计，便于定制，可无缝集成开源大模型，并首次提供命令行和网页界面工具，支持实时结果观察。我们在现代大模型上验证了动态词表方法的有效性，展示了批量推理支持，显著提升了推理吞吐量。
原文摘要 · Abstract (English)
Language models trained with a fixed vocabulary struggle to generalize to novel or out-of-vocabulary words, limiting their flexibility in handling diverse token combinations. Existing dynamic vocabulary approaches attempt to address this limitation but face challenges such as fragmented codebases, lack of support for modern LLMs, and limited inference scalability. To overcome these issues, we introduce DVAGen, a fully open-source, unified framework designed for training, evaluation, and visualization of dynamic vocabulary-augmented language models. Our framework modularizes the pipeline for ease of customization, integrates seamlessly with open-source LLMs, and is the first to provide both CLI and WebUI tools for real-time result inspection. We validate the effectiveness of dynamic vocabulary methods on modern LLMs and demonstrate support for batch inference, significantly improving inference throughput.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。