轻量级n-gram模型实验平台，低资源下高效运行
HITgram: A Platform for Experimenting with n-gram Language Models
- 支持1-4阶n-gram，集成上下文加权与动态语料管理
- 每秒处理5万词元，320MB语料生成2-gram仅需62秒
- 适合教学、原型验证或低配设备上的语言模型研究
大型语言模型虽强大但资源消耗高，限制了普及。HITgram提供一个轻量级平台，用于n-gram模型的实验，特别适用于资源受限环境。它支持从一元到四元模型，并引入上下文敏感加权、拉普拉斯平滑和动态语料管理等技术，提升预测准确率，即使面对未见词序列也能有效应对。实验表明，该平台具备高效率：在单台8GB内存机器上，处理320MB语料生成2-gram仅需62秒，吞吐量达50,000词元/秒；构建1GB语料的4-gram模型耗时不足298秒。未来计划包括多语言支持、高级平滑方法、并行处理及模型保存功能，进一步拓展应用范围。
原文摘要 · Abstract (English)
Large language models (LLMs) are powerful but resource intensive, limiting accessibility. HITgram addresses this gap by offering a lightweight platform for n-gram model experimentation, ideal for resource-constrained environments. It supports unigrams to 4-grams and incorporates features like context sensitive weighting, Laplace smoothing, and dynamic corpus management to e-hance prediction accuracy, even for unseen word sequences. Experiments demonstrate HITgram's efficiency, achieving 50,000 tokens/second and generating 2-grams from a 320MB corpus in 62 seconds. HITgram scales efficiently, constructing 4-grams from a 1GB file in under 298 seconds on an 8 GB RAM system. Planned enhancements include multilingual support, advanced smoothing, parallel processing, and model saving, further broadening its utility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。