arXiv:2501.19317cs.CL2025-01被引 3

量化降低模型精度可节省76%内存,但影响情感文本生成质量。

LLM-based Affective Text Generation Quality Based on Different Quantization Values

  • 测试8/16/32位精度在五款开源大模型上的表现
  • 8位量化使内存减少76%,大模型F1下降10个百分点
  • 低精度大模型生成效果优于高精度小模型

大型语言模型在语言生成与理解方面表现出色,能生成更具情感共鸣的文本。然而其庞大的参数量导致训练与推理需大量计算资源,在资源受限场景下难以部署。通过降低精度位数可提升内存效率,但会牺牲准确率。本文研究不同量化值(8、16、32位)在情感文本生成中的权衡关系,使用情绪分类器和十组种子提示,在两个家族共五款开源模型上进行测试。结果表明,量化可实现76%的内存节省,但大模型在8位时F1分数最多下降10个百分点,小模型则上升10个百分点,且推理时间约翻倍。值得注意的是,低精度大模型在文本质量上普遍优于高精度小模型,且所需内存相近。

原文摘要 · Abstract (English)

Large language models exhibit a remarkable capacity in language generation and comprehension. These advances enable AI systems to produce more human-like and emotionally engaging text. However, these models rely on a large number of parameters, requiring significant computational resources for training and inference. In some scenarios, accessing these resources can be challenging (e.g., budget or hardware limitations). Techniques like reducing precision bits can make models more memory-efficient, reducing the computational resources needed, at the cost of reduced accuracy. This paper addresses the trade-off between different quantization values, GPU RAM utilization, and text quality in affective text generation (e.g., "I really enjoy running in the snow-covered forest"). To evaluate, we use an emotion classifier and ten seed prompts to generate affective text. We test three setups of precision bits (8, 16, and 32) across five open-weight language models from two different families. Our findings demonstrate that bit reductions lead to memory savings, achieving a reduction of 76%. However, this optimization comes with a trade-off, leading to a decrease of up to 10 pp in F1 score for larger models and an increase of 10 pp for smaller models, along with roughly double the inference time. In terms of text quality, larger models at lower quantization levels generally outperform smaller, higher-precision models -- while requiring similar memory.

大模型量化情感生成内存优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。