在有限算力下验证大模型微调对报告摘要的有效性
Fine-Tuning LLMs for Report Summarization: Analysis on Supervised and Unsupervised Data
- 用监督与无监督数据并行微调大模型，适配本地部署环境
- 微调后摘要质量提升，无效内容显著减少
- 适合需本地处理敏感报告的机构使用
我们研究了大语言模型（LLMs）在报告摘要任务（政府档案、新闻、情报报告）上的微调效果。该应用面临两大挑战：一是真实摘要可能不可得（如政府档案），二是计算需在本地进行，多数实验仅使用一到两个A100 GPU卡。在此条件下，我们探讨两个问题：微调是否可在本地资源受限环境下实现；如何评估摘要质量。通过对比两种微调方法，发现微调在多数情况下可提升摘要质量，并有效减少无效或垃圾摘要数量。
原文摘要 · Abstract (English)
We study the efficacy of fine-tuning Large Language Models (LLMs) for the specific task of report (government archives, news, intelligence reports) summarization. While this topic is being very actively researched - our specific application set-up faces two challenges: (i) ground-truth summaries maybe unavailable (e.g., for government archives), and (ii) availability of limited compute power - the sensitive nature of the application requires that computation is performed on-premise and for most of our experiments we use one or two A100 GPU cards. Under this set-up we conduct experiments to answer the following questions. First, given that fine-tuning the LLMs can be resource intensive, is it feasible to fine-tune them for improved report summarization capabilities on-premise? Second, what are the metrics we could leverage to assess the quality of these summaries? We conduct experiments on two different fine-tuning approaches in parallel and our findings reveal interesting trends regarding the utility of fine-tuning LLMs. Specifically, we find that in many cases, fine-tuning helps improve summary quality and in other cases it helps by reducing the number of invalid or garbage summaries.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。