用小模型+合成数据，快速打造可控制长度的电话摘要系统
Prompting and Fine-Tuning of Small LLMs for Length-Controllable Telephone Call Summarization
- 用强模型生成合成数据，微调小模型实现可控摘要
- 微调后模型在准确性和简洁性上媲美GPT-4
- 适合需要快速落地、定制化摘要长度的场景
本文探索利用大语言模型（LLMs）快速构建电话通话摘要系统。先通过提示工程测试现有LLM生成摘要效果，再利用前沿模型构建定制化合成训练数据集，重点提升数据多样性和摘要长度可控性。采用两种基于LLM-as-a-judge的评估方法验证摘要质量。结果表明，微调后的Llama-2-7B模型在事实准确性、完整性与简洁性上达到与GPT-4相当水平。研究证明了快速构建高效实用电话摘要系统的可行性。
原文摘要 · Abstract (English)
This paper explores the rapid development of a telephone call summarization system utilizing large language models (LLMs). Our approach involves initial experiments with prompting existing LLMs to generate summaries of telephone conversations, followed by the creation of a tailored synthetic training dataset utilizing stronger frontier models. We place special focus on the diversity of the generated data and on the ability to control the length of the generated summaries to meet various use-case specific requirements. The effectiveness of our method is evaluated using two state-of-the-art LLM-as-a-judge-based evaluation techniques to ensure the quality and relevance of the summaries. Our results show that fine-tuned Llama-2-7B-based summarization model performs on-par with GPT-4 in terms of factual accuracy, completeness and conciseness. Our findings demonstrate the potential for quickly bootstrapping a practical and efficient call summarization system.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。