用大模型分析用户报修文本，自动定位硬件故障部件。
Evaluating LLMs and Prompting Strategies for Automated Hardware Diagnosis from Textual User-Reports
- 测试27个开源及2个闭源大模型，对比四种提示策略效果。
- 最高F1达0.76，三款小模型在性能与显存间平衡优异。
- 适合希望在手机、笔记本上部署的终端设备开发者使用。
计算机制造商提供用户通过文本描述设备故障的平台，例如“我的屏幕在闪烁”。从报告中识别出故障组件对自动化测试和提升用户体验至关重要。然而，这些报告通常模糊且缺乏细节，使任务极具挑战性。大型语言模型（LLMs）在此方面展现出潜力。本研究评估了27个开源模型（参数量1B-72B）及2个闭源模型，采用四种提示策略：零样本（Zero-Shot）、少样本（Few-Shot）、思维链（Chain-of-Thought, CoT）以及CoT+少样本（CoT+FS）。共执行98,948次推理，处理超过5100万输入token，生成1300万输出token。实验结果表明，最高F1-score达到0.76。结果显示，三个模型在模型规模与性能之间取得最佳平衡：mistral-small-24b-instruct以及两个更小的模型llama-3.2-1b-instruct和gemma-2-2b-it，其性能与低显存占用兼具，可在现代带NPU的笔记本或智能手机上实现高效推理。
原文摘要 · Abstract (English)
Computer manufacturers offer platforms for users to describe device faults using textual reports such as "My screen is flickering". Identifying the faulty component from the report is essential for automating tests and improving user experience. However, such reports are often ambiguous and lack detail, making this task challenging. Large Language Models (LLMs) have shown promise in addressing such issues. This study evaluates 27 open-source models (1B-72B parameters) and 2 proprietary LLMs using four prompting strategies: Zero-Shot, Few-Shot, Chain-of-Thought (CoT), and CoT+Few-Shot (CoT+FS). We conducted 98,948 inferences, processing over 51 million input tokens and generating 13 million output tokens. We achieve f1-score up to 0.76. Results show that three models offer the best balance between size and performance: mistral-small-24b-instruct and two smaller models, llama-3.2-1b-instruct and gemma-2-2b-it, that offer competitive performance with lower VRAM usage, enabling efficient inference on end-user devices as modern laptops or smartphones with NPUs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。