评测大模型对福祉概念的解释能力，发现微调后表现更优。
Are Today's LLMs Ready to Explain Well-Being Concepts?
- 构建4.3万条解释数据集，用双模型评估解释质量。
- 微调模型在不同人群和主题上表现显著提升。
- 偏好学习方法让小模型超越大模型，适合专业解释任务。
福祉涵盖心理、生理和社交维度，关乎个人成长与决策。随着人们越来越多地依赖大语言模型（LLMs）理解福祉概念，一个关键问题浮现：这些模型能否生成既准确又适配不同受众的解释？高质量解释需兼具事实正确性与用户期望契合度。本文构建了一个大规模数据集，包含由10种不同LLMs生成的2,194个福祉概念的43,880条解释。提出一种基于原则的LLM-as-a-judge评估框架，采用双评委机制评估解释质量。进一步表明，通过监督微调（SFT）和直接偏好优化（DPO）对开源模型进行微调，能显著提升解释质量。结果表明：(1) 所提出的LLM评委与人类评估高度一致；(2) 解释质量在模型、受众和类别间存在显著差异；(3) 经过DPO和SFT微调的模型优于其更大版本，证明了基于偏好的学习在专业化解释任务中的有效性。
原文摘要 · Abstract (English)
Well-being encompasses mental, physical, and social dimensions essential to personal growth and informed life decisions. As individuals increasingly consult Large Language Models (LLMs) to understand well-being, a key challenge emerges: Can LLMs generate explanations that are not only accurate but also tailored to diverse audiences? High-quality explanations require both factual correctness and the ability to meet the expectations of users with varying expertise. In this work, we construct a large-scale dataset comprising 43,880 explanations of 2,194 well-being concepts, generated by ten diverse LLMs. We introduce a principle-guided LLM-as-a-judge evaluation framework, employing dual judges to assess explanation quality. Furthermore, we show that fine-tuning an open-source LLM using Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) can significantly enhance the quality of generated explanations. Our results reveal: (1) The proposed LLM judges align well with human evaluations; (2) explanation quality varies significantly across models, audiences, and categories; and (3) DPO- and SFT-finetuned models outperform their larger counterparts, demonstrating the effectiveness of preference-based learning for specialized explanation tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。