用BERTopic分析大模型对话，发现不同话题下用户偏好有规律。
Investigating Thematic Patterns and User Preferences in LLM Interactions using BERTopic
- 用BERTopic从多语言对话数据中提取29个主题
- 发现特定话题中某些大模型更受用户青睐
- 适合关注大模型优化与用户满意度的研究者
本研究采用基于Transformer的BERTopic主题建模方法，对lmsys-chat-1m数据集进行分析。该数据集为来自大型语言模型（LLMs）对抗评测的多语言对话语料，每条用户提问配有两个匿名模型回复及人类偏好标签，用于评估模型输出优劣。研究旨在揭示对话中的主题模式及其与用户偏好的关联，特别是特定主题下是否存在模型偏好一致性。研究设计了针对多语言差异的预处理流程，平衡对话轮次，并清理噪声或遮蔽数据。BERTopic成功提取出超过29个语义连贯的主题，涵盖人工智能、编程、伦理、云基础设施等。通过分析主题与模型偏好之间的关系，识别出模型-主题匹配趋势。可视化手段包括主题间距离图、主题概率分布和模型-主题矩阵。研究结果为提升真实场景中大模型性能与用户满意度提供了领域特定微调与优化策略。
原文摘要 · Abstract (English)
This study applies BERTopic, a transformer-based topic modeling technique, to the lmsys-chat-1m dataset, a multilingual conversational corpus built from head-to-head evaluations of large language models (LLMs). Each user prompt is paired with two anonymized LLM responses and a human preference label, used to assess user evaluation of competing model outputs. The main objective is uncovering thematic patterns in these conversations and examining their relation to user preferences, particularly if certain LLMs are consistently preferred within specific topics. A robust preprocessing pipeline was designed for multilingual variation, balancing dialogue turns, and cleaning noisy or redacted data. BERTopic extracted over 29 coherent topics including artificial intelligence, programming, ethics, and cloud infrastructure. We analysed relationships between topics and model preferences to identify trends in model-topic alignment. Visualization techniques included inter-topic distance maps, topic probability distributions, and model-versus-topic matrices. Our findings inform domain-specific fine-tuning and optimization strategies for improving real-world LLM performance and user satisfaction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。