用对话动态推断用户技术水平，让聊天机器人更懂你。
ProfiLLM: An LLM-Based Framework for Implicit Profiling of Chatbot Users
- 通过对话内容自动推断用户技术水平，无需主动填写信息。
- 在一次对话后，预测准确率提升55%至65%。
- 适合需要个性化交互的高专业度领域，如网络安全。
尽管对话式AI取得显著进展，大型语言模型（LLM）驱动的聊天机器人在根据用户的技术能力、学习风格和沟通偏好进行个性化回应方面仍存在困难，尤其在技术密集型领域如信息技术/网络安全（ITSec）中更为突出。现有个性化方法多依赖静态用户分类或显式自我报告，难以适应交互过程中用户认知的动态变化。本文提出ProfiLLM，一种基于LLM的隐式动态用户画像框架，包含可适配多领域的分类体系与基于LLM的画像生成方法。为验证其有效性，我们将其应用于ITSec领域，利用故障排查对话推断用户技术熟练度。具体构建了ProfiLLM[ITSec]版本，在263个模拟用户的1,760次类人对话上进行评估。结果表明，ProfiLLM[ITSec]能快速且准确地推断出用户画像，单次提示后实际与预测得分差距缩小55%至65%，随后出现微小波动并持续优化。此外，本文还提出基于LLM的角色模拟方法、一个结构化的ITSec能力分类体系、代码库及对话数据集，以支持后续研究。
原文摘要 · Abstract (English)
Despite significant advancements in conversational AI, large language model (LLM)-powered chatbots often struggle with personalizing their responses according to individual user characteristics, such as technical expertise, learning style, and communication preferences. This lack of personalization is particularly problematic in specialized knowledge-intense domains like IT/cybersecurity (ITSec), where user knowledge levels vary widely. Existing approaches for chatbot personalization primarily rely on static user categories or explicit self-reported information, limiting their adaptability to an evolving perception of the user's proficiency, obtained in the course of ongoing interactions. In this paper, we propose ProfiLLM, a novel framework for implicit and dynamic user profiling through chatbot interactions. This framework consists of a taxonomy that can be adapted for use in diverse domains and an LLM-based method for user profiling in terms of the taxonomy. To demonstrate ProfiLLM's effectiveness, we apply it in the ITSec domain where troubleshooting interactions are used to infer chatbot users' technical proficiency. Specifically, we developed ProfiLLM[ITSec], an ITSec-adapted variant of ProfiLLM, and evaluated its performance on 1,760 human-like chatbot conversations from 263 synthetic users. Results show that ProfiLLM[ITSec] rapidly and accurately infers ITSec profiles, reducing the gap between actual and predicted scores by up to 55--65\% after a single prompt, followed by minor fluctuations and further refinement. In addition to evaluating our new implicit and dynamic profiling framework, we also propose an LLM-based persona simulation methodology, a structured taxonomy for ITSec proficiency, our codebase, and a dataset of chatbot interactions to support future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。