让大模型同时精准预测常见和罕见用户行为
Tuning Language Models for Robust Prediction of Diverse User Behaviors
- 分两阶段微调：先保留通用知识，再用难样本平衡数据提升尾部行为预测
- 在两个真实数据集上，对高频与低频行为均实现稳定高精度预测
- 特别适合需要捕捉小众行为的智能助手场景
预测用户行为对智能助手服务至关重要，但深度学习模型常难以捕捉长尾行为。大型语言模型（LLMs）凭借其在海量语料上的预训练，蕴含丰富行为知识，具备潜力。然而，现有微调方法往往过度拟合高频的“锚点”行为，削弱了对低频“尾部”行为的预测能力。本文提出BehaviorLM，一种渐进式微调方法：第一阶段在锚点行为上微调，同时保持通用行为知识；第二阶段使用基于样本难度的均衡数据子集进行微调，提升尾部行为预测，且不牺牲锚点性能。在两个真实数据集上的实验表明，BehaviorLM能稳健预测锚点与尾部行为，并在少样本条件下有效利用LLM的行为知识掌握尾部行为预测。
原文摘要 · Abstract (English)
Predicting user behavior is essential for intelligent assistant services, yet deep learning models often struggle to capture long-tailed behaviors. Large language models (LLMs), with their pretraining on vast corpora containing rich behavioral knowledge, offer promise. However, existing fine-tuning approaches tend to overfit to frequent ``anchor'' behaviors, reducing their ability to predict less common ``tail'' behaviors. In this paper, we introduce BehaviorLM, a progressive fine-tuning approach that addresses this issue. In the first stage, LLMs are fine-tuned on anchor behaviors while preserving general behavioral knowledge. In the second stage, fine-tuning uses a balanced subset of all behaviors based on sample difficulty to improve tail behavior predictions without sacrificing anchor performance. Experimental results on two real-world datasets demonstrate that BehaviorLM robustly predicts both anchor and tail behaviors and effectively leverages LLM behavioral knowledge to master tail behavior prediction with few-shot examples.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。