构建用户写作偏好数据集，助力个性化创作模型训练
LiteraryTaste: A Preference Dataset for Creative Writing Personalization
- 采集60人自述与实际文本偏好，构建双维度偏好数据集
- 微调模型对个人与集体偏好的预测准确率分别达75.8%和67.7%
- 发现自述偏好难以反映真实偏好，适合内容个性化研究者
人们在创意写作上有不同偏好，而当前大语言模型多基于统一偏好训练。为推动个性化创意写作模型发展，我们提出LiteraryTaste数据集，包含60名用户的阅读习惯自述（陈述偏好）及对100组短篇创意文本的偏好标注（揭示偏好）。分析显示：1）个体间创意写作偏好差异显著；2）微调Transformer编码器可实现75.8%和67.7%的个人与集体揭示偏好预测准确率；3）陈述偏好在建模揭示偏好时作用有限。通过基于LLM的可解释性分析管道，我们进一步揭示了偏好差异模式。本工作旨在为个性化创意写作技术奠定基础。
原文摘要 · Abstract (English)
People have different creative writing preferences, and large language models (LLMs) for these tasks can benefit from adapting to each user's preferences. However, these models are often trained over a dataset that considers varying personal tastes as a monolith. To facilitate developing personalized creative writing LLMs, we introduce LiteraryTaste, a dataset of reading preferences from 60 people, where each person: 1) self-reported their reading habits and tastes (stated preference), and 2) annotated their preferences over 100 pairs of short creative writing texts (revealed preference). With our dataset, we found that: 1) people diverge on creative writing preferences, 2) finetuning a transformer encoder could achieve 75.8% and 67.7% accuracy when modeling personal and collective revealed preferences, and 3) stated preferences had limited utility in modeling revealed preferences. With an LLM-driven interpretability pipeline, we analyzed how people's preferences vary. We hope our work serves as a cornerstone for personalizing creative writing technologies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。