构建首个风格转换的提示恢复基准数据集，助力逆向还原大模型输入。
StyleRec: A Benchmark Dataset for Prompt Recovery in Writing Style Transformation
- 用大模型辅助生成高质量数据，确保提示恢复任务的可靠性。
- 微调和单样本学习效果最佳，但传统相似度指标不适用。
- 适合研究提示逆向工程、模型可解释性的学者与工程师。
提示恢复（Prompt Recovery）——从大语言模型（LLMs）输出中重建输入提示——随着模型普及变得日益重要。由于多数用户仅通过API访问模型，无法获取内部权重，只能依赖输出和logits，这使得恢复难度增加。本文聚焦于风格转换与改写任务中的提示恢复，而非常见的问答场景。我们提出一个由大模型辅助构建的数据集，采用多种技术保障质量，并测试了零样本、少样本、越狱、思维链、微调以及一种新颖的规范提示回退策略（用于表现不佳情况）。实验表明，单样本学习和微调效果最优，但揭示了传统句子相似度指标在评估提示恢复时存在缺陷。贡献包括：(1) 一个基准数据集；(2) 全面的提示恢复策略实验；(3) 当前评估指标局限性的识别，推动无结构输入提示的通用提示恢复研究发展。
原文摘要 · Abstract (English)
Prompt Recovery, reconstructing prompts from the outputs of large language models (LLMs), has grown in importance as LLMs become ubiquitous. Most users access LLMs through APIs without internal model weights, relying only on outputs and logits, which complicates recovery. This paper explores a unique prompt recovery task focused on reconstructing prompts for style transfer and rephrasing, rather than typical question-answering. We introduce a dataset created with LLM assistance, ensuring quality through multiple techniques, and test methods like zero-shot, few-shot, jailbreak, chain-of-thought, fine-tuning, and a novel canonical-prompt fallback for poor-performing cases. Our results show that one-shot and fine-tuning yield the best outcomes but highlight flaws in traditional sentence similarity metrics for evaluating prompt recovery. Contributions include (1) a benchmark dataset, (2) comprehensive experiments on prompt recovery strategies, and (3) identification of limitations in current evaluation metrics, all of which advance general prompt recovery research, where the structure of the input prompt is unrestricted.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。