用提示词引导大模型首词预测,提升多选题评估可靠性
Improving LLM First-Token Predictions in Multiple-Choice Question Answering via Output Prefilling
- 在输出前加结构化提示,让模型直接给出答案选项
- 多模型多数据集测试中,准确率显著提升且结果更稳定
- 无需改模型,适合快速评估和对比实验
大语言模型在多选题问答任务中常使用首词概率(FTP)进行评估,即选择初始词概率最高的选项。但该方法易受干扰:模型可能给无关词分配高概率(错位),或仅将正确词作为通用开头的一部分(误读),影响评估可靠性。本文提出‘预填充攻击’——在输出前添加结构化自然语言前缀(如“正确选项是:”),引导模型生成清晰答案。该方法源自人工智能安全领域,现被重新用于提升评估效果,无需修改模型参数。实验证明,加入预填充后,多种模型与多个基准上的首词概率准确率、校准性与输出一致性均显著提升,优于标准FTP,常接近全量解码+外部分类器的性能,却效率更高。结果表明,预填充是一种简单、鲁棒且低成本的提升多选题评估可靠性的有效策略。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly evaluated on multiple-choice question answering (MCQA) tasks using *first-token probability* (FTP), which selects the answer option whose initial token has the highest likelihood. While efficient, FTP can be fragile: models may assign high probability to unrelated tokens (*misalignment*) or use a valid token merely as part of a generic preamble rather than as a clear answer choice (*misinterpretation*), undermining the reliability of symbolic evaluation. We propose a simple solution: the *prefilling attack*, a structured natural-language prefix (e.g., "*The correct option is:*") prepended to the model output. Originally explored in AI safety, we repurpose prefilling to steer the model to respond with a clean, valid option, without modifying its parameters. Empirically, the FTP with prefilling strategy substantially improves accuracy, calibration, and output consistency across a broad set of LLMs and MCQA benchmarks. It outperforms standard FTP and often matches the performance of open-ended generation approaches that require full decoding and external classifiers, while being significantly more efficient. Our findings suggest that prefilling is a simple, robust, and low-cost method to enhance the reliability of FTP-based evaluation in multiple-choice settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。