让大模型一次算出任意前后文条件下的生成结果
Simplifying the Modeling of Arbitrary Conditionals in Natural Language
- 在因果Transformer上做简单修改，支持任意上下文条件
- 单次前向传播即可处理过去、未来或混合条件
- 不改变原有训练方式，适合现有大模型微调
因果Transformer通过自回归分解联合分布来建模序列，支持高效的从左到右解码和条件似然计算。然而，它们无法有效采样或评估任意条件——例如一段文本同时依赖过去和未来标记。现有方法虽试图解决此问题，但常导致条件建模不佳和生成质量下降。本文提出任意条件GPT（AC-GPT），仅对标准因果Transformer进行简单修改，即可在单次前向传播中实现任意条件的评估与采样，包括过去、未来及混合上下文。与先前方法不同，该方法保持原有的从左到右顺序和下一个词预测目标，确保模型性能和训练效率。关键优势在于兼容现有大模型，可直接用于微调。实验表明，该方法在任意条件建模上优于基线，且未降低标准自回归性能。
原文摘要 · Abstract (English)
Causal Transformers model sequences through an autoregressive factorization of the joint distribution, which enables efficient left-to-right decoding and conditional likelihood computation. However, they cannot tractably sample from or evaluate arbitrary conditionals -- e.g., a block of text conditioned on past and future tokens. Recent work aims to solve this problem through novel architectures, but they often lead to sub-optimal modeling of such conditionals and degraded generations. We propose Arbitrary Conditionals GPT (AC-GPT) which introduces a simple modification to standard causal Transformers to enable evaluating and sampling from arbitrary conditionals -- including past, future, and mixed contexts -- within a single forward pass. Unlike prior approaches, our method preserves the standard left-to-right ordering and next-token prediction objective essential for both strong performance and efficient training on natural language. Crucially, this compatibility allows existing LLMs to be fine-tuned for arbitrary conditioning. Our empirical results indicate that our method outperforms baselines on modeling arbitrary conditionals, without degrading standard left-to-right performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。