通过最小化任务测试语言模型的创造力极限，发现现有方法有盲点。
Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction
- 设计抽象算法任务模拟真实世界创造性挑战
- 多标记方法在生成多样性和原创性上优于单标记预测
- 输入层注入噪声比输出层温度采样更有效且保持连贯
我们设计了一套最小化的算法任务，作为开放性现实任务的松散抽象。这使我们能够清晰、可控地量化当前语言模型的创造性边界。与需要远见和创造性思维的真实任务类似，这些任务要求隐式的、开放性的随机规划步骤，或（a）在抽象知识图谱中发现新关联（如文字游戏、类比推理或科研），或（b）构建新模式（如设计数学题或新蛋白质）。在这些任务中，我们实证并概念性地论证了下一词预测的短视性；相比之下，无教师训练和扩散模型等多标记方法在生成多样化和原创性输出方面表现更优。其次，为激发随机性而不损害连贯性，我们发现向输入层注入噪声（称为种子条件）的效果出人意料地不亚于（甚至在某些条件下优于）从输出层进行温度采样。因此，本工作提供了一个原则性强、最小化的测试平台，用于分析开放性创造性能力，并为超越下一词学习和温度采样提供了新论据。部分代码已公开于 https://github.com/chenwu98/algorithmic-creativity
原文摘要 · Abstract (English)
We design a suite of minimal algorithmic tasks that are a loose abstraction of open-ended real-world tasks. This allows us to cleanly and controllably quantify the creative limits of the present-day language model. Much like real-world tasks that require a creative, far-sighted leap of thought, our tasks require an implicit, open-ended stochastic planning step that either (a) discovers new connections in an abstract knowledge graph (like in wordplay, drawing analogies, or research) or (b) constructs new patterns (like in designing math problems or new proteins). In these tasks, we empirically and conceptually argue how next-token learning is myopic; multi-token approaches, namely teacherless training and diffusion models, comparatively excel in producing diverse and original output. Secondly, to elicit randomness without hurting coherence, we find that injecting noise at the input layer (dubbed seed-conditioning) works surprisingly as well as (and in some conditions, better than) temperature sampling from the output layer. Thus, our work offers a principled, minimal test-bed for analyzing open-ended creative skills, and offers new arguments for going beyond next-token learning and temperature sampling. We make part of the code available under https://github.com/chenwu98/algorithmic-creativity
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。