GPT-1 验证生成式预训练

论文标题把方法说得很直白：Improving Language Understanding by Generative Pre-Training。2018 年，OpenAI 后来被称为 GPT-1 的那篇工作，配方只有两步。

第一步：在 BooksCorpus 上做语言模型预训练。模型是 12 层 Transformer 解码器，约 1.17 亿参数，隐层宽度 768，12 个注意力头，上下文 512 个 token。目标是标准的下一个词预测。书籍语料提供连续长散文，比打乱的句子袋更适合学篇章级依赖。

第二步：主体权重不动，用有监督数据微调。文本蕴含、问答、语义相似度、分类——这些任务被改写成模型能读的序列：前提与假设拼接，问题与选项拼接，必要时加上分隔符与任务式标记。输出端接上线性层，在任务标签上继续训练。同一套 BPE 词表与位置编码贯穿两个阶段。

论文报告在 12 个任务里有 9 个刷新当时最佳结果。历史更在意的不是分数今天还成不成立，而是迁移方式：同一具解码器，先在无标注文本上学会「接下来大概会写什么」，再在小得多的标注集上学会「这个问题要输出哪个标签」。词向量只能迁移词级几何；这里迁移的是整段上下文里的条件结构。

GPT-1 仍然依赖微调，也还没展示后来那种只靠提示就切换任务的流畅度。它的位置在于把「通用预训练 + 任务适配」写成可扩缩的简单故事：加层、加宽、换更大语料，骨架可以不动。与同时期的双向编码器路线相比，生成式预训练对右侧上下文利用不足，却天然会续写。BooksCorpus 避开部分网页噪声，也限制了领域覆盖；其余三个未刷榜的任务，则暴露出格式敏感与数据仍不足。

1.17 亿参数在后来的表上毫不起眼。2018 年它足够证明：Transformer 解码器不只是翻译系统的半边，也可以先当语言的通用压缩器，再当多种理解基准的共享骨架。十二个有监督任务被写成同一种序列接口——标注格式开始朝模型靠拢，而不是永远让模型朝每个任务的传统特征集靠拢。接口一统一，扩展规模才有抓手。

九项刷新最佳会旧。两段式配方不会：先无标注语言建模，再有监督适配。句子不长，却足以组织此后数年的实验预算。

The title states the method without ceremony: Improving Language Understanding by Generative Pre-Training. OpenAI’s 2018 paper—later called GPT-1—has a two-step recipe.

Step one: language-model pretraining on BooksCorpus. The model is a 12-layer Transformer decoder with about 117 million parameters, hidden size 768, twelve attention heads, and a 512-token context. The objective is ordinary next-token prediction. Book text supplies long contiguous prose better suited to document-level dependence than bags of shuffled sentences.

Step two: keep the body, fine-tune on labeled data. Entailment, question answering, semantic similarity, and classification are rewritten as sequences the model can read: premise concatenated with hypothesis, question with options, delimiters and task-style markers where needed. A linear head trains on the task labels. The same BPE vocabulary and positional encodings span both stages.

The paper reported state-of-the-art results on nine of twelve tasks. What matters historically is less whether every score still holds than how transfer worked: one decoder body first learns, on unlabeled text, what tends to come next, then learns on much smaller labeled sets which label a problem should emit. Word vectors transfer lexical geometry; this transfers conditional structure over spans of context.

GPT-1 still needed fine-tuning and did not yet show the later fluency of switching tasks by prompt alone. Its place is the simple, scalable story of general pretraining plus task adaptation: deepen the stack, widen the hidden size, grow the corpus, and leave the skeleton unchanged. Against contemporary bidirectional encoder lines, generative pretraining underuses right-side context but naturally continues text. BooksCorpus avoided some web noise and also limited domain coverage; the three tasks that missed the top of the board exposed format sensitivity and remaining data limits.

117 million parameters look modest on later tables. In 2018 they were enough to show that a Transformer decoder was not only half of a translation system. It could first compress language as a general model, then serve as a shared backbone for many understanding benchmarks. Casting twelve supervised tasks as one sequence interface meant annotation formats could move toward the model, instead of forever forcing the model toward each task’s traditional feature set. Once the interface unified, scale had a handle.

Nine of twelve then-best scores would age. The two-stage recipe would not: unlabeled language modeling, then supervised adaptation. Short sentence, long enough to organize experimental budgets for years.