BERT 发布

训练时，大约百分之十五的词被挖掉。

2018 年，Jacob Devlin 等人要解决的问题很具体。GPT-1 已经证明生成式预训练有效，但它从左到右读文本，做阅读理解或句子关系判断时只能看见单向上下文；ELMo 提供了双向信息，却把浅层拼接挂在任务相关的架构上。有没有一种深层双向表示，可以预训练一次，再以很小代价接到许多理解任务上？

答案是掩码语言建模（MLM）。编码器必须根据左右两侧同时可见的上下文，把被遮的 WordPiece token 填回去：其中 80% 换成 [MASK]，10% 随机替换，10% 保持原词，以减轻预训练与微调的不一致。下一句预测（NSP）则给一对句子，判断第二句是否原文中的后继。模型主体是 Transformer 编码器：BERT-Base 约 1.1 亿参数（12 层、隐层 768、12 头），BERT-Large 约 3.4 亿参数（24 层、隐层 1024、16 头）。预训练语料包括 BooksCorpus 与英文维基百科，规模约 33 亿词。掩码是一种人为伤口；愈合伤口所需的上下文，恰好逼模型同时看见前后文。

微调阶段几乎无礼地轻：在 [CLS] 或 span 表示上接小型分类或指针层，用任务数据更新参数，不必为每个任务重设计整网。论文报告 BERT-Large 在 GLUE 平均分、SQuAD v1.1/v2.0 等共 11 项任务上刷新当时最佳结果；SQuAD v1.1 测试集 F1 进入 90 分段，GLUE 平均分也大幅抬升。数字会随复现与后续模型被超过，机制却留了下来：先在大规模无标注文本上学会双向条件，再把表示当作通用初始化。

输入表示经过精心设计：WordPiece 子词、位置嵌入与分段嵌入相加，使单句与句对任务共享同一编码器接口。[CLS] 汇总序列用于分类，[SEP] 分隔句子。微调超参相对简单——学习率、batch、epoch 在小网格内搜索——这降低了工业落地门槛。此后“先下载 BERT，再换任务头”成为默认剧本，直到更大的生成式模型改写剧本为止。

BERT 擅长填空式理解、分类、抽取与问答，并不直接做开放式长文生成——那是解码器路线的事。它真正改变的是 NLP 工程默认值：与其从随机初始化为每个任务苦练，不如下载一个预训练检查点。模型中心、Hugging Face 式分发、以及“先预训练再微调”的行业口语，都在这条双向编码器上找到了早期样板。

工业落地往往比论文表格更吵杂。搜索排序、内容审核、客服意图分类与文档问答，纷纷以预训练检查点为初始化，再在自有日志上微调。显存占用、序列长度截断与领域词表不匹配，成为工程日常。下一句预测后来被证明并非总是必要，掩码比例与掩码策略也有大量后续工作，但这些修正都建立在“双向编码器预训练”已被接受的前提上。生成式大模型兴起后，纯编码器路线不再占据新闻标题，却仍在检索、分类与嵌入服务中以蒸馏与小模型形式大量运行——理解任务并不都需要会写长文。

三亿四千万参数的大号编码器，在当年硬件上微调已需小心批量与序列长度。资源约束没有阻止范式迁移，反而塑造了工程习惯：先压缩或蒸馏，再上线。十一项任务上的刷新会过时，微调范式却留下了。下载检查点、换任务头、小网格搜学习率——这套动作成为自然语言处理工程的肌肉记忆。

During training, about fifteen percent of the words were cut out.

In 2018 Jacob Devlin and coauthors faced a concrete gap. GPT-1 had shown generative pretraining worked, but left-to-right reading limited tasks that need both sides of a sentence. ELMo supplied bidirectional signals, yet as shallow concatenations attached to task-specific architectures. Could a deep bidirectional representation be pretrained once and attached cheaply to many understanding tasks?

Masked language modeling (MLM) was the answer. The encoder must recover the originals using context on both sides at once: of the corrupted WordPiece tokens, 80% are replaced with [MASK], 10% with a random token, and 10% left unchanged, so the pretrain/fine-tune mismatch softens. Next-sentence prediction (NSP) takes sentence pairs and asks whether the second follows the first in the original text. The body is a Transformer encoder: BERT-Base has about 110 million parameters (12 layers, hidden size 768, 12 heads); BERT-Large about 340 million (24 layers, 1024, 16 heads). Pretraining used BooksCorpus and English Wikipedia, on the order of 3.3 billion words. Masking is an artificial wound; healing it forces the model to see both directions at once.

Fine-tuning is almost rude in its lightness: attach a small classification or span head on the [CLS] token or token states, update on task data, and skip redesigning the whole net per problem. The paper reported new state-of-the-art results on eleven tasks including the GLUE average and SQuAD v1.1/v2.0; SQuAD v1.1 test F1 entered the 90s, and GLUE averages rose sharply. Numbers age as later models pass them; the mechanism stayed. Learn bidirectional conditioning on large unlabeled text first, then treat the representation as a universal initialization.

Input design was deliberate: WordPiece, position embeddings, and segment embeddings sum so single-sentence and sentence-pair tasks share one encoder interface. The classification token pools a sequence; separators mark boundaries. Fine-tuning hyperparameters lived in a small grid—learning rate, batch size, epochs—lowering the industrial floor. “Download BERT, swap the task head” became the default script until larger generative models rewrote the script.

BERT excels at cloze-style understanding, classification, extraction, and question answering. It does not directly do open-ended long-form generation—that is decoder territory. What it changed was the default in NLP engineering: rather than train every task from random weights, download a pretrained checkpoint. Model hubs, Hugging Face-style distribution, and the industry phrase “pretrain then fine-tune” found an early template in this bidirectional encoder.

Industrial deployment was noisier than paper tables. Search ranking, moderation, intent classification, and document QA all initialized from pretrained checkpoints, then fine-tuned on private logs. Memory, sequence truncation, and domain vocabulary mismatch became daily engineering. Later work questioned next-sentence prediction and mask rates, but those revisions assumed bidirectional encoder pretraining was already the default. Generative models took the headlines; encoder stacks still run, distilled, inside classification and retrieval services. Understanding tasks do not all need to write long prose.

A 340-million-parameter encoder was already heavy to fine-tune on 2018 hardware. Resource pressure did not stop the paradigm shift; it shaped the habit—compress or distill, then ship. Scores on eleven tasks go stale; the fine-tuning muscle memory remains: download a checkpoint, swap a head, search a small hyperparameter grid.