Chinchilla 修正算力最优缩放配比

2800 亿参数的 Gopher，败给了 700 亿参数、却吃了约 1.4 万亿 token 的 Chinchilla。

2022 年 3 月 29 日，DeepMind 把一篇很长的经验论文丢上 arXiv：Training Compute-Optimal Large Language Models。第一作者 Jordan Hoffmann 等人要回答的问题，听起来像采购会的议程：给定固定的训练算力，模型参数该多大、数据该喂多少，才不浪费 FLOPs？会议桌上若只有“再加参数”一种语言，预算审查会默认赞成更大的 (N)；这篇论文试图给第二种语言：token 是否够。

背景是 Kaplan 等人 2020 年的缩放曲线。那些曲线让“加大参数”变得可辩护，也让一些团队在数据预算相对固定时把模型越做越大。Hoffmann 团队的诊断很刺耳：许多公开的大模型显著“训练不足”（undertrained）——参数堆上去了，token 没有按比例跟上。损失曲线在参数轴上好看，并不保证下游任务已经吃饱数据。为了重新标定，他们训练了 400 多个语言模型，规模从约 7000 万参数到超过 160 亿参数，训练 token 从 50 亿到 5000 亿量级，在网格上拟合损失。网格的意义在于：配比不是纸上臆测，而是可画等高线的实验结果。

三条估计路径指向同一条工程口号：在算力最优的制度下，模型大小与训练 token 数量应近似同步放大——参数翻倍，token 也大致翻倍。口号需要实验条件：架构族、数据混合、优化器与评估口径都写在论文里；它不是物理学常数，而是一张在 DeepMind 网格上测得的等高线。换数据质量、换重复策略、换推理时额外计算，等高线会移动。作者自己也把结论放在“当前设定”下陈述，而不是宣布缩放研究结束。

验证用的是同算力对照。Gopher 有 2800 亿参数；Chinchilla 把参数降到 700 亿，把训练数据提到约 1.4 万亿 token，占用与 Gopher 相当的训练计算。结果在作者报告的大范围下游任务上全面且显著更好；MMLU 平均准确率 67.5%，较 Gopher 提高逾 7 个百分点。更小的稠密模型还意味着微调与推理更省——论文把这一点写成对“下游使用”的直接好处，而不只是排行榜第一名。对要服务真实流量的人，激活参数更小往往比训练时的名义规模更刺痛账单。

影响写在训练计划表上。研究经理开始问：我们的 70B 是否已经看见足够多的 token？还是只是在用更大的 (N) 掩盖数据饥饿？数据采集、清洗与去重预算因此获得与芯片订单同等的发言权。LLaMA 等后续工作强调“相对较小的模型 + 充足 token”，与这条修正同向，尽管数据配比、词表与训练细节各有配方。Chinchilla 没有取消缩放定律，它改写了缩放时优先拧哪一个旋钮。Kaplan 原文确立“规模变量可拟合、可外推”的习惯；Chinchilla 调整在特定制度下什么叫算力最优。两者是前后修订，不是互相消灭。

后来的世界会继续修正等高线：重复数据、高质量过滤、课程学习、推理时计算、混合专家架构，都会让“最优”移动。过度解读成“永远该训 20 token 每参数”一类口诀，会在新数据制度下撞墙。成熟的读法是把 Chinchilla 当作一次可证伪的预算假说更新：在 2022 年的网格与任务上，吃饱的 70B 可以打败饿着的 280B。对仍要提前数月锁定芯片的人来说，这句话比“参数越大越先进”更得罪人，也更有用——因为它允许你在表格里写下：也许该买的是数据管线，而不只是更多层的权重。

A 280-billion-parameter Gopher lost to a 70-billion-parameter Chinchilla that had eaten about 1.4 trillion tokens.

On 29 March 2022 DeepMind put a long empirical paper on arXiv: Training Compute-Optimal Large Language Models. Jordan Hoffmann and coauthors asked a procurement-meeting question: given a fixed training FLOPs budget, how large should the model be and how many tokens should it see? If the only language at the table is “add parameters,” budget review defaults to larger (N). This paper tried to supply a second language: are there enough tokens?

The backdrop was Kaplan et al.’s 2020 scaling curves. Those curves made “add parameters” defensible—and nudged some teams to grow (N) while data budgets lagged. Hoffmann’s diagnosis is sharp: many public large models were significantly undertrained—parameter counts rose without matching token counts. A pretty loss curve along the parameter axis does not guarantee that downstream tasks have been fed enough data. To recalibrate, the team trained over 400 language models from about 70 million to over 16 billion parameters on five to 500 billion tokens, fitting loss across the grid. The grid’s point is that the ratio is not pen-and-paper guesswork; it is an experiment that can draw contour lines.

Three estimation approaches pointed at one engineering slogan: for compute-optimal training in their regime, model size and training tokens should scale roughly in tandem—double parameters, roughly double tokens. The slogan carries experimental conditions: architecture family, data mix, optimizer, and eval protocol live in the paper. It is not a physical constant; it is a contour map measured on DeepMind’s grid. Change data quality, repetition policy, or test-time compute, and the contours move. The authors state the conclusion under their present setting rather than declaring scaling research finished.

Validation used matched compute. Gopher held 280 billion parameters. Chinchilla cut parameters to 70 billion, raised training data to about 1.4 trillion tokens, and spent comparable training compute. On the authors’ broad downstream suite it won uniformly and significantly; average MMLU accuracy reached 67.5%, more than seven points above Gopher. A smaller dense model also meant cheaper fine-tuning and inference—the paper lists that as a direct benefit for downstream use, not only a leaderboard rank. For people serving real traffic, fewer active parameters often sting the bill more than nominal training size.

Effects land on planning spreadsheets. Managers started asking whether their 70B had seen enough tokens, or whether a larger (N) was masking data hunger. Data collection, cleaning, and deduplication budgets gained a voice equal to chip orders. Later work such as LLaMA stressed comparatively smaller models with ample tokens in the same direction, even when recipes differed in data, vocabulary, and schedule. Chinchilla did not cancel scaling laws; it rewrote which knob to turn first when FLOPs are scarce. Kaplan established the habit that scale variables can be fit and extrapolated; Chinchilla adjusted what “compute-optimal” means under a given regime. They are successive revisions, not mutual annihilation.

Later practice keeps moving the contour—deduplication, quality filters, curriculum learning, test-time compute, mixture-of-experts all shift “optimal.” Over-reading into slogans like “always train 20 tokens per parameter” collides with new data regimes. A mature reading treats Chinchilla as a falsifiable budget update: on 2022’s grid and tasks, a well-fed 70B can beat a hungry 280B. For anyone locking chips months ahead, that sentence is ruder—and more useful—than “bigger parameters are progress,” because it lets you write in the table that you may need data pipelines, not only deeper weight files.