反向传播算法

先从答案的错误处往回走。

网络给出一个输出，与目标相比产生差值。输出层的权重各自对这个差值负有多少责任，可以直接计算；再往前一层，隐藏单元没有标准答案，只能依据它对后续单元的影响分摊误差信号。链式法则把这些局部导数连起来，误差便从输出端逐层传回输入方向。每个权重随后朝使总体误差降低的方向移动一点——这就是梯度下降在网络上的具体执行。新的样本到来，过程再做一遍。信息向前形成预测，责任向后形成梯度。

1986 年 10 月，戴维·鲁梅尔哈特、杰弗里·辛顿与罗纳德·威廉姆斯在《自然》发表短文 Learning representations by back-propagating errors。标题的重心落在“表征”（representations）上。论文摘要明确指出，权重反复调整后，不属于输入也不属于输出的隐藏单元会表示任务领域的重要特征，规律被这些单元之间的相互作用捕捉。它与更简单的感知机收敛过程的差别，正在于能创造有用的新特征，而不是只在固定特征上调节分类边界。

论文用对称性检测、编码/解码以及亲属关系等任务展示这种变化。对称性任务要求网络判断输入模式是否对称；编码任务迫使中间层形成压缩表示；亲属关系任务则把人物与关系映射到隐藏空间，使同一套权重支持多组查询。对亲属关系，输入并不是直接通向一个手工定义的“家族概念”；训练迫使隐藏层找到一种内部组织。研究者不必先规定每个隐藏单元必须表示什么，它们的位置由降低误差的共同需要形成。论文的说服力一部分来自这些可检查的内部结构，而不只来自最终分类对错。

反向传播的数学来源比 1986 年早得多。链式法则当然不是新发明；逆向求导、控制论中的灵敏度分析，以及 1960–1970 年代多位研究者对多层网络训练的尝试，都构成先导。保罗·沃博斯等人更早提出过相关思想，只是未在连接主义社群中获得同等回响。三位作者的重要贡献，是把它清楚地放进多层连接主义模型，给出令人信服的实验，并展示隐藏表示为何值得研究。历史若只保留一个“发明日”，会把一条累积已久的数学路线误写成突然启示。

同样，算法可用不等于深层网络立刻好用。梯度会衰减或爆炸；S 型激活函数在饱和区导数接近零；初始化、学习率、优化方法、数据规模和计算能力都限制实际深度。论文实验使用的网络深度与宽度以今天标准看都不大，其目的是展示表示学习，而不是刷新大规模基准。1990 年代，支持向量机等方法在许多任务上更省事、更稳。反向传播提供了训练多层网络的通用语法，却没有替后来几十年的工程工作预付答案——ReLU、更好的初始化、正则化、大批量数据与 GPU 算力，都是语法之外的词汇。

论文篇幅不长，却把“隐藏层学到了什么”推到前台。对称性任务中，部分隐藏单元对特定位置关系更敏感；编码任务中，中间层维数被压低，迫使网络丢弃无关变化、保留可重建的结构。这些观察在当时连接主义争论里很重要：内部状态不必预先贴上符号标签，仍可能形成可分析的组织。批评者可以继续质疑泛化与生物学合理性；支持者则第一次有了一套共享的训练故事可讲。

反向传播改变了错误的用途。一次失败的计分，变成穿过网络、重新分配内部结构的信号。算法从结尾倒着计算，让模型能够从结果反过来塑造自己看世界的中间方式。今天自动微分框架里那句看起来平淡的 loss.backward()，执行的仍是同一基本过程：把终点的不满意，拆成对每一层参数的指示。差别在于，1986 年人们还要亲手推导链式法则在网络上的展开；现在框架替你展开，但“误差如何变成表示”的问题并没有因此自动解决。

Begin where the network is wrong.

A forward pass produces an output. Comparing that output with the target yields an error. For weights immediately connected to the output, their contribution to the error can be calculated directly. Hidden units have no answer sheet of their own. Their responsibility must be inferred from how strongly they influenced the units ahead of them. The chain rule joins those local effects, carrying a gradient from the output layer toward the input. Each weight moves a little in a direction expected to reduce the total error—gradient descent executed on a network. Another example arrives, and the accounting begins again. Information travels forward to make a prediction. Responsibility travels backward to alter the machinery that made it.

David Rumelhart, Geoffrey Hinton, and Ronald Williams published “Learning representations by back-propagating errors” in Nature in October 1986. Representations is the decisive word in the title. After repeated adjustment, they argued, hidden units—neither input nor answer—could come to encode features important to the task. Patterns would reside not in one explicitly programmed symbol but in the interactions among learned units. The contrast with a simple perceptron is exact here: the latter adjusts a boundary over fixed features; backpropagation can create useful new features in the middle.

Their experiments included symmetry detection, encoding–decoding, and family relationships. In the symmetry task, the network had to decide whether a pattern was symmetric. In the encoding task, a narrow middle layer was forced to compress and recover structure. In the kinship task, the network was not supplied with a handcrafted concept of family structure for each hidden unit to represent. It had to organize its intermediate state so that shared weights could support mappings among people and relations. The hidden layer was useful precisely because its meaning was not assigned in advance; the demands of reducing many errors together gave it structure. Part of the paper’s force came from making those internal organizations inspectable, beyond final accuracy scores alone.

The mathematics did not appear in 1986 from nowhere. The chain rule was old. Reverse-mode differentiation, sensitivity analysis in control theory, and several earlier attempts to train multilayer networks in the 1960s and 1970s form a longer lineage; Paul Werbos and others had proposed related ideas before the connectionist community absorbed them at scale. Rumelhart, Hinton, and Williams made the method clear in a multilayer connectionist setting, paired it with persuasive experiments, and showed why the learned interior deserved attention. Converting that accumulation into one invention date would make a long technical lineage look like revelation.

Nor did a general training procedure make deep networks immediately easy. Gradients could vanish or explode. Sigmoid activations flatten in saturation. Initialization, learning rates, optimization, data, and available computation limited useful depth. The networks in the paper look small by later standards; their purpose was to show representation learning, not to refresh large-scale benchmarks. Through much of the 1990s, methods such as support vector machines remained preferable on many tasks. Backpropagation supplied a reusable grammar for training multilayer models; decades of engineering—ReLU-like activations, better initialization, regularization, large datasets, and GPU throughput—would decide which sentences could be written with it.

The paper is short, yet it pushes “what the hidden layer learned” to the front. In the symmetry task, some hidden units become more sensitive to particular positional relations. In the encoding task, the middle layer is deliberately narrow, forcing the network to discard irrelevant variation and keep structure that can be rebuilt. Those observations mattered in the connectionist debates of the period: internal states need not wear hand-written symbol labels in advance and may still form analyzable organization. Critics could continue to question generalization and biological plausibility; advocates at least shared a training story that others could reimplement.

Its lasting transformation was to change the use of failure. An error ceased to be only a score at the end. Differentiated and sent backward, it became instructions for rearranging the system’s internal way of seeing. The ordinary loss.backward() call in a modern automatic-differentiation framework still performs that basic process: dissatisfaction at the end is decomposed into instructions for every parameter along the path. The difference is operational. In 1986, researchers often expanded the chain rule on the network by hand; frameworks now expand it for you. Whether error becomes useful representation still depends on architecture, data, and optimization—the grammar is shared, the sentences are not free.