LSTM 长短期记忆网络

循环神经网络理论上记得过去，训练时却经常忘得很快。当前输出发生错误，梯度沿时间一步步回传；每经过一次容易饱和的非线性，它都可能再缩小一点。等信号走到很早的输入，已经接近于零。网络于是学会依赖最近几个符号，跨越较长时间的信用分配仍停留在纸面上。

Sepp Hochreiter 与 Jürgen Schmidhuber 把问题改写成一条路径的设计。1997 年发表的 LSTM 论文提出记忆单元，让内部状态可以跨时间步保存，并用可学习的门控制信息何时写入、何时读出。

所谓“恒定误差传送”关注的正是反向传播：在合适条件下，误差信号可以沿记忆状态的环路继续走，而不在每一步都被压扁。

可以把普通循环层想成一条不断漏水的管道，远处的压力很难传到终点。LSTM 增加了蓄水段和阀门，让状态不必在每个时间步被完全重写。门是连续可微的，不是人工设定的开关，因此网络仍能端到端使用梯度下降训练。改变的不是损失函数，而是损失回去寻找责任时走的内部道路。

1997 年的结构不应与后来教材里最常见的完整门控图直接画等号。遗忘门等组件来自后续扩展，算力、初始化和训练技巧也经历了多年改进。原始工作的关键，是把“长程依赖很难学”从一句经验抱怨拆成具体的优化病理，再给出一个可以训练的记忆机制。

当误差信号能够跨过更多时间步，语音、手写识别和序列建模中的延迟关系开始更容易优化。进入 2010 年代，多层 LSTM 成为许多语音系统与神经机器翻译模型的常用骨干，Seq2Seq 也依赖这类可训练的循环单元读取和生成整段序列。这并不保证任意长度的记忆，工程上仍需要控制序列长度、初始化和梯度。

LSTM 没有解决语言理解的所有问题，它解决的是更窄也更基础的一件事：让过去在训练中不至于太早失去发言权。当当前输出出错时，那个很久以前的输入终于有机会收到消息——“这里的责任，也有你的一部分。”

Recurrent neural networks could remember the past in theory and forgot it quickly during training. When a current output was wrong, the gradient traveled backward through time. Each passage through a saturating nonlinearity could make the signal smaller. By the time it reached an early input, almost nothing remained. The network learned to depend on recent symbols while credit assignment across a long delay remained mostly a promise on paper.

Sepp Hochreiter and Jürgen Schmidhuber reframed the problem as the design of a path. Their 1997 LSTM paper proposed a memory cell whose internal state could persist across time, with learned gates controlling when information was written and when it was exposed.

The “constant error carousel” referred directly to backpropagation: under suitable conditions, an error signal could continue around the memory-state loop rather than being crushed at every step.

A plain recurrent layer can be imagined as a leaking pipe through which distant pressure rarely arrives. LSTM added a reservoir and valves so that state did not have to be overwritten at every tick. The gates were continuous and differentiable rather than hand-set switches, keeping the network trainable end to end by gradient descent. The loss function did not change. What changed was the internal road the loss used to assign responsibility.

The 1997 design should not be equated directly with the complete gate diagram most familiar from later textbooks. Components such as the forget gate came through subsequent extensions, and compute, initialization, and training methods continued to improve for years. The original work's key move was to decompose “long dependencies are hard to learn” into a specific optimization pathology and then supply a trainable memory mechanism.

With error able to cross more time steps, delayed relationships in speech, handwriting recognition, and sequence modeling became more practical to optimize. During the 2010s, multilayer LSTMs became common backbones in speech systems and neural machine translation. Seq2Seq models also relied on trainable recurrent cells of this kind to read and generate whole sequences. None of that guaranteed memory of arbitrary length; engineers still had to manage sequence length, initialization, and gradients.

LSTM did not solve every problem in language understanding. It addressed something narrower and more foundational: preventing the past from losing its voice too early during training. When a present output was wrong, an input from much earlier finally had a chance to receive the message: some of the responsibility belongs here too.