注意力机制进入神经机器翻译

整句源文被压进一个固定长度向量，再交给解码器逐词吐出译文。

句子一长，行李箱就不够用。编码器—解码器翻译不是“不懂语言”，而是被迫用一个向量装下整趟旅程。Sutskever 等人的 Seq2Seq 已经证明端到端映射可行，也把这个行李箱问题摆上桌面：信息全挤在编码器末状态里，长句尤其难受。

Dzmitry Bahdanau、Kyunghyun Cho 与 Yoshua Bengio 的做法听起来几乎太直接：别只交一个行李箱，让解码器每走一步都重新打开源句。论文题为 Neural Machine Translation by Jointly Learning to Align and Translate，预印本在 2014 年出现，会议版本常见记为 ICLR 2015。结构仍是循环编码器加循环解码器，但在预测每一个目标词之前，先根据当前解码状态，对编码器各时间步的隐状态算一组权重——权重高的源位置“被看重”——再把加权后的向量当作这一步的上下文，和已生成前缀一起预测下一词。对齐（align）与翻译（translate）共用同一套可训练参数，不再先跑独立的词对齐模块再交给短语表。

这就是后来口语里“注意力”（attention）在神经机器翻译里的早期定型：不是平均看所有源词，也不是硬选一个源词，而是软的、可微的加权读取。训练信号来自翻译损失本身；模型为了把下一词预测对，会学会在合适的时候把质量放在合适的源位置上。论文还展示了注意力权重随时间变化的可视化：生成某个目标词时，高亮落在哪些源词上。那些热力图并不等于完美语言学对齐，却给研究者一种新的检查习惯——打开黑箱的一角，看看网络是否在“看”合理的地方。

相对 Seq2Seq 的改动，概念上很小，后果很大。固定向量瓶颈被放松：信息可以留在编码器的整条隐状态序列里，按需提取。长句不再完全依赖最后一步的摘要。更重要的是接口形状：任何“当前查询”对“一组键值记忆”的打分—加权—求和，都可以叙述成注意力。随后几年的变体改查询怎么构造、分数怎么算、是否多头、是否去掉循环，但“每一步回看源侧”的许可，是在这篇工作里写进主流叙事的。

需要划清边界。这里的注意力仍坐在循环网络之上，不是 2017 年 Transformer 里那种完全由自注意力堆成的并行架构；编码器、解码器的时间步进依然存在。它也没有消灭数据与词表工程：稀有词、拷贝机制、子词切分仍会在后续系统里单独处理。统计短语机器翻译在 2014–2015 年仍是工业强手；神经模型带着注意力开始追赶，并不等于立刻清空机房。

实现层面，注意力分数通常由当前解码状态与各源位置隐状态的兼容函数给出，再经归一化变成权重；不同论文会换点积、加性网络或其它打分式，但“查询对记忆做软选择”的骨架不变。训练仍可用教师强制：真实目标前缀驱动解码状态，注意力在每一步重新计算。对工程团队，这意味着显存里要保住整条编码器状态序列，长句的代价从“信息挤爆一个向量”部分转成“每步对源长度做一次加权读”。

读这篇论文，值得盯住“联合学习”四个字。对齐不是后处理，也不是必须由语言学规则提供的外部表；它是生成过程中的中间计算，梯度可以从翻译错误流回注意力权重。当解码器卡在某个目标词上，它可以多看源句里对应的片段，而不是死盯一个已经压缩过头的句向量。行李箱还在，但旅行者被允许随时回头打开地图——这一许可，改变了序列到序列模型默认该怎么读输入。

The whole source sentence was crushed into one fixed-length vector, then handed to a decoder that emitted words one by one.

As sentences grew, the suitcase ran out of room. Encoder–decoder translation was not simply “bad at language”; it was forced to pack an entire trip into one vector. Seq2Seq had already shown that end-to-end mapping could work—and that the suitcase problem was real: everything sat in the encoder’s final state, and long sentences suffered most.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio’s move was almost too plain to state: stop delivering only one suitcase; let the decoder reopen the source at every step. Their paper, Neural Machine Translation by Jointly Learning to Align and Translate, appeared as a 2014 preprint and is commonly tied to ICLR 2015. The skeleton remains a recurrent encoder and recurrent decoder, but before each target word the model uses the current decoder state to score encoder hidden states across time—positions with higher weight are “looked at” more—then forms a weighted context vector that, with the prefix so far, predicts the next token. Alignment and translation share one trainable objective; there is no separate word-alignment stage feeding a phrase table.

That is the early, durable shape of what people later call attention in neural MT: not uniform averaging over all source words, and not a hard pick of one source word, but a soft, differentiable weighted read. The training signal is the translation loss itself. To predict the next word correctly, the model learns when to put mass on which source positions. The paper also shows attention-weight visualizations over time: as a target word is generated, which source words light up. Those heatmaps are not perfect linguistic alignments, but they give researchers a new inspection habit—open a corner of the black box and ask whether the net is looking somewhere plausible.

Conceptually the change from Seq2Seq is small; the consequences are large. The fixed-vector bottleneck loosens: information can stay in the full sequence of encoder states and be fetched on demand. Long sentences no longer depend entirely on a final summary step. More important is the interface shape: any “current query” scoring a set of key–value memories, then weighting and summing, can be told as attention. Later variants change how queries are built, how scores are computed, whether there are multiple heads, whether recurrence disappears—but the license to re-read the source at each step enters the mainstream story here.

Boundaries matter. Attention in this paper still sits on recurrent nets; it is not the 2017 Transformer stack built only from self-attention and feed-forward layers. Encoder and decoder still step through time. Data and vocabulary engineering do not vanish: rare words, copy mechanisms, and subword segmentation remain separate problems in later systems. Statistical phrase-based MT was still strong in industry around 2014–2015; neural models with attention began to catch up without emptying the machine rooms overnight.

In implementation, attention scores usually come from a compatibility function between the current decoder state and each source hidden state, then normalize into weights. Papers swap dot products, additive networks, or other scoring forms; the skeleton—“a query soft-selects a memory”—stays. Training can still use teacher forcing: true target prefixes drive decoder states, and attention recomputes at every step. For engineering teams, that means keeping the full encoder state sequence in memory: the cost of long sentences partly shifts from “one vector bursting” to “a weighted read over source length at each step.”

Reading the paper, keep the phrase “jointly learning” in view. Alignment is not post-processing and need not be an external table supplied by linguistic rules; it is intermediate computation inside generation, and gradients from translation errors flow back into the attention weights. When the decoder stalls on a target word, it can look harder at the corresponding source span instead of staring at one over-compressed sentence vector. The suitcase remains, but the traveler is allowed to reopen the map—and that permission changed how sequence-to-sequence models were expected to read their inputs.