Watkins 提出 Q-learning

如果问今天最热门的 AI 技术里，哪一块的理论历史最长，强化学习一定榜上有名。而在强化学习的家族谱上，有一个名字绕不过去：Q-learning。它来自 1989 年一位博士生 Chris Watkins 的博士论文，理念却朴素得惊人——维护一张表，记录「在某个状态做某件事值多少」，然后不断用实际获得的奖励更新它。

在 Q-learning 出现之前，强化学习领域的工具相当有限。很多算法要求你先有一个环境模型，才能计算怎么决策；没有模型的环境，往往只能靠手工设计规则。Watkins 想证明的是：一个智能体完全可以在不知道环境如何运转的情况下，光靠试错和一点点数学技巧，逐步逼近最优做法。

这个数学技巧就是时序差分。智能体每走一步，就用「眼前获得的奖励 + 对下一步的最好预期」来修正当前这一步的价值估计。听起来简单，但它有一个关键性质：不需要环境模型，而且能在理论上保证收敛。正是这种「无模型、自举」的干净框架，让 Q-learning 成为经典。

几十年后，当 DeepMind 想把强化学习和深度学习结合时，选中的算法恰恰是 Q-learning——DQN 就是把 Q 表换成了深度网络。从此 Q-learning 的思想流进了游戏 AI、机器人控制、推荐系统、对话系统，成为现代智能体训练的两大支柱之一。

回看 Q-learning，它提醒我们一件事：AI 领域最深远的影响，有时来自一篇读起来不难的论文、一个简单的更新公式。它没有华丽的演示，却为后来一切「AI 靠试错学会做事」的技术，提供了第一块基石。

If you ask which of today's hottest AI technologies has the longest theoretical history, reinforcement learning is high on the list. And in RL's family tree one name is unavoidable: Q-learning. It came from a 1989 doctoral thesis by Chris Watkins, and its idea is almost disarmingly simple—keep a table recording "how much is doing this in this state worth", and keep updating it with actual rewards received.

Before Q-learning, the field's toolkit was limited. Many algorithms required a model of the environment to compute decisions; for model-free environments you often fell back on hand-written rules. Watkins wanted to prove an agent could approach optimal behavior through trial and error plus a bit of math, without knowing how the environment worked at all.

That bit of math was temporal difference. With every step, the agent corrects its estimate of the current state's value using "the reward just received plus the discounted best expectation of the next state". Simple as it sounds, it had a crucial property: no environment model needed, and theoretical convergence guaranteed. That clean, model-free, bootstrapping frame made Q-learning a classic.

Decades later, when DeepMind wanted to combine RL with deep learning, the algorithm they reached for was Q-learning—DQN simply swapped the Q-table for a deep network. From there its ideas flowed into game AI, robot control, recommendation, and dialogue systems, making it one of the two pillars of modern agent training.

Looking back at Q-learning, it reminds us that the deepest impact in AI sometimes comes from an easy-to-read paper and a simple update rule. It had no dazzling demo, yet it supplied the first block for everything that later taught machines to learn by trial and error.