Word2Vec 发布

标题写得很老实：《向量空间中词表示的高效估计》。

2013 年，Mikolov、Chen、Corrado 与 Dean 关心的第一件事，不是诗意的语义空间，而是怎样在数十亿词的语料上，把训练词向量的计算量砍到可以日常实验的程度。此前，one-hot 把每个词当成彼此无关的坐标轴，词与词之间没有几何关系；早期神经语言模型能学到嵌入，却常因大词表 softmax 与较深结构而昂贵，难以在“整个网页时代语料”上反复试错。Google 随后开源的工具链，让工业界与实验室都能在普通服务器上扫过大规模文本，把共现统计压成几百维的实数向量。

Word2Vec 把问题收成两种浅层预测。CBOW（continuous bag-of-words）根据周围词预测中心词；Skip-gram 反过来，用中心词预测上下文窗口里的词。网络几乎没有隐藏非线性：输入是词的嵌入表，输出是对词汇表的分类。省算力的关键在训练技巧——分层 softmax 用二叉树把归一化从全词表摊到一条路径；负采样只更新出现的词和少量噪声词，避免每次对整个词表求梯度。窗口大小、负样本数与子采样高频词等超参，决定了哪些共现被强调。

论文与后续笔记展示了向量的几何性质：相近的词落在相近的位置；某些类比关系近似满足向量差，例如常被引用的 king − man + woman ≈ queen 一类结果，以及首都—国家、动词时态等模式。这很有说服力，也很容易被过度解读。向量算术不是可靠的符号演算；窗口、语料领域与目标函数都会改变哪些关系显得整齐。一词多义被压成同一个点——bank 的“银行”与“河岸”共用坐标。语料里的社会刻板印象同样会写进距离结构：职业与性别、种族相关的最近邻，后来被系统测量成可复现的几何偏差，而不是道德附录。

Skip-gram 与 CBOW 在不同语料规模上的表现并不对称：数据极多时 Skip-gram 常给出更好的稀有词向量，CBOW 则往往训练更快。子采样高频词（如“的”“the”）减少无信息的共现更新，使窗口把注意力花在中等频率的内容词上。这些选择说明 Word2Vec 首先是一套可调的估计程序，其次才是那张著名的类比幻灯片。

Word2Vec 的即时影响是工程性的。搜索排序、推荐、文本分类、命名实体识别，以及后续神经模型的输入层，忽然多了一种可下载、可微调的稠密表示。不必先训练完整语言模型，也能获得词级语义；迁移学习在 NLP 里有了廉价的第一层。它把“分布式表示”从研究论文推进到默认组件，同时为后来的 ELMo、BERT 提出了问题：静态向量无法随句子改变意义，上下文必须进入表示本身。2013 年的答案止于词表上的点；对当时的流水线，那已经足够改写第一层。

开源工具发布后，工业界把词向量当作可插拔组件：搜索同义词扩展、广告查询改写、推荐侧的文本侧特征，常常先加载一份预训练向量再视任务微调。学术基准上的类比准确率成为快速验收指标，尽管它与下游任务相关但不等价。静态词向量的时代在上下文化模型出现后结束得很快，可“用大规模无标注文本先学表示”的习惯被完整继承。

高效，在这里不是谦辞。它意味着研究者可以在可接受的墙上时钟时间内扫过海量文本，用预测任务逼出有用的几何。语义关系以副产品的形式出现；标题选择强调的，始终是估得动、估得快。当人们后来只记得类比算术的演示幻灯片时，值得回到那一行标题：先解决十亿词级的估计问题，其它故事才有材料可讲。

The title is blunt: Efficient Estimation of Word Representations in Vector Space.

In 2013, Mikolov, Chen, Corrado, and Dean cared first about cost—how to learn word vectors on corpora of billions of tokens without making every experiment a research project of its own. One-hot encodings treated every word as an unrelated axis. Earlier neural language models could learn embeddings, but large-vocabulary softmax layers and deeper networks made them expensive, hard to iterate on web-scale text. Google’s open tools soon let industry and labs sweep large corpora on ordinary servers and compress co-occurrence statistics into a few hundred real-valued dimensions.

Word2Vec reduced the problem to two shallow prediction tasks. Continuous bag-of-words (CBOW) predicts a center word from its neighbors; Skip-gram reverses the direction and predicts context words from the center. The networks carry almost no hidden nonlinearity: inputs are embeddings, outputs are classifications over the vocabulary. The real savings came from training tricks. Hierarchical softmax walks a binary tree instead of normalizing over the full vocabulary. Negative sampling updates only the true word and a handful of noise words, avoiding a full-vocabulary gradient at every step. Window size, negative-sample count, and subsampling of frequent words decide which co-occurrences get emphasized.

Papers and follow-up notes showed geometric regularities: similar words land nearby; some analogies approximately satisfy vector offsets, as in the often-cited king − man + woman ≈ queen family of examples, along with capital–country and verb-tense patterns. The results are persuasive and easy to overread. Vector arithmetic is not reliable symbolic calculus. Window size, corpus, and objective all change which relations look clean. Polysemy is crushed into a single point—bank as “financial institution” and bank as “river edge” share one coordinate. Social stereotypes in the training text enter the distance structure as well; later work measured bias as a reproducible geometric fact rather than a moral appendix.

Skip-gram and CBOW are asymmetric across corpus sizes: with abundant data Skip-gram often yields better rare-word vectors, while CBOW trains faster. Subsampling frequent tokens (“the,” and their equivalents) reduces uninformative co-occurrence updates so the window spends capacity on content words. Those choices show that Word2Vec is first a tunable estimation procedure, and only second the famous analogy slide.

Word2Vec’s immediate force was industrial. Search, recommendation, text classification, named-entity recognition, and the input layers of later neural models suddenly had a dense representation one could download and fine-tune. Teams could obtain lexical semantics without first training a full language model. Distributed representations moved from papers into default components, and they posed the problem later solved by ELMo and BERT: a static vector cannot change meaning with the sentence. The 2013 answer stopped at points on a vocabulary. That was already enough to rewrite the first layer of many NLP pipelines.

After the tools shipped, industry treated embeddings as pluggable components for synonym expansion, query rewriting, and text features in recommenders. Analogy accuracy on academic benchmarks became a quick acceptance test—related to, but not the same as, downstream task quality. Static vectors aged out once contextual models arrived; the habit of learning representations from unlabeled text first did not.

Efficiency, here, is not false modesty. It means a researcher could sweep massive text in acceptable wall-clock time and force useful geometry out of a prediction task. Semantic structure arrived as a byproduct. The title chose to emphasize what could be estimated at all, and fast. When people later remember only the analogy slides, it is worth returning to that one line: solve billion-word estimation first; the other stories need material to stand on.