Mistral 发布 Mixtral 8x7B 混合专家模型

“8x7B”看起来像一道简单乘法题，Mixtral 却要求读者先放下参数表上的直觉。2023 年 12 月 11 日，Mistral AI 发布 Mixtral 8x7B：一个稀疏混合专家模型，并以 Apache 2.0 许可开放权重。它的吸引力既来自结构，也来自团队熟悉、可商用、可修改和再分发的许可文本。

模型需要同时记两本账。第一本是总容量：约 467 亿参数，每层的前馈部分包含 8 组专家。第二本是一次处理某个 token 时真正参与计算的部分：路由器只选择其中 2 个专家，因此每 token 激活约 129 亿参数。专家池决定模型可以容纳多少不同模式，路由决定这一刻为哪些容量付计算费用。

这并不意味着部署一台 Mixtral 就等同于部署普通 13B 稠密模型。未激活的专家仍需要存储，检查点和显存布局要按更大的总参数准备；批处理、专家并行、路由实现和内存通信都会影响真实延迟。纸面激活量解释了计算潜力，不能代替机器上的吞吐测试。

发布材料给出 32k 上下文，强调英语、法语、意大利语、德语、西班牙语和代码能力。同日提供的 Instruct 版本经过监督微调与 DPO，Mistral 报告其 MT-Bench 为 8.3，并声称在多项基准上超过 Llama 2 70B、接近或超过 GPT-3.5。这些句子的主语必须保留为发布方；是否适合具体业务，仍需在自己的数据、延迟与硬件上复测。

Apache 2.0 让法务路径相对直接，却不替部署者承担训练数据、输出合规和内容安全责任。工程团队得到的是一个容量较大、激活较稀疏的可商用权重，以及新的优化对象：专家怎样分布，路由是否均衡，推理框架能否高效调度。宽松许可降低进入门槛，MoE 结构则把复杂度移到运行时。

Mixtral 让开放模型的比较多出了一列，不能再只问“有多少参数”。完整答案至少包含总参数、每 token 激活量、许可和实测吞吐。专家池像一座有八间厨房的餐厅；每份订单只开两间火，但整栋楼的租金仍然存在。

“8x7B” looked like a simple multiplication problem, but Mixtral required readers to set aside their usual intuition about parameter tables. On 11 December 2023, Mistral AI released Mixtral 8x7B, a sparse mixture-of-experts model with weights under Apache 2.0. Its appeal came from both the structure and a familiar license allowing commercial use, modification, and redistribution.

The model required two ledgers. The first recorded total capacity: about 46.7 billion parameters, with eight feed-forward experts in each layer. The second recorded what participated when processing a particular token: a router selected only two experts, leaving about 12.9 billion parameters active per token. The expert pool determined how many patterns the model could contain; routing determined which part of that capacity incurred compute at a given moment.

That did not make deploying Mixtral equivalent to deploying an ordinary dense 13B model. Inactive experts still had to be stored, and checkpoint and memory layout reflected the larger total. Batching, expert parallelism, routing implementation, and communication all affected real latency. Paper active-parameter counts explained compute potential; they could not replace a throughput test on actual hardware.

Release materials listed a 32k context and emphasized English, French, Italian, German, Spanish, and code. An Instruct version trained with supervised fine-tuning and DPO arrived at the same time. Mistral reported an MT-Bench score of 8.3 and claimed that Mixtral exceeded Llama 2 70B on many benchmarks while matching or exceeding GPT-3.5 on several measures. The subject of those sentences must remain the publisher. Suitability still had to be retested on a team's own data, latency targets, and machines.

Apache 2.0 made the legal path relatively direct without assuming responsibility for training-data provenance, output compliance, or content safety. Engineering teams received commercially usable weights with high total capacity and sparse activation, along with new optimization problems: where experts lived, whether routing remained balanced, and whether the inference stack scheduled them efficiently. Permissive licensing lowered the entrance barrier; MoE moved complexity into the runtime.

Mixtral added a column to open-model comparison. “How many parameters?” no longer produced a complete answer. The minimum account now included total parameters, active parameters per token, license, and measured throughput. The expert pool was like a restaurant with eight kitchens: each order opened only two, but the rent for the whole building still existed.