Mixtral 8x7B: the European alternative to GPT-3.5 by Mistral AI

Large language models, while they continue to surprise us, are not yet perfect. Their lack of reliability and robustness remains a significant issue and is slowing down their adoption in the enterprise. To address this shortcoming, new models are emerging every day in an effort to outperform their predecessors.

Among the companies keen to develop these models is Mistral AI. This French startup, based in Paris and born in 2023, had already surprised the community with the launch of fully open source and particularly high-performance models. Their motivations did not fail to seduce us: "We are convinced that by forming our own models, publishing them openly and encouraging contributions from the community, we can build a credible alternative to the emerging AI oligopoly". Driven by these motivations, Mistral AI has just delivered a particularly promising new model to the community, which we will discuss in this article.

Large Language Models (LLMs), particularly energy-hungry technologies

According to the current paradigm, improving model performance means increasing their size. This is because language models are generally trained to have a very wide scope of action, which translates into mastery of a large number of languages and the ability to respond to numerous tasks (question answering, summary generation, translations, etc.). As a result, large language models are often delivered in several formats: 7B, 13B, 70B, etc. (i.e. "billion parameters"), with the largest formats delivering the best performance.

However, this increase in size goes hand in hand with a rise in the computational power required to train and operate these systems, which in turn leads to significant material and energy consumption. With energy efficiency now a top priority, generative AI is rightly coming under scrutiny.

This is why efforts are being made to resolve to use the major language models in their smallest variations (7B), in return for a sacrifice in performance.

Mixtral 8x7B, the Mistral AI alternative

Mistral AI has just launched Mixtral 8x7B!

Why 8x7B? Because it’s a somewhat unique model, consisting of 8 expert models and a routing model. During inference, the routing model determines which two of the eight submodels will be responsible for processing the prompt provided as input.

So, although the model has a total of 45 billion parameters (45B), only one sample (12 billion) is used to process each token in the prompt. Consequently, inference is performed with the same cost (in computing power) and latency as if the model had only 12 billion parameters, but benefits from the fact that each of the sub-models is specialized in a precise domain.

This strategy, known as SMoE (for Sparse Mixture of Experts), is not new and is enjoying renewed interest for use in deep learning. For Mistral AI, this paradigm shift has enabled them to claim that the model performs as well as, if not better than, the well-known GPT-3.5 and LLama2-70B, the most widely used OpenAI and Meta language models.

The model also supports a large pop-up window (32k tokens) and 5 languages (English, French, Italian, German and Spanish). An Instruct version of the model is also available.

We will be closely monitoring developments in large language models that utilize this technique, and there is no doubt that Mixture-8x7B will be added to our test suite for integration into business applications via the Wikit Semantics platform.