Mistral AI Unleashes Mixtral 8x7B: A Breakthrough in Sparse Mixture of Experts Language Models

A New Era of LLM Innovation Begins

In the realm of artificial intelligence, language models have emerged as powerful tools, capable of generating human-like text, translating languages, answering questions, and performing various other tasks. Among these language models, sparse mixture of experts (SMoE) models have gained attention for their ability to achieve impressive performance while maintaining efficiency.

Mistral AI, a leading player in the field of open-source AI, has recently unveiled Mixtral 8x7B, a groundbreaking SMoE large language model (LLM) that sets new benchmarks in terms of performance and efficiency.

Mixtral 8x7B: A Game-Changer in the LLM Landscape

Key Features:

• Model Size and Efficiency: Mixtral 8x7B boasts a total of 46.7 billion parameters, making it a large-scale language model. However, it performs inference at the same speed and cost as models one-third its size. This remarkable efficiency is attributed to the SMoE architecture, which enables the model to leverage a subset of its experts for each inference, resulting in faster and more cost-effective operations.
• Context Length and Language Support: Mixtral 8x7B possesses a context length of 32k tokens, allowing it to retain and process a substantial amount of information. Additionally, it supports five languages: Spanish, French, Italian, German, and English, demonstrating its versatility in handling multilingual tasks.
• Fine-tuned Variant: Mixtral 8x7B Instruct: Mistral AI also introduced Mixtral 8x7B Instruct, a fine-tuned version of the base model specifically optimized for instruction-following tasks. This fine-tuning was achieved using direct preference optimization (DPO), a simpler and more effective method compared to reinforcement learning from human feedback (RLHF), which was used to train models like ChatGPT.
• Open-Source Availability: In line with Mistral AI’s commitment to open-source initiatives, the weights for both Mixtral 8x7B and Mixtral 8x7B Instruct are released under the Apache 2.0 license, making them freely accessible to the developer community. Moreover, the model has been integrated into the vLLM open-source project, further promoting collaboration and innovation in the field of LLMs.

Unveiling the Power of Mixture of Experts Models

Background and Significance:

• Historical Roots and Google’s Contribution:The concept of mixture of experts (MoE) models dates back to 1991, demonstrating its enduring relevance in the field of machine learning. Google’s application of MoE to Transformer-based LLMs in 2021 marked a significant advancement, showcasing the potential of this architecture for enhancing LLM performance.
• Recent Success Stories:In 2022, InfoQ highlighted two notable examples of MoE models achieving remarkable results. Google’s Image-Text MoE model, LIMoE, surpassed the performance of CLIP in image-text matching tasks. Meta’s NLB-200 MoE translation model demonstrated exceptional capabilities in translating between over 200 languages. These successes underscore the growing prominence and effectiveness of MoE models in various domains.

Delving into the Architecture and Training Details

Key Concepts:

• MoE Architecture Overview:MoE models replace the feed-forward layers of the Transformer block with a combination of a router and a set of expert layers. During inference, the router selects a subset of experts to activate, increasing efficiency while preserving model performance.
• Mixtral’s Expert Selection Mechanism:In the Mixtral model, the output for each Transformer block is computed by applying the softmax function to the top two experts. This selection strategy contributes to the model’s impressive performance and efficiency.
• Fine-tuning with Direct Preference Optimization:The fine-tuned version, Mixtral 8x7B Instruct, was trained using DPO, a simpler and more effective method compared to RLHF. DPO leverages a dataset of paired responses, where one is ranked higher than the other, without the need for creating a separate reward function for reinforcement learning.

Benchmarking and Evaluation Results

Performance Highlights:

• Outperforming Peers:Mixtral 8x7B demonstrated exceptional performance across several LLM benchmarks, outperforming Llama 2 70B on nine out of twelve benchmarks and GPT-3.5 on five benchmarks. These results solidify Mixtral 8x7B’s position as a top-performing LLM.
• Chatbot Benchmark Success:According to Mistral AI, Mixtral 8x7B Instruct achieved the highest score among open-weights models on the MT-Bench chatbot benchmark as of December 2023. This accolade further cements the model’s prowess in conversational tasks.
• LMSYS Leaderboard Ranking:The LMSYS leaderboard currently ranks Mixtral 8x7B as the 7th best LLM, surpassing notable models such as GPT-3.5, Claude 2.1, and Gemini Pro. This ranking is a testament to the model’s overall capabilities and competitiveness.

Accessibility and Availability

Embracing Open-Source and Hosted Versions:

• HuggingFace Integration:Mistral AI has made Mixtral 8x7B and Mixtral 8x7B Instruct readily available on HuggingFace, a popular platform for sharing and using machine learning models. This integration facilitates seamless access and utilization of these models by developers and researchers.
• Hosted API Endpoint:For those seeking a convenient and low-maintenance solution, Mistral AI offers a hosted version of the model through their mistral-small API endpoint. This option allows users to leverage the power of Mixtral 8x7B without the need for local setup and infrastructure management.

Conclusion: Embarking on a New Era of LLM Innovation

Mistral AI’s release of Mixtral 8x7B marks a significant milestone in the evolution of LLMs. The model’s impressive performance, efficiency, and open-source availability position it as a valuable resource for advancing research and innovation in the field of natural language processing. As the community continues to explore the full potential of LLMs, Mixtral 8x7B stands as a testament to the transformative power of open collaboration and the pursuit of cutting-edge AI technologies.