Meta plans to take on GPT-4 with a rumored Llama 3, which is still free



According to a first rumor, Llama 3 will be able to compete with GPT-4, but will still be freely available under the Llama license.

This was overheard by OpenAI engineer Jason Wei, formerly of Google Brain, at a Generative AI Group social event organized by Meta. Wei says he picked up on a conversation that Meta now has enough computing power to train Llama 3 and 4. Llama 3 is planned to reach the performance level of GPT-4, but will remain freely available.

Image: Jason Wei via Twitter

Even if Wei himself is a credible source, the statements he heard could be wrong, or the plans could still change. There is no official statement on if or when Llama 3 will be released.

Meta took about five months between the release of Llama 1 in late February 2023 and the release of Llama 2 in late July 2023.



GPT-4 has a more sophisticated architecture than your standard Llama

GPT-4 likely achieves its high performance by using a more complex mixture-of-experts architecture with 16 expert networks, each with about 111 billion parameters.

Jumping from Llama 2 to Llama 3 may therefore be more challenging than simply scaling through more training, and may take longer than moving from Llama 1 to Llama 2.

Llama 2 reaches the level of GPT-3.5 in some applications and is also being optimized by the open-source community through fine-tuning and additional features.

For example, the recently released Code Llama, which is based on Llama 2, achieves GPT-3.5 and GPT-4 level results (depending on the type of measurement) in the HumanEval coding benchmark through fine-tuning.

However, in the paper on Llama 2, Meta itself notes that there is still a large performance gap with closed-source models such as GPT-4 and Google’s PaLM-2.


Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top