Meta’s latest open-source LLM for code beats GPT-4 in coding benchmark



summary
Summary

  • Code Llama 70B is available

Update from 29. January 2024:

Meta has just released the latest version of Code Llama. Code Llama 70B is a powerful open-source LLM for code generation. It is available in two variants, CodeLlama-70B-Python and CodeLlama-70B-Instruct. Meta says it is suitable for both research and commercial projects, and the usual Llama licenses apply.

The new 70B-instruct-version scored 67.8 on HumanEval, just ahead of GPT-4 and Gemini Pro for prompts without examples (zero-shot). The first version of Code Llama achieved up to 48.8 points.

According to Meta, Code Llama’s performance makes it an ideal basis for refining code generation models and should advance the open-source community as a whole. This seems likely since the 34B version of Code Llama has already been significantly improved by the open-source community and brought up to GPT-4 levels. Code Llama 70B could have even more room for improvement.

Ad

Ad

Code Llama 70B and other Llama models are available here after asking Meta. More information is available on Github.

Original article from August 24, 2024:

Code Llama is Meta’s refined Llama 2 variant for code generation.

According to Meta, Code Llama is an evolution of Llama 2 that has been further trained with 500 billion code tokens and code-related tokens from Llama 2’s code-specific datasets. To train Code Lama, Meta used more code data over a longer period of time.

Compared to Llama 2, Code Lama has enhanced programming capabilities and can, for example, generate appropriate code in response to a natural language prompt such as “Write me a function that outputs the Fibonacci sequence.” Similar to Github Copilot, it can also complete and debug code.

Recommendation

HumanEval and Mostly Basic Python Programming (MBPP) benchmarks, Code Llama 34B achieves results on par with GPT-3.5, but is far behind GPT-4 in Human Eval. Code Llama outperforms Llama 2, which is not optimized for code, and other open-source models tested.

Image: Meta AI

Meta releases Code Llama under the same Llama license as Llama 2 on Github. The application and the content it generates can be used for scientific and commercial purposes.

The Open Source Initiative criticized Meta for marketing the models as open source, saying the license restricts commercial use and certain areas of application and thus does not fully meet the open-source definition.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top