What Is Code Llama? Meta’s Open-Source LLM for Generating Code

Code Llama is Meta’s dedicated version of the Llama 2 large language model, fine‑tuned for generating software source code and discussing code.

As an extension of Llama 2 released in July, Meta announced yesterday a new AI programming tool called “Code Llama”.

Code Llama is a large language model (LLM) that generates code from text prompts and is the code‑specific version of Llama 2.

The model is built by further training on a code‑focused dataset, sampling more data from the same source for a longer period. It can produce code and natural‑language documentation from mixed code‑and‑text prompts, such as “write a function that outputs the Fibonacci sequence”. It also supports code completion and debugging, covering languages like Python, C++, Java, PHP, TypeScript, JavaScript, C#, and Bash.

Meta states that Code Llama has the potential to serve as a productivity and education tool, helping programmers write more powerful and well‑documented software.

According to Meta, Code Llama outperforms open‑source, code‑specific Llama models and Llama 2 on two benchmarks (HumanEval and MBPP), matching the performance of OpenAI’s ChatGPT.

Code Llama can handle up to 100,000 context tokens, enabling it to evaluate long programs. In contrast, OpenAI’s ChatGPT typically handles only about 4,000–8,000 tokens, though longer‑context models are available via OpenAI’s API.

Meta explains that the longer context not only allows generation of longer programs but also unlocks new use cases: users can feed the model more of their codebase to obtain more relevant suggestions, and developers can debug large codebases by providing the entire relevant code to the model.

Code Llama is free for research and commercial use. Three model sizes are released—7B, 13B, and 34B parameters—each trained on 500 B code tokens and code‑related data. The 7B and 13B base and instruction models are trained with fill‑in‑the‑middle (FIM) capability, allowing insertion of generated code into existing files and supporting out‑of‑the‑box code completion.

The three models meet different service‑latency requirements; the 7B model can run on a single GPU, while the 34B model delivers the best results and richer coding assistance. Meta also offers two variants: Code Llama‑Python, further fine‑tuned on 100 B Python tokens, and Code Llama‑Instruct, fine‑tuned to follow natural‑language instructions.

Meta’s community license permits commercial use of the models, subject to a limit of 700 million monthly active users.

Developers can access Code Llama from Meta AI’s website:

https://ai.meta.com/resources/models-and-libraries/llama-downloads