Meta’s Llama 3.1 405B: How the Open‑Source Giant Stands Up to GPT‑4 and Claude 3.5

Meta announced the release of Llama 3.1, the latest open‑source large language model, positioning it against GPT‑4o and Claude 3.5 Sonnet.

According to Meta’s blog, Llama 3.1 405B is the first publicly available LLM that matches top‑tier models in common‑sense reasoning, manipulability, mathematics, tool use, and multilingual translation. The company claims the model enables unprecedented growth, synthetic‑data generation, and large‑scale model distillation.

The 405B version outperforms or matches GPT‑4, GPT‑4o, and Claude 3.5 Sonnet on many benchmarks, especially in mathematics, reasoning, and coding.

Key Features and Capabilities

Llama 3.1 405B can perform tasks such as code generation, solving math problems, and summarizing documents in eight languages. It is currently text‑only; multimodal versions that handle images, video, and speech are under development but not yet released.

Training Data and Scale

The model was trained on more than 150 trillion tokens (approximately 7.5 trillion words) using an optimized training stack and over 16 k H100 GPUs. Meta improved data management and quality‑assurance processes and also used synthetic data from other AI models for fine‑tuning, though the exact data sources remain undisclosed for competitive and legal reasons.

Context Window and Tools

Llama 3.1 405B features a 128,000‑token context window, allowing it to summarize longer texts and maintain context better than previous versions. Meta also released two smaller models, Llama 3.1 8B and Llama 3.1 70B, which share the same window and can be paired with third‑party tools and APIs for recent‑event Q&A, math solving, and code verification.

Performance and Licensing

Performance is comparable to OpenAI’s GPT‑4, with strong results in code generation and chart creation but weaker multilingual abilities and general reasoning. The model’s size demands substantial hardware resources. Meta updated the Llama license to permit developers to use model outputs for third‑party AI models, though developers with applications exceeding 700 million monthly users must obtain a special license.

The 8B and 70B variants are also available for download from Meta’s website or Hugging Face.

References

Meta Llama Official Site

Llama 3.1 on Hugging Face