NewBeeNLP

Apr 2, 2024 · Artificial Intelligence

Jamba: How AI21 Labs Merged Mamba and Transformer for 3× Faster 128k Contexts

Jamba, a hybrid Mamba‑Transformer model from AI21 Labs, combines state‑space and attention layers with Mixture‑of‑Experts to deliver up to three times the throughput of comparable 52‑billion‑parameter LLMs on 128k context windows while maintaining high output quality and low memory usage.