OpenAI’s Real Bet with Jalapeño: Not Just a Faster Chip

Jalapeño’s First Results Focus on Work‑Per‑Watt, Not Pure Speed

OpenAI released the initial measurements of its custom inference chip, Jalapeño, emphasizing that the key goal is "how much useful AI work can be done per watt under the same interactive experience" rather than simply achieving higher peak performance.

Benchmark Gains Across Three Public Models

Using the SemiAnalysis InferenceX benchmark, Jalapeño achieved 1.5–1.9× higher throughput‑per‑watt and 1.7–3.6× lower end‑to‑end latency on GPT‑OSS 120 B, DeepSeek R 1 670 B, and Kimi K 2.5 1 T. Specific figures include:

GPT‑OSS 120 B: mixed TPS/kW 85,448 vs 44,960 (≈1.9×), latency 1.03 s vs 1.80 s (≈1.7×), token rate 22,935 vs 427 tok/s/user (≈53.7×).

DeepSeek R 1 670 B: mixed TPS/kW 19,641 vs 11,781 (≈1.7×), latency 1.65 s vs 5.99 s (≈3.6×), token rate 12,258 vs 118 tok/s/user (≈104.3×).

Kimi K 2.5 1 T: mixed TPS/kW 18,195 vs 11,862 (≈1.5×), latency 1.56 s vs 5.31 s (≈3.4×), token rate 6,744 vs 120 tok/s/user (≈56.1×).

System‑Level Design Philosophy

The chip is built as part of a full inference system that treats the request chain as a single bottleneck. Three core principles guide the design:

Minimize state movement by keeping KV cache and model state close to compute resources.

Compose compute, memory, and network per stage instead of dedicating chips to a single phase (prefill or decode).

Make the network a first‑class architectural element, using a larger interconnect domain to keep a request on the same fabric and reduce cross‑chip data transfer.

OpenAI describes Jalapeño as a "balanced and fungible accelerator" that can adapt to varying workloads as models shift from pure Q&A to longer reasoning, tool use, and agent workflows.

Hardware Integration and Collaboration

The chip is not a standalone die; it is integrated into a board, rack, and interconnect system with contributions from Broadcom (chip‑to‑network) and Celestica (board‑level and rack integration). Video clips show the chip being seated, assembled, and tested, underscoring that inference acceleration requires full stack validation.

Why Not Talk Tokens‑Per‑Chip?

Prefill and decode phases consume different resources—compute versus memory bandwidth—so OpenAI measures performance per watt to reflect real user experience. For chat, the difference is a few milliseconds per token; for agents, the cumulative delays become significant, making the per‑watt metric more meaningful.

AI‑Assisted Chip Development

OpenAI reports that the entire design‑to‑tape‑out process took nine months, with AI (Codex and GPT‑Astra) used to explore implementation options and accelerate arithmetic circuit design. In a subsequent two‑month effort, AI helped retune three open‑weight models, achieving 1.5–1.8× speedup for the selected blocks.

Limitations of the Current Data

Four caveats are highlighted:

The benchmark uses an 8 k/1 k single‑round workload, which may not reflect multi‑round agent scenarios.

Comparisons across hardware generations (e.g., HBM 4 vs. older systems) require careful accounting of software configuration and speculative decoding.

Speculative decoding is not a universal multiplier; its benefits depend on model and request patterns.

Laboratory curves still need validation in mass production, mature software stacks, and broader model suites.

Author’s Assessment

The most significant shift is treating inference experience—latency and energy per user task—as a primary design input. Future scrutiny should focus on latency stability in long‑chain agent workloads, the speed of bringing new models to high performance, and how the saved watts translate into cost and user‑experience gains.

References

OpenAI – Jalapeño’s first results (official performance data)

OpenAI status video post

OpenAI + Broadcom unveil Jalapeño announcement

The full stack behind abundant intelligence

SemiAnalysis – OpenAI Jalapeño: Better Than Nvidia Blackwell

Hacker News discussion thread (as methodological context)