How a 128 GB Mac Loaded 1.6 TB of Kimi K3 Weights and Ran Inference

Kimi K3, a 2.8‑trillion‑parameter LLM with official MXFP4 weights (~1.56 TB), is typically thought to require massive GPU memory (1.5 TB) and multiple H100 GPUs. A developer tested whether a personal device could run it.

Using a Mac with 128 GB unified memory (M5 Max), the developer streamed the weights: about 97 GiB of static weights stayed resident, while the remaining layers were loaded on‑demand from an SSD during inference. This approach avoids loading the full model into RAM.

The Mac could answer simple prompts (e.g., "How are you?" → "I am fine, thank you") and explain attention mechanisms, but the inference speed was very slow—approximately 0.98 token/s for input processing and 0.32 token/s for generation, meaning roughly three seconds per token.

While the streaming method solves the memory‑capacity issue, the constant disk‑to‑RAM transfers limit throughput. The developer plans to further compress the model to Q2 and test on two Mac Studio machines with 512 GB unified memory to improve speed.

Separately, the Ning team built a cluster of ten nodes, each with eight RTX 5090 GPUs (total 80 GPUs), providing 2.56 TB of VRAM and ~143 TB/s aggregated bandwidth. Using the same MXFP4 weights without quantization, they achieved 20 token/s on the first day of testing—over sixty times faster than the Mac setup.

Although the RTX 5090 cards are consumer‑grade (GDDR7, no HBM or NVLink) and the nodes communicate via 25 GbE Ethernet, the cluster’s performance demonstrates that large‑scale LLM inference is possible without high‑end data‑center GPUs, albeit with significant investment in hardware, power, and cooling.

The authors note that bandwidth alone does not dictate inference speed; multi‑node communication overhead, kernel efficiency, and scheduling also affect results. Nonetheless, the 20 token/s figure shows room for further optimization, as earlier 4090 clusters reached 110 token/s on GLM‑5.2.