Mesh LLM: Layer-Wise Splitting Runs 235B Models on Heterogeneous Hardware

What Mesh LLM Does

Mesh LLM lets you connect multiple heterogeneous machines — such as a Mac mini, an old desktop with an RTX 4070, and a dusty laptop — into a single "grid." The combined VRAM and system memory may be enough for a large model, but no single machine can hold it alone. Instead of buying new GPUs or renting cloud instances, Mesh LLM splits the model by layers across the available devices and exposes a single OpenAI-compatible endpoint at http://localhost:9337/v1.

Each machine runs a mesh-llm process. When a request arrives, the grid decides automatically:

If the local machine can hold the entire model, it runs inference locally with zero cross-machine traffic.

If another node has the required model, the request is routed there.

If the model is too large for any single node, it is split into contiguous layer segments distributed across several machines.

Installation takes three commands:

curl -fsSL https://raw.githubusercontent.com/Mesh-LLM/mesh-llm/main/install.sh | bash
mesh-llm setup
mesh-llm serve --auto

serve --auto

selects the backend, downloads models as needed, joins the discovered public grid, starts the inference interface on port 9337, and launches the web console on port 3131. Any OpenAI client can then point its base_url to that address.

Two Independent Products in One Repository

The repository contains two distinct Cargo workspace members:

Skippy — the inference engine. Handles model downloads, native inference, staged execution, and the OpenAI API.

Mesh — the network layer. Manages node discovery, encrypted transport, request routing, plugins, and the web console.

They share a single lockfile. Skippy can run standalone ( just skippy), so the grid is an optional deployment mode, not a hard dependency.

The model ecosystem follows the llama.cpp/GGUF path. The project officially certifies 72 P0/P1 model families covering Qwen, Llama, Gemma, Mistral, DeepSeek, GLM, MiniMax, Phi, Granite, Hunyuan, EXAONE, Cohere, Falcon, and RWKV. Multimodal splitting is certified for Qwen2-VL, Qwen3-VL, HunyuanOCR, and DeepSeek-OCR.

Key Features

Grid startup modes: serve (public/private/publish/with invite token), client (client-only, no compute contribution).

Model splitting: --split with a layer-package repository.

Local-only mode: --local-model-only bypasses the grid, discovery, and console for debugging.

Multi-model arbitration: set "model":"mesh" to send the same prompt to all models in the grid and merge answers via a code-layer arbiter.

Coding agent integration: mesh-llm goose, opencode, claude, pi launch pre-configured clients.

Plugins: Flash-MoE (stream expert weights from SSD), DwarfStar (run DeepSeek-family models on Apple Silicon).

Operations: management API, web console, runtime install for native runtimes, setup --service to install as a system service.

Ports are fixed pairs: 9337 for inference, 3131 for management and web console ( --headless keeps only the management interface, suitable for servers).

Real-World Deployment: Running a 235B Model on Three Machines

Goal: run a 235B quantized model across three machines.

Install on all three:

curl -fsSL https://raw.githubusercontent.com/Mesh-LLM/mesh-llm/main/install.sh | bash mesh-llm setup

Start the coordinator on machine A with the layer package and splitting enabled:

mesh-llm serve \
  --model meshllm/Qwen3-235B-A22B-UD-Q4_K_XL-layers@REV \
  --split \
  --max-vram 5 \
  --bind-port 7842

Use an immutable @REV reference (commit hash) instead of a drifting main branch.

Join machines B and C using the token printed by A:

mesh-llm serve \
  --model meshllm/Qwen3-235B-A22B-UD-Q4_K_XL-layers@REV \
  --split \
  --max-vram 5 \
  --join TOKEN \
  --bind-port 7843

If a node has multiple NICs, add --bind-ip LAN-IP so the invite token advertises a reachable address.

Verify readiness :

curl -sS http://127.0.0.1:3131/api/status | jq '.runtime.stages'
curl -sS http://127.0.0.1:9337/v1/models | jq '.data[].id'

Call the model :

curl http://localhost:9337/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"meshllm/Qwen3-235B-A22B-UD-Q4_K_XL-layers","messages":[{"role":"user","content":"hello"}]}'

Point coding agents (Claude Code, OpenCode, etc.) to base_url: http://localhost:9337 or use the built-in launcher mesh-llm claude.

Layer-Wise Splitting vs. Tensor Parallelism: The Exo Comparison

The best-known similar project is Exo. Both aim to run one model across multiple machines, but they choose opposite splitting strategies.

Exo uses tensor parallelism: it splits individual layer weights across devices, requiring an all-reduce collective communication after every layer. This yields low latency when bandwidth is high (Exo bets on Thunderbolt 5 RDMA, claiming 99% inter-device latency reduction).

Mesh LLM uses layer-wise (pipeline) splitting: contiguous layer blocks are assigned to different machines. Each machine computes its block independently and passes the activation frame to the next. No collective communication occurs during inference. The trade-off: each machine must fit its assigned layers in VRAM, but network bandwidth requirements are drastically lower, enabling cross-city or cross-datacenter grids.

Two splitting approaches: top shows per-layer synchronization, bottom shows only layer-to-layer handoff

This design choice defines their niches: Exo suits a few Thunderbolt-connected Macs on a desk; Mesh LLM suits a pile of heterogeneous machines with ordinary networking.

Transport uses QUIC with end-to-end encryption; relays only forward encrypted packets and cannot inspect inference content.

Layer Packages: The Critical Prerequisite

Models don't come pre-split. Someone must take a GGUF file, cut it into segments, and publish a "layer-package" repository on Hugging Face containing a model-package.json manifest and the per-segment GGUF shards. Each node downloads only its assigned segment, not the entire multi-GB model.

Startup order matters: the coordinator plans the contiguous layer assignment, then starts stages from the last layer backward . Only after all stages report ready does the layer-0 router open to external requests, ensuring downstream stages are waiting when a request arrives.

A safety guard: splitting is allowed only for certified model architectures by default. Uncertified models require an explicit --allow-uncertified-split flag, and the logs/status API will mark them uncertified_override. This is far friendlier than "split anything and debug garbled output later."

Public vs. Private Grids and Access Control

--publish

broadcasts the grid on Nostr for discovery; omitting it keeps the grid unlisted. However, the documentation explicitly warns: unlisted does not equal access control. The invite token encodes endpoint identity and connection address — it is connection information, not an authorization credential.

To truly restrict membership, you must configure admission policies separately: owner allowlist, grid version requirements, release attestation requirements. All three must be set and verified; you cannot rely on "the address isn't public."

Private deployment is simpler: mesh-llm serve --model Qwen3-8B-Q4_K_M It prints a join token; another machine runs mesh-llm serve --join TOKEN. A client-only node uses mesh-llm client --join TOKEN. Tokens can be read from a file, and the file is re-read on every reconnect, so rotating a token only requires overwriting the file — no process restart needed.

Toward a Compute Marketplace

Since version 0.78, the codebase includes payment-related components: a wallet plugin, provider-side lifecycle events, and settlement events carrying credited amounts and fees. The project describes itself as enabling "private or public sharing of compute for agents and chat." The direction is clear: if compute can be discovered, admitted, and metered, the grid can extend beyond the LAN into a compute marketplace. Whether that path succeeds remains to be seen, but the code is already written that way.

Web Console

The built-in management UI shows:

Top-left: Node ID and role.

Center: six real-time metrics (node count, active model count, total grid VRAM, in-flight requests).

Below: live topology graph.

Right: model catalog with architecture, provider, serving node count, and VRAM consumption per model.

Web console: node stats, live topology, and model catalog

The screenshot comes from the README's official mockup (showing 31 nodes, 277.6 GB total VRAM, 5 active models) — not your local data. The console's value is troubleshooting: which nodes serve a model, whether a node has dropped, remaining VRAM — all in one screen.

Who Should Use It (and Who Should Wait)

Good fit: you have two or three heterogeneous machines (Mac + NVIDIA GPU + CPU-only all work), want to run a model that doesn't fit on any single one, and don't plan to buy new hardware. Backends cover Metal, CUDA, ROCm, Vulkan, and CPU, with pre-built releases for each combination.

Wait if: you only have one machine. For models that fit locally, Mesh LLM uses the local path and you gain no grid benefit while adding a management layer.

Known caveats:

Windows native path still has distribution and loading bugs. The official workaround requires a pre-release build, manual CUDA runtime install, and adding the DLL directory to PATH. WSL2 is the easier route.

System memory pressure during split inference is often higher than VRAM pressure. In one test, a 65-layer segment reserved ~589 GiB of host-side compute workspace. Estimating capacity by package size and VRAM alone will lead to OOM.

Cold-start long-context requests must use streaming. Non-streaming prefill at this scale can exceed 300 seconds, causing a 504 from the frontend while the backend continues computing.

Concurrency admission and suffix-cache reuse are marked by the maintainers as "verified gaps."

Mesh LLM's value is turning "multi-machine model stitching" from experimental scripts into an engineered path with certified models, layer packages, and admission policies. The exact models that work will keep evolving.

GitHub: github.com/Mesh-LLM/mesh-llm