What Happens When a Token Travels Through GPU Villages via RDMA and NVLink?

First Stop: Travel Agency Processing 📋

Token, a piece of data living in house 0 of the GPU village, receives an invitation selected by eight experts. She visits the "Layout Dispatch Travel Agency" by invoking get_dispatch_layout and presents her invitation topk_idx. The agency prints three key files: num_tokens_per_rank – a table of token counts per village. num_tokens_per_rdma_rank – a cross‑nation token statistics sheet. is_token_in_rank – a "Where to go?" map.

Second Stop: Boarding the Dispatch Train 🚂

At the Dispatch station, the conductor calls notify_dispatch to alert all RDMA international express stations and NVLink local express stations. Token is placed on a special train composed of three carriages:

RDMA Sending Carriage : Workers (warp) pack Token’s data x, its identity SourceMeta, and top‑k information into a symmetric buffer SymBuffer.

RDMA→NVL Transfer Carriage : This hub extracts Token from the RDMA buffer, checks SourceMeta to determine the destination NVLink village, and uses the Tensor Memory Accelerator (TMA) conveyor belt to move her into the NVLink buffer.

NVL Receiving Carriage : The final stop where Token arrives at the GPU village; staff retrieve her from the NVLink buffer and translate her global expert ID into a local ID.

Third Stop: Expert Meeting 🎓

Token finally meets the selected experts. In the test, the experts are indifferent and simply return Token unchanged.

Fourth Stop: Return on the Combine Train 🔙

After the meeting, Token boards the Combine train for the return journey, which mirrors the outbound route:

NVL Sending Carriage : Gathers Token copies from all experts.

NVL→RDMA Transfer Carriage : Uses the magic combine_token to merge multiple Token copies into a single one.

RDMA Receiving Carriage : Delivers the merged Token back home and adds two bias gifts.

Travel Verification ✅

Upon arrival, testers check Token’s status. They find that after the trip Token has been split into is_token_in_rank.sum() copies, each copy shares the load evenly, while the original data remains intact.

Performance Tuning: Finding the Fastest Route 🏎️

The travel agency conducts an experiment varying the chunk size (tokens per dispatch) to locate the quickest path. Results show that the RDMA international express can achieve 43–58 GB/s, while the NVLink local express approaches its theoretical bandwidth limit.