Posts on Under the Model

RDMA Network Locality for Multi-Node Jobs with Kueue

Tue, 20 Oct 2026 00:00:00 +0000

The pods of a multi-node training job talk to each other all the time. So where they land on the RDMA network matters. OCI’s cluster network is a three-tier Clos. OCI’s write-up of the design gives these numbers for the three tiers of switches:

The first tier serves up to 256 GPUs, with up to 2 µs of latency.
The second tier serves up to 2,048 GPUs, with up to 5 µs.
The third tier serves up to 131,072 GPUs, with up to 8 µs.

The write-up calls the first tier Tier-0. The diagram below calls the other two Tier-1 and Tier-2.

Why Large RoCE Frames Disappeared on OVS-DOCA 3.5

Tue, 20 Oct 2026 00:00:00 +0000

I looked into GB300 hosts where RDMA broke after an image update. ib_write_bw between two hosts worked with 512-byte messages. With 64 KiB messages, it showed 0 Gb/s. The links were up on the host and on the switch. Hosts on the older image worked.

The cause was a small change in how OVS-DOCA reuses its receive buffer pools. On the new version, the pools kept the size for a 1500-byte MTU, although every port said 9216. RoCE frames of about 4.2 KB did not fit, and no counter called them dropped.

Why torchrun Can't Find Node Two on the Host Network

Tue, 20 Oct 2026 00:00:00 +0000

If you want RDMA inside a Kubernetes pod, the simplest way is the host network. The pod uses the network namespace of the node, NCCL sees the RDMA NICs directly, and you get the bandwidth you paid for. I tested on two BM.GPU4.8 nodes on OKE, with 8 A100 GPUs and 16 RDMA NICs each. NCCL all_reduce reached about 191 GB/s bus bandwidth with an 8 GiB buffer.

Then torchrun gets confused. With the host network, every pod has the host name of its node, and torchrun uses host names to make two decisions. On these nodes, both decisions went wrong. Each node resolves its own name to 127.0.1.1, and the other node cannot resolve it at all. Either no node starts the rendezvous store, or node two cannot find node one.