https://oguzpastirmaci.com/Recent content on Under the ModelHugoen-USTue, 20 Oct 2026 00:00:00 +0000https://oguzpastirmaci.com/posts/why-torchrun-cant-find-node-two-on-the-host-network/Tue, 20 Oct 2026 00:00:00 +0000https://oguzpastirmaci.com/posts/why-torchrun-cant-find-node-two-on-the-host-network/<p>If you want RDMA inside a Kubernetes pod, the simplest way is the host network. The pod uses the network namespace of the node, NCCL sees the RDMA NICs directly, and you get the bandwidth you paid for. On two BM.GPU4.8 nodes on OKE (8 A100 GPUs and 16 RDMA NICs each), NCCL <code>all_reduce</code> reached about 191 GB/s bus bandwidth with an 8 GiB buffer.</p> <p>Then torchrun gets confused. With the host network, every pod has the host name of its node. torchrun uses host names to make two decisions, and both go wrong. Either no node starts the rendezvous store, or node two cannot find node one.</p>