https://oguzpastirmaci.com/tags/rdma/Recent content in Rdma on Under the ModelHugoen-USTue, 20 Oct 2026 00:00:00 +0000https://oguzpastirmaci.com/posts/why-torchrun-cant-find-node-two-on-the-host-network/Tue, 20 Oct 2026 00:00:00 +0000https://oguzpastirmaci.com/posts/why-torchrun-cant-find-node-two-on-the-host-network/<p>If you want RDMA inside a Kubernetes pod, the simplest way is the host network. The pod uses the network namespace of the node, NCCL sees the RDMA NICs directly, and you get the bandwidth you paid for. On two BM.GPU4.8 nodes on OKE (8 A100 GPUs and 16 RDMA NICs each), NCCL <code>all_reduce</code> reached about 191 GB/s bus bandwidth with an 8 GiB buffer.</p> <p>Then torchrun gets confused. With the host network, every pod has the host name of its node, and torchrun uses host names to make two decisions. On these nodes, both decisions went wrong, because each node resolves its own name to <code>127.0.1.1</code> and the other node cannot resolve it at all. Either no node starts the rendezvous store, or node two cannot find node one.</p>