Why pods on DOCA-OFED nodes read the host’s RoCE GIDs as zero, what the driver does, and what works instead.
Posts for: #Kubernetes
ARP, rp_filter and accept_ra for RDMA Pods on Kubernetes
Where the ARP, rp_filter and accept_ra settings of a multi-rail RDMA node live on Kubernetes, and who sets them.
Gang Scheduling and Topology-Aware Scheduling in Upstream Kubernetes
Which parts of Kueue’s topology-aware scheduling are moving into kube-scheduler, tested on Kubernetes v1.37.1 with the same topology and mostly the same experiments as my Kueue post.
RDMA Network Locality for Multi-Node Jobs with Kueue
How Kueue’s topology-aware scheduling keeps the pods of a multi-node job in one RDMA local block or network block, tested step by step on a cluster without Kueue.
Why torchrun Can’t Find Node Two on the Host Network
Why torchrun loses track of itself when Kubernetes pods use the host network for RDMA, two related PyTorch issues open since 2022, and what fixed it on two A100 nodes.