Why torchrun loses track of itself when Kubernetes pods use the host network for RDMA, two related PyTorch issues open since 2022, and what fixed it on two A100 nodes.
Notes on GPUs, RDMA and Kubernetes.
Notes on GPUs, RDMA and Kubernetes.
Why torchrun loses track of itself when Kubernetes pods use the host network for RDMA, two related PyTorch issues open since 2022, and what fixed it on two A100 nodes.