Why torchrun loses track of itself when Kubernetes pods use the host network for RDMA, the PyTorch issues that have been open since 2022, and what fixed it on two A100 nodes.
It worked on one node.
It worked on one node.
Why torchrun loses track of itself when Kubernetes pods use the host network for RDMA, the PyTorch issues that have been open since 2022, and what fixed it on two A100 nodes.