https://oguzpastirmaci.com/tags/ipv6/Recent content in Ipv6 on Under the ModelHugoen-USSun, 01 Nov 2026 00:00:00 +0000https://oguzpastirmaci.com/posts/arp-rp-filter-and-accept-ra-for-rdma-pods-on-kubernetes/Sun, 01 Nov 2026 00:00:00 +0000https://oguzpastirmaci.com/posts/arp-rp-filter-and-accept-ra-for-rdma-pods-on-kubernetes/<p>A GPU node with 16 RoCE rails has 16 network interfaces, often in one subnet. Linux treats an IP address as owned by the host, not by an interface. That default is fine for one NIC. With 16 rails, it breaks RDMA in ways that look like RDMA bugs.</p> <p>These are some of the failures I saw:</p> <ul> <li><code>ib_write_bw</code> failed on 6 of 16 rails, and the failing rails changed between runs.</li> <li>Two peer addresses resolved to the same MAC address. One of them got the MAC of a VF on another rail.</li> <li>NCCL stopped with <code>ibv_modify_qp failed with 61 No data available</code>, because most rails had no IPv6 address.</li> </ul> <p>The fix is always a few sysctls. The hard part on Kubernetes is where they live. A pod can get its RDMA interface in several ways. Each way decides which network namespace the interface ends up in, and who owns its settings.</p>