FABNetv4 Connectivity Stopped Working Between Storage VM and Multi-Sites

[Arash]

I have a Storage VM at MICH (10.131.3.2/24) that needs to communicate with FPGA nodes at five other sites (KANS: 10.138.2.2, SALT: 10.134.1.2, DALL: 10.133.131.2, GPN: 10.135.1.2, LOSA: 10.137.4.2) via FABNetv4. This setup was working perfectly two days ago, but now the Storage VM cannot reach any of the remote nodes (0% success rate, 100% packet loss to all sites). I’ve verified that the Storage VM has the correct route configured (10.128.0.0/10 via 10.131.3.1 dev eth1), the local gateway (10.131.3.1) is reachable, the interface is UP with the correct IP, and no firewall is blocking traffic.

Has there been a recent change to FABNetv4 cross-site routing policy, or is there a known outage affecting inter-site connectivity? Since this was working days ago with no configuration changes on my end, I’m wondering if this is a temporary infrastructure issue or if I need to reconfigure my setup to use L2 networks instead for multi-site connectivity. This is currently blocking our multi-site FPGA development workflow. Any guidance on troubleshooting or recommended architecture would be greatly appreciated. I can provide full diagnostic outputs, slice IDs, and routing tables if helpful.

[Tom Lehman]

There was some fiber maintenance last night, which may have caused this outage.  It looks like the FABNetv4 service to FABRIC MICH is working again.  Please test and verify.  Thanks, Tom

[Author]

Posts