Bandwidth on FABRIC links

Bandwidth QoS provisioning of WAN links is still being developed.  Stay tuned.

• Basic NICs: The existing Basic NICs are implemented as SR-IOV virtual functions on a 100Gbps ConnectX-6.  The only limitation is that the bandwidth is shared with the other Basic NICs on that port.
• ConnectX-6/5s: The dedicated ConnectX-6s come with 2 100Gpbs ports while the ConnectX-5s have 2 25Gbps ports.  The dedicated ConnectX-6/5s are fully dedicate to a single VM and have full bandwidth to the switch.

Currently, there is little competition for bandwidth and you can see very nearly the full bandwidth in most cases (even with Basic NICs).  This is especially true for connections that stay within a site.

WAN links vary in performance. Eventually, they will nearly all be on 100+ Gbps L1 connections owned by FABRIC. However, they are currently being deployed as fast as we can.  In the mean time, many of the links are AL2S or other L2 service while we wait for the real links to be deployed. You will likely see lower bandwidth on these links.

Also, there are some quirks we are trying to work out where some SR-IOV NICs occasionally only get 25-30Gbps.  It seem like they are being left in a weird state by a previous experiment.  We are trying to figure out how to detect and reset these cases.

Generally, you should expect at lest 25Gbps and will often get close to 100Gbps.  Note that in order to get these speeds you will need a bit more memory and cores than the default.  Also, the app will need to be multi-threaded and many tools like iperf3 are single threaded even if you use ‘-P’ (https://fasterdata.es.net/performance-testing/network-troubleshooting-tools/iperf/multi-stream-iperf3/)

• This reply was modified 1 year, 8 months ago by Paul Ruth.

The Basic NICs/VFs are all best effort through the NIC itself with a cap of 100Gbps shared between all VFs on a physical host.  It is possible that each would max out at ~780Mbps but it is extremely unlikely.  In practice you will likely see 10s of Gbps.  Currently, your max bandwidth will likely be very close to 100Gbps.

If you want dedicated bandwidth, you will need to reserve dedicated NICs.   These NICs are dedicated to your experiment and are limited only by their hardware.  When QoS reservations are available they will be on the WAN links between the sites. Most of these links will be 100Gbps that can be divided and allocated to individual experiments.  The “super core” links will be 1200Gbps and can be divided and allocated as well.   One of the main uses of the “super core” will be to allocate many dedicated 100Gbps QoS links to individual experiments.

Yes, that is correct.

It might be useful to add that with the SRI-OV NICs the virtual switching in the host is done in hardware.  In most other cloud-ish systems the multi-tenant switching is done in software.  These software switches use a lot of CPU which causes performance interference between the computation and networking (even between experiments).  The SR-IOV solution should isolate the performance a bit more.

Also, it should be rare that there would be enough other active network-heavy experiments that you would see less than 10 or 20Gbps in the Basic NICs (I’m interesting testing this when we have more users).  I think you could get a lot of good experiments done by using tc to rate limit your own traffic to some modest amount. (10-20 Gbps). I suspect, a lot of the network contention would be from your own experiment.   Then, when you are ready, we could move you too a dedicated NIC an you could try higher bandwidths.