get_physical_os_interface()[‘ifname’] failed

We are still working on tuning all the links and trying to figure out best practices for achieving very high bandwidths. There are no artificial limitations on that link and, in theory, 100G is possible. This is just going to require a bunch of tuning, both on the edge and in probably in the core.

I know some of our students where looking at this and achieved ~100G between pairs of sites that are closer to each other. I’m not sure what the current best bandwidth achieved is on the longer spans, but I remember seeing them getting at least 30G for some tests. We would be interested in knowing about any successes you have with achieving higher bandwidths.

What tuning did you perform in your nodes?

In general, there are a lot of variable that can prevent bandwidths at these rates. You might reduce some of those variable by starting with a pair of sites that a close to each other (maybe UTAH/SALT) and use dedicated connectX-6 cards.

Your UDP test has a 98% loss. Given that the card is a 100G card it can easily overwhelm an intermediary switch which can result in huge packet losses like that. You might try UDP test with lower bandwidths and slowly increase the bandwidth until you packet loss starts to grow. Then try different tuning parameters to see if you can get it higher.

I’m going to see if one of our student who was working on this can add an more here…

Those are all great resources.

mtu = 9000 (jumbo frames) is important too.

With 100G NICs this part from fasterdata.es.net is important too:

We also strongly recommend reducing the maximum flow rate to avoid bursts of packets that could overflow switch and receive host buffers.

For example for a 10G host, add this to a boot script:

/sbin/tc qdisc add dev ethN root fq maxrate 8gbit
For for a host running data transfer tools that use 4 parallel streams, do this:

/sbin/tc qdisc add dev ethN root fq maxrate 2gbit
Where 'ethN' is the name of the ethernet device on your system.

Hello.

I was able to get 70Gbps on a same-site experiment (first notebook attached), and 19Gbps on different-site experiments (second and third notebooks attached). The fourth notebook just has a simple UI bit of code that might be helpful for you to modify the parameters and plot the results on the graph.

Below are the tuning parameters that I use.

net.core.rmem_max = 2147483647
net.core.wmem_max = 2147483647
net.ipv4.tcp_rmem = 4096 87380 2147483647
net.ipv4.tcp_wmem = 4096 65536 2147483647
net.ipv4.tcp_congestion_control = htcp
net.ipv4.tcp_mtu_probing = 1
net.core.default_qdisc = fq

I use 32 parallel streams, and the largest window size I can get, like so: -P 32 -w 999M. I use iperf, not iperf3. I tried to get the 70Gbps result on iperf3 before, but no luck.