What is the Maximum throughput achieved in Fabric Testbed?

This topic has 8 replies, 3 voices, and was last updated 1 year, 3 months ago by Ilya Baldin.

Viewing 9 posts - 1 through 9 (of 9 total)

Author

Posts
August 29, 2023 at 3:36 pm #5179
Nagmat Nazarov
Participant
Dear Community,

I am doing a basic experiment to check the throughput between 2 nodes.

For that, I created 2 nodes on DALL and UCSD and connected them with the L2STS Network service. (See attachment)

After configuring the network the maximum throughput I got was: 613 Mbits/sec using iperf.

My question is: What is the Maximum throughput that can be achieved using a Fabric Testbed between testbeds?

How can I get better results? (How can I get 100Gb/s)

Kind regards,

Nagmat
August 29, 2023 at 3:46 pm #5183
Paul Ruth
Keymaster
Nagmat,

Try looking the iPerf3 examples in the JupyterHub.

The basic version should get at least 30 Gbps. (https://github.com/fabric-testbed/jupyter-examples/blob/main/fabric_examples/complex_recipes/iPerf3/iperf3.ipynb)

If you want to get closer to 100Gbps you will need to look at the NUMA tuning examples (https://github.com/fabric-testbed/jupyter-examples/blob/main/fabric_examples/complex_recipes/iPerf3/iperf3_optimized.ipynb)

Note that a few sites are still connected with slower bandwidth links but they are all capped at at least 10 Gbps. 10 Gbps should be attainable with minimal tuning required.

Paul
August 29, 2023 at 4:31 pm #5184
Nagmat Nazarov
Participant
Paul Ruth wrote:

should get at least 30

I tried the example from the first link, but I got 817 Mbits/sec only. Not near 30 GB/s.

Below is the snapshot of the example :

`
3: ens7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
 link/ether 26:27:49:84:cd:8d brd ff:ff:ff:ff:ff:ff
 altname enp0s7
 inet 10.133.133.2/24 scope global ens7
 valid_lft forever preferred_lft forever
 inet6 fe80::aaa4:4e25:fba5:5251/64 scope link noprefixroute 
 valid_lft forever preferred_lft forever
[rocky@Node2 ~]$ iperf3 -s
———————————————————–
Server listening on 5201
———————————————————–
Accepted connection from 10.132.7.2, port 46800
[ 5] local 10.133.133.2 port 5201 connected to 10.132.7.2 port 46802
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 75.9 MBytes 636 Mbits/sec 
[ 5] 1.00-2.00 sec 98.9 MBytes 830 Mbits/sec 
[ 5] 2.00-3.00 sec 99.2 MBytes 832 Mbits/sec 
[ 5] 3.00-4.00 sec 99.1 MBytes 831 Mbits/sec 
[ 5] 4.00-5.00 sec 100 MBytes 841 Mbits/sec 
[ 5] 5.00-6.00 sec 101 MBytes 844 Mbits/sec 
[ 5] 6.00-7.00 sec 98.7 MBytes 828 Mbits/sec 
[ 5] 7.00-8.00 sec 98.8 MBytes 829 Mbits/sec 
[ 5] 8.00-9.00 sec 101 MBytes 844 Mbits/sec 
[ 5] 9.00-10.00 sec 101 MBytes 845 Mbits/sec 
[ 5] 10.00-10.04 sec 2.80 MBytes 637 Mbits/sec 
- – – – – – – – – – – – – – – – – – – – – – – – -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.04 sec 976 MBytes 815 Mbits/sec receiver
<code></code>`

What may I be doing wrong that I am not getting 30 GB/s?
August 29, 2023 at 5:37 pm #5185
Nagmat Nazarov
Participant
Between WASH and INDI, the maximum I got was 1.28 Gbits/s as shown on the attachment.
August 30, 2023 at 11:33 am #5188
Ilya Baldin
Participant
I am not sure why at this point but I just tried the same slice (WASH/INDI) and got 46Gbps.

I can’t explain it for the moment. Try a couple of other pairs – e.g. INDI/STAR, UCSD/DALL – you can just override the site selection in the notebook:
```
#[site1, site2] = fablib.get_random_sites(count=2)
[site1, site2] = ('STAR', 'INDI')
print(f"Sites: {site1}, {site2}")
```
Example from my run:
```
Connecting to host 10.133.3.2, port 5201
[ 5] local 10.140.5.2 port 37080 connected to 10.133.3.2 port 5201
[ 7] local 10.140.5.2 port 37088 connected to 10.133.3.2 port 5201
[ 9] local 10.140.5.2 port 37090 connected to 10.133.3.2 port 5201
[ 11] local 10.140.5.2 port 37102 connected to 10.133.3.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-10.01 sec 11.9 GBytes 10.2 Gbits/sec 3557 54.1 MBytes (omitted)
[ 7] 0.00-10.01 sec 12.6 GBytes 10.8 Gbits/sec 6110 45.3 MBytes (omitted)
[ 9] 0.00-10.01 sec 21.2 GBytes 18.2 Gbits/sec 18196 62.8 MBytes (omitted)
[ 11] 0.00-10.01 sec 17.1 GBytes 14.7 Gbits/sec 4092 65.9 MBytes (omitted)
[SUM] 0.00-10.01 sec 62.8 GBytes 53.9 Gbits/sec 31955 (omitted)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 0.00-10.01 sec 10.9 GBytes 9.36 Gbits/sec 35 45.1 MBytes 
[ 7] 0.00-10.01 sec 8.72 GBytes 7.49 Gbits/sec 80 39.2 MBytes 
[ 9] 0.00-10.01 sec 17.5 GBytes 15.0 Gbits/sec 140 71.4 MBytes 
[ 11] 0.00-10.01 sec 17.8 GBytes 15.3 Gbits/sec 36 71.4 MBytes 
[SUM] 0.00-10.01 sec 55.0 GBytes 47.2 Gbits/sec 291 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 10.01-20.01 sec 10.1 GBytes 8.68 Gbits/sec 293 55.2 MBytes 
[ 7] 10.01-20.01 sec 8.31 GBytes 7.14 Gbits/sec 41 83.8 MBytes 
[ 9] 10.01-20.01 sec 17.7 GBytes 15.2 Gbits/sec 57 107 MBytes 
[ 11] 10.01-20.01 sec 18.2 GBytes 15.6 Gbits/sec 226 83.3 MBytes 
[SUM] 10.01-20.01 sec 54.3 GBytes 46.6 Gbits/sec 617 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 20.01-30.00 sec 14.7 GBytes 12.6 Gbits/sec 1652 79.7 MBytes 
[ 7] 20.01-30.00 sec 7.91 GBytes 6.80 Gbits/sec 322 58.0 MBytes 
[ 9] 20.01-30.00 sec 17.7 GBytes 15.3 Gbits/sec 693 80.4 MBytes 
[ 11] 20.01-30.00 sec 13.3 GBytes 11.4 Gbits/sec 1073 60.4 MBytes 
[SUM] 20.01-30.00 sec 53.6 GBytes 46.1 Gbits/sec 3740 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-30.00 sec 35.7 GBytes 10.2 Gbits/sec 1980 sender
[ 5] 0.00-30.03 sec 36.0 GBytes 10.3 Gbits/sec receiver
[ 7] 0.00-30.00 sec 24.9 GBytes 7.14 Gbits/sec 443 sender
[ 7] 0.00-30.03 sec 25.1 GBytes 7.19 Gbits/sec receiver
[ 9] 0.00-30.00 sec 53.0 GBytes 15.2 Gbits/sec 890 sender
[ 9] 0.00-30.03 sec 52.7 GBytes 15.1 Gbits/sec receiver
[ 11] 0.00-30.00 sec 49.3 GBytes 14.1 Gbits/sec 1335 sender
[ 11] 0.00-30.03 sec 49.1 GBytes 14.1 Gbits/sec receiver
[SUM] 0.00-30.00 sec 163 GBytes 46.6 Gbits/sec 4648 sender
[SUM] 0.00-30.03 sec 163 GBytes 46.6 Gbits/sec receiver
```
August 30, 2023 at 12:13 pm #5193
Nagmat Nazarov
Participant
Ilya Baldin wrote:

I am not sure why at this point but I just tried the same slice (WASH/INDI) and got 46Gbps.

I can’t explain it for the moment. Try a couple of other pairs – e.g. INDI/STAR, UCSD/DALL – you can just override the site selection in the notebook:
```
#[site1, site2] = fablib.get_random_sites(count=2)
[site1, site2] = ('STAR', 'INDI')
print(f"Sites: {site1}, {site2}")
```
Example from my run:
```
Connecting to host 10.133.3.2, port 5201
[ 5] local 10.140.5.2 port 37080 connected to 10.133.3.2 port 5201
[ 7] local 10.140.5.2 port 37088 connected to 10.133.3.2 port 5201
[ 9] local 10.140.5.2 port 37090 connected to 10.133.3.2 port 5201
[ 11] local 10.140.5.2 port 37102 connected to 10.133.3.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-10.01 sec 11.9 GBytes 10.2 Gbits/sec 3557 54.1 MBytes (omitted)
[ 7] 0.00-10.01 sec 12.6 GBytes 10.8 Gbits/sec 6110 45.3 MBytes (omitted)
[ 9] 0.00-10.01 sec 21.2 GBytes 18.2 Gbits/sec 18196 62.8 MBytes (omitted)
[ 11] 0.00-10.01 sec 17.1 GBytes 14.7 Gbits/sec 4092 65.9 MBytes (omitted)
[SUM] 0.00-10.01 sec 62.8 GBytes 53.9 Gbits/sec 31955 (omitted)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 0.00-10.01 sec 10.9 GBytes 9.36 Gbits/sec 35 45.1 MBytes 
[ 7] 0.00-10.01 sec 8.72 GBytes 7.49 Gbits/sec 80 39.2 MBytes 
[ 9] 0.00-10.01 sec 17.5 GBytes 15.0 Gbits/sec 140 71.4 MBytes 
[ 11] 0.00-10.01 sec 17.8 GBytes 15.3 Gbits/sec 36 71.4 MBytes 
[SUM] 0.00-10.01 sec 55.0 GBytes 47.2 Gbits/sec 291 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 10.01-20.01 sec 10.1 GBytes 8.68 Gbits/sec 293 55.2 MBytes 
[ 7] 10.01-20.01 sec 8.31 GBytes 7.14 Gbits/sec 41 83.8 MBytes 
[ 9] 10.01-20.01 sec 17.7 GBytes 15.2 Gbits/sec 57 107 MBytes 
[ 11] 10.01-20.01 sec 18.2 GBytes 15.6 Gbits/sec 226 83.3 MBytes 
[SUM] 10.01-20.01 sec 54.3 GBytes 46.6 Gbits/sec 617 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 20.01-30.00 sec 14.7 GBytes 12.6 Gbits/sec 1652 79.7 MBytes 
[ 7] 20.01-30.00 sec 7.91 GBytes 6.80 Gbits/sec 322 58.0 MBytes 
[ 9] 20.01-30.00 sec 17.7 GBytes 15.3 Gbits/sec 693 80.4 MBytes 
[ 11] 20.01-30.00 sec 13.3 GBytes 11.4 Gbits/sec 1073 60.4 MBytes 
[SUM] 20.01-30.00 sec 53.6 GBytes 46.1 Gbits/sec 3740 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-30.00 sec 35.7 GBytes 10.2 Gbits/sec 1980 sender
[ 5] 0.00-30.03 sec 36.0 GBytes 10.3 Gbits/sec receiver
[ 7] 0.00-30.00 sec 24.9 GBytes 7.14 Gbits/sec 443 sender
[ 7] 0.00-30.03 sec 25.1 GBytes 7.19 Gbits/sec receiver
[ 9] 0.00-30.00 sec 53.0 GBytes 15.2 Gbits/sec 890 sender
[ 9] 0.00-30.03 sec 52.7 GBytes 15.1 Gbits/sec receiver
[ 11] 0.00-30.00 sec 49.3 GBytes 14.1 Gbits/sec 1335 sender
[ 11] 0.00-30.03 sec 49.1 GBytes 14.1 Gbits/sec receiver
[SUM] 0.00-30.00 sec 163 GBytes 46.6 Gbits/sec 4648 sender
[SUM] 0.00-30.03 sec 163 GBytes 46.6 Gbits/sec receiver
```
Thanks for the clarification.

I am getting errors on the run phase from Jupiter notebook. The error is :

“docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? See ‘docker run –help’. “, that’s why I am logging in to the nodes and running the iperf3 commands manually. Probably that may be causing the issue.

I chose SALT and WASH this time, but still getting :

“[ ID] Interval Transfer Bitrate

[ 5] 0.00-10.05 sec 617 MBytes 515 Mbits/sec receiver“

How can I resolve the issue?
August 30, 2023 at 1:50 pm #5194
Ilya Baldin
Participant
Please tell us which version of fablib you are using (it is displayed in the notebook as part of the first fablib.show_config() command and also which version of Jupyter Notebooks you are using.

Also please indicate if you are using our Jupyter Hub or trying to run from your laptop.

Your error indicates that the user you are logging in as is not in docker group and cannot start Docker containers. You can remedy this by doing

$ sudo usermod -G docker rocky

However this **should not be necessary** if you are running a recent version of everything.
August 30, 2023 at 2:28 pm #5198
Nagmat Nazarov
Participant
My Fablib version is Version 1.5.4 as shown in the attachment.

I am using Jupyter Hub.

After “sudo usermod -G docker rocky” and tuning the parameters for both hosts, I am getting30Gbits/s.

I am doing the same experiment by connecting to 2 nodes(scheme attached) with network service(direct connection), but I am getting 20Gbits/s only.

Is it the network service causing 10Gbits/s lost in the ethernet connection?

Kind regards,

Nagmat
August 30, 2023 at 2:53 pm #5201
Ilya Baldin
Participant
I cannot answer readily what the problem in your setup is.

Tuning network performance to 100Gbps and beyond depends on a large number of parameters. The notebooks Paul pointed out show some of them. Things like

– number of available cores and RAM

– affinity between vCPUs, physical CPUs and network cards

– type of network card (to get close to 100Gbps you need dedicated not shared NICs)

– MTU size

– Number of threads used by the data transfer app

– etc etc etc

Network service type will not affect the performance. The type of network card used will.
Author

Posts

Viewing 9 posts - 1 through 9 (of 9 total)

You must be logged in to reply to this topic.