[Paul Ruth]

Update:

I tried this myself and was able to get ~6Gbps but only after tuning as suggested by the ESnet site.  I did this with VMs at UKY and LBNL. Both VMs were bigger than the default (32 cores, 64G ram… this is probably bigger than necessary).

I also found that jumbo frames is not yet possible between these sites. We are working on making this possible soon.

[Paul Ruth]

Generally, eth0 is used as a management interface. This is the network that you use when you ssh to the node from the Internet.  You should avoid using this network for experiments.

The interfaces numbered eth1 (or higher) will be the ones associated with the network component(s) that you have added to your node. These are the ones you should use for experiments.

Re: the slow performance of the experimental network.  Our initial deployment does not yet use the dedicated L1 circuits that we will have as they become available. Instead it uses I2 AL2S.  However, even with AL2S you should be able to get much higher bandwidth.  I would expect you could get over 10Gbps (maybe even as much at 100Gbps).  There are a couple of possible issues:

1. Our network deployment needs to be configured/tuned correctly. This is such low bandwidth that I suspect something in the path is dropping packets.  What slice configuration did you use? I assume you have one node at UKY and one at LBNL, is this true? Also, which components did you include on the nodes?
2. Your end hosts need to be tuned for high-latency, high-bandwidth data transfers.  From the ifconfig info I can see that your nodes are not using jumbo frames.  There are probably some other tuning optimization you can make. ESnet has a great resource for learning about this:  https://fasterdata.es.net/host-tuning/linux/ 

Please let us know which components you are using in the VMs. I would like to try this myself and see if I have the same issues.

Paul

 

 

[Paul Ruth]

Can you post post the current version of the notebook ipynb file?

[Author]

Posts