Forum Replies Created
-
Posts
-
I am not sure why at this point but I just tried the same slice (WASH/INDI) and got 46Gbps.
I can’t explain it for the moment. Try a couple of other pairs – e.g. INDI/STAR, UCSD/DALL – you can just override the site selection in the notebook:
#[site1, site2] = fablib.get_random_sites(count=2) [site1, site2] = ('STAR', 'INDI') print(f"Sites: {site1}, {site2}")Example from my run:
Connecting to host 10.133.3.2, port 5201 [ 5] local 10.140.5.2 port 37080 connected to 10.133.3.2 port 5201 [ 7] local 10.140.5.2 port 37088 connected to 10.133.3.2 port 5201 [ 9] local 10.140.5.2 port 37090 connected to 10.133.3.2 port 5201 [ 11] local 10.140.5.2 port 37102 connected to 10.133.3.2 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-10.01 sec 11.9 GBytes 10.2 Gbits/sec 3557 54.1 MBytes (omitted) [ 7] 0.00-10.01 sec 12.6 GBytes 10.8 Gbits/sec 6110 45.3 MBytes (omitted) [ 9] 0.00-10.01 sec 21.2 GBytes 18.2 Gbits/sec 18196 62.8 MBytes (omitted) [ 11] 0.00-10.01 sec 17.1 GBytes 14.7 Gbits/sec 4092 65.9 MBytes (omitted) [SUM] 0.00-10.01 sec 62.8 GBytes 53.9 Gbits/sec 31955 (omitted) - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 0.00-10.01 sec 10.9 GBytes 9.36 Gbits/sec 35 45.1 MBytes [ 7] 0.00-10.01 sec 8.72 GBytes 7.49 Gbits/sec 80 39.2 MBytes [ 9] 0.00-10.01 sec 17.5 GBytes 15.0 Gbits/sec 140 71.4 MBytes [ 11] 0.00-10.01 sec 17.8 GBytes 15.3 Gbits/sec 36 71.4 MBytes [SUM] 0.00-10.01 sec 55.0 GBytes 47.2 Gbits/sec 291 - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 10.01-20.01 sec 10.1 GBytes 8.68 Gbits/sec 293 55.2 MBytes [ 7] 10.01-20.01 sec 8.31 GBytes 7.14 Gbits/sec 41 83.8 MBytes [ 9] 10.01-20.01 sec 17.7 GBytes 15.2 Gbits/sec 57 107 MBytes [ 11] 10.01-20.01 sec 18.2 GBytes 15.6 Gbits/sec 226 83.3 MBytes [SUM] 10.01-20.01 sec 54.3 GBytes 46.6 Gbits/sec 617 - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 20.01-30.00 sec 14.7 GBytes 12.6 Gbits/sec 1652 79.7 MBytes [ 7] 20.01-30.00 sec 7.91 GBytes 6.80 Gbits/sec 322 58.0 MBytes [ 9] 20.01-30.00 sec 17.7 GBytes 15.3 Gbits/sec 693 80.4 MBytes [ 11] 20.01-30.00 sec 13.3 GBytes 11.4 Gbits/sec 1073 60.4 MBytes [SUM] 20.01-30.00 sec 53.6 GBytes 46.1 Gbits/sec 3740 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-30.00 sec 35.7 GBytes 10.2 Gbits/sec 1980 sender [ 5] 0.00-30.03 sec 36.0 GBytes 10.3 Gbits/sec receiver [ 7] 0.00-30.00 sec 24.9 GBytes 7.14 Gbits/sec 443 sender [ 7] 0.00-30.03 sec 25.1 GBytes 7.19 Gbits/sec receiver [ 9] 0.00-30.00 sec 53.0 GBytes 15.2 Gbits/sec 890 sender [ 9] 0.00-30.03 sec 52.7 GBytes 15.1 Gbits/sec receiver [ 11] 0.00-30.00 sec 49.3 GBytes 14.1 Gbits/sec 1335 sender [ 11] 0.00-30.03 sec 49.1 GBytes 14.1 Gbits/sec receiver [SUM] 0.00-30.00 sec 163 GBytes 46.6 Gbits/sec 4648 sender [SUM] 0.00-30.03 sec 163 GBytes 46.6 Gbits/sec receiver
What you observed was part of the consequences of the outage. We believe most of them have been by now cleaned up (including UCSD and DALL) however the banner remains on the portal because we are still testing.
MICH continues to be unreachable for us due to UMich campus network outage of unknown anticipated duration. We’ve placed the site into maintenance and will update on the FABRIC Announcements forums when it becomes available again.
We will look into it – it is likely because the refresh token is expired by then. Generally restarting the container should cure it, but probably because by then your portal cookie has expired you cannot get a new token.
One way to deal with it is to go to the Credential Manager (from the Portal -> Experiments -> Manage Tokens -> Open Credential Manager). Generate a new token (tuple) and save it into your JH under /home/fabric/.tokens.json
Fraida,
This is working ‘as designed’. Your regular token consists of two parts
– a short-lived API token we use to authorize user actions (4 hours)
– a longer-lived ‘refresh’ token which allows you to refresh the API token (24 hours)
Any time you refresh the token, you actually get back a tuple of new <API token, refresh token> the latter being good for 24 hours again. This is part of the security posture that allows to minimize damage in case of an API token leakage.
So in general if you run a fablib operation inside Jupyter Hub within a 24 hour period, your token tuple will be auto-refreshed so long as you are logged in.
We are in the process of developing a long-lived token feature reserved for those who want to run unattended experiments. It will be by special permission.
Given it is only a network outage, VM slivers should still be there assuming they do not timeout in the interim.
Just to bring this back up – we are working with NVidia/Mellanox engineering support on this. Their engineers are able to reproduce the problem (which is good news). They are trying to figure out the difference between a working setup and a non-working setup.
If you can remember where you got the code snippet you were trying to use @Nishanth please point it to us – it is likely from an older version of the API and we should correct it.
Without Slice.NoLimitLifetime you can only renew in increments of two weeks (14 days). See this article: https://learn.fabric-testbed.net/knowledge-base/fabric-user-roles-and-project-permissions/ for more on project permission structure.
Please let us know:
1. Which container you are using if you are using Jupyter Hub (there are 4 to choose from).
2. What is the version of fablib you are using:
pip freeze | grep fabrictestbedFuture versions of this notebook will be more explicit about what
hostparameter is for and how to use it.
