Takes long time for complete the Fablib API

Hello, It may be due to the change in system.

FABRIC Production Infrastructure Instability this weekend 09/16-09/18

These calls can take longish time simetimes (the results of calls are cached, so the first caller gets a delay, but others do not for a while), however for the past couple of days we are seeing some connectivity issues between our Jupyter Hub hosted in Google and the rest of the testbed manifesting as various connection retries which can also cause additional delays. We are investigating the reasons for it. It appears to be specific to the Jupyter Hub environment.

In my experience, the FABRIC API has never been *fast*, but it’s definitely been notably slower recently in JupyterHub.

I, too, am getting retries doing various operations. Specifically, this seems to be caused by timeouts to both cm.fabric-testbed.net and orchestrator.fabric-testbed.net.

This has resulted in a notebook that reliably completed in maybe 17 minutes or so becoming one that unreliably completes in upwards of 30.

Manually making connections to orchestrator.fabric-testbed.net from the JupyterHub terminal confirms that sometimes these connections just hang (it’s hard to say exactly what’s going on since I can’t run tcpdump in this environment, but it’s certainly the case that the TCP connection isn’t getting established).

Manually making connections to orchestrator.fabric-testbed.net from elsewhere seems to reliably work just fine.

tracerouting to orchestrator.fabric-testbed.net from the JupyterHub terminal gets paths like this *most* of the time:

 4  ws-gw-to-hntvl-gw.ncren.net (128.109.9.22)  32.705 ms  32.858 ms  32.481 ms  32.101 ms  33.195 ms
 5  renci-to-ws-gw.ncren.net (128.109.70.174)  35.452 ms  35.676 ms  35.631 ms  35.553 ms  35.630 ms
 6  152.54.15.60 (152.54.15.60)  35.950 ms !X  35.669 ms !X  35.478 ms !X  35.570 ms !X  36.120 ms !X

(Line 6 is orchestrator.fabric-testbed.net)

However, there are sometimes timeouts starting at renci-to-ws-gw.ncren.net. Indeed, that line usually shows up fine or not all (all timeouts). 152.54.15.60/orchestrator shows occasional timeouts, which seem to be correlated with timeouts seen at renci-to-ws-gw.ncren.net.

When I ran the tests from elsewhere, the path didn’t go through either of these ncren routers. I didn’t see any unusual timeouts via traceroute or ping.

I don’t know what diagnosis y’all have done so far, but could this be as simple as packet loss between those two ncren routers? I can’t ping from JupyterHub and am sort of shooting in the dark, but I estimate there might be something like 3% or 4% loss there.

Thank you for your analysis. We are about where you are – there is either a route flapping or, perhaps, some kind of non-trivial packet loss specific to the path from JH to RENCI (we have been testing with just curl to various hosts at RENCI and the results are the same). We have notified the MCNC NOC as well as UNC ITS and are waiting to see what they say.

The problem does not appear to manifest itself from the worker nodes hosting JH, only from the Dockers inside so we are thinking perhaps a middle-box somewhere that is dropping some connections specific to JH originating IPs because they have a high rate of transactions to our infrastructure compared to the background of regular IPs. But this is just a theory. We will continue our investigation and we apologize for the inconvenience.

To me, it looks  to be back to its old self!  Thank you!