Slices stuck in closing; internal server error

[Kenneth Calvert]

1. I have a notebook that spreads 24 VMs  across 5  sites.  It tries to put nodes on sites with available resources.  But it keeps failing on WASH, which seems to have enough resources, but  then doesn’t when the slice  is submitted.  (Hussam mentioned a bug in the inter-site network setup – we are only using  unique local addresses.) But  then the slice persists in “closing” state for more than 24 hours.
2. When I asked the “Ask me  a question” bot  how to  delete the slices, it gave incorrect  answers (saying the slices would show up in my profile, for example). When I asked it for a notebook that  would delete the slice, it gave me code, but it did not work.
3. I changed WASH to  EDUKY, and then the slice got to StableOK, but failed in post boot config with an “internal server error” (OrchestratorHTTPError) in modify_slice().

What led to that was apparently an HTTP request timing out 3 times:

WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=3, read=2, redirect=None, status=3)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='orchestrator.fabric-testbed.net', port=443): Read timed out. (read timeout=30.0)")': /slices/modify/8458dcaf-3582-4786-9479-a4819757d752

[Komal Thareja]

1. WASH may have enough resources in aggregate, but individual hosts may lack the capacity needed for your requested VMs, resulting in the Insufficient Resources error.
2. Resources associated with a Closing slice are released, as they are for a Dead slice. Delayed responses to resource closure requests can sometimes leave a slice stuck in the Closing state.
3. This may have been a transient error. The slice looks healthy, and all requested resources are now Active.

Best,

Komal

[Komal Thareja]

1. WASH may have enough resources in aggregate, but individual hosts may lack the capacity needed for your requested VMs, resulting in the Insufficient Resources error.
2. Resources associated with a Closing slice are released, as they are for a Dead slice. Delayed responses to resource closure requests can sometimes leave a slice stuck in the Closing state.
3. This may have been a transient error. The slice looks healthy, and all requested resources are now Active.

Best,

Komal

[Kenneth Calvert]

Komal – If it was a transient error, it’s still going on. I just ran the notebook again with a different slice name, and got the same error.  It appears that all the nodes completed post-config OK, but execution stopped after the third HTTP error (apparently on a PUT).

Yesterday you said that the resources were there, and I was able to attach to the slice today and run part of the rest of the notebook before I ran out of time.   But it seems like there is a problem with the “Orchestrator” somewhere.

Ken

[Komal Thareja]

Looking at it now. May i please ask which JH container option are you running?

I mean which option did you select when you logged in to JH?

Best,

Komal

 

[Kenneth Calvert]

The default – 1.9 I think.

[Author]

Posts