1. Slices stuck in closing; internal server error

Slices stuck in closing; internal server error

Home › Forums › FABRIC General Questions and Discussion › Slices stuck in closing; internal server error

Viewing 6 posts - 1 through 6 (of 6 total)
  • Author
    Posts
  • #10095
    Kenneth Calvert
    Participant
      1. I have a notebook that spreads 24 VMs  across 5  sites.  It tries to put nodes on sites with available resources.  But it keeps failing on WASH, which seems to have enough resources, but  then doesn’t when the slice  is submitted.  (Hussam mentioned a bug in the inter-site network setup – we are only using  unique local addresses.) But  then the slice persists in “closing” state for more than 24 hours.
      2. When I asked the “Ask me  a question” bot  how to  delete the slices, it gave incorrect  answers (saying the slices would show up in my profile, for example). When I asked it for a notebook that  would delete the slice, it gave me code, but it did not work.
      3. I changed WASH to  EDUKY, and then the slice got to StableOK, but failed in post boot config with an “internal server error” (OrchestratorHTTPError) in modify_slice().

      What led to that was apparently an HTTP request timing out 3 times:

      WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=3, read=2, redirect=None, status=3)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='orchestrator.fabric-testbed.net', port=443): Read timed out. (read timeout=30.0)")': /slices/modify/8458dcaf-3582-4786-9479-a4819757d752
      #10096
      Komal Thareja
      Moderator
        1. WASH may have enough resources in aggregate, but individual hosts may lack the capacity needed for your requested VMs, resulting in the Insufficient Resources error.
        2. Resources associated with a Closing slice are released, as they are for a Dead slice. Delayed responses to resource closure requests can sometimes leave a slice stuck in the Closing state.
        3. This may have been a transient error. The slice looks healthy, and all requested resources are now Active.

        Best,

        Komal

        #10097
        Komal Thareja
        Moderator
          1. WASH may have enough resources in aggregate, but individual hosts may lack the capacity needed for your requested VMs, resulting in the Insufficient Resources error.
          2. Resources associated with a Closing slice are released, as they are for a Dead slice. Delayed responses to resource closure requests can sometimes leave a slice stuck in the Closing state.
          3. This may have been a transient error. The slice looks healthy, and all requested resources are now Active.

          Best,

          Komal

          #10099
          Kenneth Calvert
          Participant

            Komal – If it was a transient error, it’s still going on. I just ran the notebook again with a different slice name, and got the same error.  It appears that all the nodes completed post-config OK, but execution stopped after the third HTTP error (apparently on a PUT).

            Yesterday you said that the resources were there, and I was able to attach to the slice today and run part of the rest of the notebook before I ran out of time.   But it seems like there is a problem with the “Orchestrator” somewhere.

            Ken

            
            
            				
            #10100
            Komal Thareja
            Moderator

              Looking at it now. May i please ask which JH container option are you running?

              I mean which option did you select when you logged in to JH?

              Best,

              Komal

               

              #10101
              Kenneth Calvert
              Participant

                The default – 1.9 I think.

              Viewing 6 posts - 1 through 6 (of 6 total)
              • You must be logged in to reply to this topic.