failed lease update – all units failed in priming

[Bruce Hartpence]

So, running a pretty straight-forward build but lately have been getting the above error. This one happens to be on MAX.

Going to try again on MASS but this has been happening quite a bit lately. I see others with a similar problem but have not seen a solution though I may have missed it while looking through the posts.

Any thoughts?

[Bruce Hartpence]

Quick update – tried MASS an got the same problem but looking a little deeper at the error I see this:

No more IP addresses available on network 0073521c-8e08-4a7d-9abf-b9ec59b0d749

 

[Mert Cevik]

Hi Bruce,

I cannot see the specific error on your message, but as I can see, MAX is loaded with VMs to consume all floating IP address space on it. On FABRIC Testbed, majority of the sites (nodes) are configured with IPv6 management (ssh to VMs) network. Some sites (nodes) have IPv4 connectivity for management. On the IPv4 sites, number of VMs that can be provisioned is limited with the available IPv4 addresses in the subnet. On IPv6 sites, we don’t have any such limitation for the number of VMs caused by the addresses. I will suggest using IPv6 sites and I will add a list below.

• Sites with IPv4 management network: MAX, TACC, MASS, UCSD, FIU, SRI, BRIST, TOKY (upcoming)
• Sites with IPv6 management network: STAR, MICH, UTAH, NCSA, WASH, DALL, SALT, GPN, CLEM, GATECH, LOSA, NEWY, KANS, ATLA, SEAT, PRIN, INDI, PSC, RUTG, CERN, AMST, HAWI

Note: Among the IPv4 sites, MAX, TACC, MASS, BRIST have smaller IPv4 subnets. (MAX, TACC, MASS are already loaded, BRIST has space to accommodate VMs). UCSD, FIU, SRI have fairly larger subnets.

[Mert Cevik]

I did not see your recent message with the error while typing my message, so that’s actually a problem caused by the available IPv4 addresses. I hope the information I posted can be useful. Also, we are working with MAX, MASS and TACC sites to switch to IPv6 management subnets. This will be possible in the next few weeks and we will announce.

[Bruce Hartpence]

Thanks – I’ll give that a try and get used to IPv6 man interfaces.

[Bruce Hartpence]

Quick update – tried WASH but it failed during provisioning for all nodes. Tried NEWY and I’m up and running.

[Mert Cevik]

Hi Bruce,

Provisioning on WASH is working well now. We need to investigate the issue. As a datapoint, can you let us know about the slice you are creating? If this is a Jupyter notebook, is it possible to share with us?

[Bruce Hartpence]

Sure – what is the best way to share the notebook with you?

[Mert Cevik]

Hi Bruce,

Sharing GitHub link can be fine, or one of the text sharing (paste) sites can be fine, or I created a (temporary) Google folder (link below) that you can just upload the file (then I will remove quickly).

https://drive.google.com/drive/folders/1IqIXaj-BMFQGDJg23Pnids3U_OgEDV8f?usp=sharing

 

[Anju Gopinath]

Hi,

Is it possible to get back dead slices? I forgot to renew them in time and their leases expired.

[yoursunny]

Mert Cevik wrote:

On the IPv4 sites, number of VMs that can be provisioned is limited with the available IPv4 addresses in the subnet.

Can “management IP address” show up as a resource on Fabric Portal – Resources page?
This would allow the experimenter to avoid this limitation.

[Mert Cevik]

We will prioritize the effort for switching to IPv6 management IP addresses from IPv4.

[Author]

Posts