[Komal Thareja]

Hi Nagmat,

You have 4 active slices:

Slice Name: mri_ded3 Slice ID: d54fd501-ca14-49ce-b217-50c593bd0927 Project ID: 527832fc-c273-4254-b988-16e5c2923bf9 Project Name: in-network caching

Slice Name: mri1 Slice ID: 6ce8648d-26a9-47b8-8349-4e3e724085a0 Project ID: 527832fc-c273-4254-b988-16e5c2923bf9 Project Name: in-network caching

Slice Name: mri3 Slice ID: fa527279-b9bb-4dac-bb68-7a46b63c8ad1 Project ID: 527832fc-c273-4254-b988-16e5c2923bf9 Project Name: in-network caching

Slice Name: Nagm_P4Test01 Slice ID: 140247fe-cb45-47da-a797-dca5592487dd Project ID: 527832fc-c273-4254-b988-16e5c2923bf9 Project Name: in-network caching

The last slice Nagm_P4Test01 has two VM slivers on FIU and the VM slivers are in ActiveTicketed state – which means a pending Renew. We have been observing network issues with FIU and working to resolve that since yesterday. FIU was moved to maintenance as well. I do see management IPs for these VMs but it is possible that they are not accessible because of the network issue.

All other slices have all the VMs and Networks in Active state. If you are observing failure with these slices, please do a slice = fablib.get_slice(slice_name) and then try to upload the file.

Thanks,
Komal

• This reply was modified 1 year, 10 months ago by Komal Thareja.

[Komal Thareja]

Thank you Paul!

@Nagmat,
Could you please share the slice id for your slice?

Thanks,
Komal

[Komal Thareja]

Hi Nagmat,

I looked at your slice and the error reported above, your slice is requesting more than one VM on FIU and they are being allocated to fiu-w3.fabric-testbed.net. All the VMs are of the flavor: fabric.c8.m16.d500. Current disk availability on fiu-w3.fabric-testbed.net can only accomodate one such VM so all others fail.

We do have known discrepancy in the disk availability between the software and the infrastructure. We have plans to address that soon. Could you please use a different site instead of FIU for now?

Appreciate your feedback!

Thanks,
Komal

[Komal Thareja]

Maintenance is complete.

[Komal Thareja]

Thank you for sharing this! I would clean this up from the backend and would work on fixing this bug. Appreciate your feedback.

[Komal Thareja]

Maintenance has been lifted, testbed is open for use.

[Komal Thareja]

Maintenance is complete. Testbed has been opened for use.

Thanks,
Komal

[Komal Thareja]

Dear Experimenters,

The testbed is in a brief maintenance. We are running some performance test and will inform as soon as the testbed is back online.

Thanks,
Komal

[Komal Thareja]

I can confirm it’s the same issue on MASS and UCSD as well based on the logs. Also, I noticed the slice request was for 1000G disk but it gets mapped to a flavor with 2000G as we don’t have any 1000G flavor. None of the sites/workers mentioned have that much disk available and hence the provisioning fails. Unfortunately, the error message returned by openstack is not specific and not very helpful.

Thanks again for letting us know. We will work on fixing the mismatch and address this.

Thanks,
Komal

• This reply was modified 2 years ago by Komal Thareja.
• This reply was modified 2 years ago by Komal Thareja.

[Komal Thareja]

Hi Nagmat,

Thank you for sharing your observation! The requested slice asks for a VM with 2000GB disk on FIU rack (fiu-w4.fabric-testbed.net). However, the disk isn’t available and hence VM provisioning fails. We have a known issue that there is a mismatch in the disk availability as seen by software and the actual infrastructure. We are working to address that. But for now, I would request to create a slice with smaller disk.

Also, was this issue only observed on FIU rack? If not, Could you please share the other sites where you observed this error?

Thanks,
Komal

• This reply was modified 2 years ago by Komal Thareja.

1 user thanked author for this post.

Nagmat Nazarov

[Komal Thareja]

Hello Fengping,

Thanks you for sharing your observation. I looked at your slice. The Network Service in your slice is in Closed state as NSO failed to allow 100 external IP addresses. Enclosing the snapshot of the error below.

Also, I confirmed with the Network Team and found that we have an upper limit of 90 public IPs on a FabNetV6Ext. Their recommendation is to create an additional slice if you need more than 90 public IPs. Hope this helps!

err=failed lease update- all units failed priming: Exception during modify for unit: 4a338812-470d-41fc-a510-145dfd775816 Playbook has failed tasks: NSO exists invalid params. path = /ncs:services/l3rt:l3rt{NET3-4a338812-470d-41fc-a510-145dfd775816}/external-access/permit-ipv6{100}

Thanks,
Komal

[Komal Thareja]

Thank you for sharing this and helping us make the testbed better. It was a bug. A fix has been deployed. Could you please delete the slice which has FABNetv6Ext and recreate it again?

Thanks,
Komal

[Komal Thareja]

Testbed is open for use. GATech, FIU and GPN sites are still in maintenance. We would update once GATech, FIU and GPN are open.

[Komal Thareja]

Maintenance is completed. Network AM has been updated.

[Komal Thareja]

Hi Praveen,

This error is unrelated to the number of user’s dead/closed slices. The slice requested by Manas, was requesting VMs on UCSD rack and specific workers: ucsd-w7.fabric-testbed.net, ucsd-w8.fabric-testbed.net, ucsd-w9.fabric-testbed.net.

UCSD rack has only 6 workers, so the slice could not be allocated and the error “Insufficient Resources” was returned.

Resource Type: VM Notices: Reservation 503f8da4-dbfa-49bf-8efc-12dc02253194 (Slice cluster_gatk(9f998c48-6f4f-4af7-be33-337e94f7afb7) Graph Id:116fe360-7f45-4384-9a15-2aec138c4519 Owner:mjdbz4@health.missouri.edu) is in state (Closed,None_)  (Last ticket update: Insufficient Resources: ucsd-w7.fabric-testbed.net cannot serve the requested sliver)

Resource Type: VM Notices: Reservation beebbecb-660a-4011-bae3-7457be9955f2 (Slice cluster_gatk(9f998c48-6f4f-4af7-be33-337e94f7afb7) Graph Id:116fe360-7f45-4384-9a15-2aec138c4519 Owner:mjdbz4@health.missouri.edu) is in state (Closed,None_)  (Last ticket update: Insufficient Resources: ucsd-w8.fabric-testbed.net cannot serve the requested sliver)

Resource Type: VM Notices: Reservation d22568e3-cfa1-4088-9a19-d1bb74674376 (Slice cluster_gatk(9f998c48-6f4f-4af7-be33-337e94f7afb7) Graph Id:116fe360-7f45-4384-9a15-2aec138c4519 Owner:mjdbz4@health.missouri.edu) is in state (Closed,None_)  (Last ticket update: Insufficient Resources: ucsd-w6.fabric-testbed.net cannot serve the requested sliver)

Resource Type: VM Notices: Reservation ebb64ccb-476f-4373-8876-17af819df8c6 (Slice cluster_gatk(9f998c48-6f4f-4af7-be33-337e94f7afb7) Graph Id:116fe360-7f45-4384-9a15-2aec138c4519 Owner:mjdbz4@health.missouri.edu) is in state (Closed,None_)  (Last ticket update: Insufficient Resources: ucsd-w9.fabric-testbed.net cannot serve the requested sliver)

Thanks,
Komal

[Author]

Posts