[Komal Thareja]

Maintenance is complete. Testbed has been opened for use.

Thanks,
Komal

[Komal Thareja]

Dear Experimenters,

The testbed is in a brief maintenance. We are running some performance test and will inform as soon as the testbed is back online.

Thanks,
Komal

[Komal Thareja]

I can confirm it’s the same issue on MASS and UCSD as well based on the logs. Also, I noticed the slice request was for 1000G disk but it gets mapped to a flavor with 2000G as we don’t have any 1000G flavor. None of the sites/workers mentioned have that much disk available and hence the provisioning fails. Unfortunately, the error message returned by openstack is not specific and not very helpful.

Thanks again for letting us know. We will work on fixing the mismatch and address this.

Thanks,
Komal

• This reply was modified 2 years, 4 months ago by Komal Thareja.
• This reply was modified 2 years, 4 months ago by Komal Thareja.

[Komal Thareja]

Hi Nagmat,

Thank you for sharing your observation! The requested slice asks for a VM with 2000GB disk on FIU rack (fiu-w4.fabric-testbed.net). However, the disk isn’t available and hence VM provisioning fails. We have a known issue that there is a mismatch in the disk availability as seen by software and the actual infrastructure. We are working to address that. But for now, I would request to create a slice with smaller disk.

Also, was this issue only observed on FIU rack? If not, Could you please share the other sites where you observed this error?

Thanks,
Komal

• This reply was modified 2 years, 4 months ago by Komal Thareja.

1 user thanked author for this post.

Nagmat Nazarov

[Komal Thareja]

Hello Fengping,

Thanks you for sharing your observation. I looked at your slice. The Network Service in your slice is in Closed state as NSO failed to allow 100 external IP addresses. Enclosing the snapshot of the error below.

Also, I confirmed with the Network Team and found that we have an upper limit of 90 public IPs on a FabNetV6Ext. Their recommendation is to create an additional slice if you need more than 90 public IPs. Hope this helps!

err=failed lease update- all units failed priming: Exception during modify for unit: 4a338812-470d-41fc-a510-145dfd775816 Playbook has failed tasks: NSO exists invalid params. path = /ncs:services/l3rt:l3rt{NET3-4a338812-470d-41fc-a510-145dfd775816}/external-access/permit-ipv6{100}

Thanks,
Komal

[Komal Thareja]

Thank you for sharing this and helping us make the testbed better. It was a bug. A fix has been deployed. Could you please delete the slice which has FABNetv6Ext and recreate it again?

Thanks,
Komal

[Komal Thareja]

Testbed is open for use. GATech, FIU and GPN sites are still in maintenance. We would update once GATech, FIU and GPN are open.

[Komal Thareja]

Maintenance is completed. Network AM has been updated.

[Komal Thareja]

Hi Praveen,

This error is unrelated to the number of user’s dead/closed slices. The slice requested by Manas, was requesting VMs on UCSD rack and specific workers: ucsd-w7.fabric-testbed.net, ucsd-w8.fabric-testbed.net, ucsd-w9.fabric-testbed.net.

UCSD rack has only 6 workers, so the slice could not be allocated and the error “Insufficient Resources” was returned.

Resource Type: VM Notices: Reservation 503f8da4-dbfa-49bf-8efc-12dc02253194 (Slice cluster_gatk(9f998c48-6f4f-4af7-be33-337e94f7afb7) Graph Id:116fe360-7f45-4384-9a15-2aec138c4519 Owner:mjdbz4@health.missouri.edu) is in state (Closed,None_)  (Last ticket update: Insufficient Resources: ucsd-w7.fabric-testbed.net cannot serve the requested sliver)

Resource Type: VM Notices: Reservation beebbecb-660a-4011-bae3-7457be9955f2 (Slice cluster_gatk(9f998c48-6f4f-4af7-be33-337e94f7afb7) Graph Id:116fe360-7f45-4384-9a15-2aec138c4519 Owner:mjdbz4@health.missouri.edu) is in state (Closed,None_)  (Last ticket update: Insufficient Resources: ucsd-w8.fabric-testbed.net cannot serve the requested sliver)

Resource Type: VM Notices: Reservation d22568e3-cfa1-4088-9a19-d1bb74674376 (Slice cluster_gatk(9f998c48-6f4f-4af7-be33-337e94f7afb7) Graph Id:116fe360-7f45-4384-9a15-2aec138c4519 Owner:mjdbz4@health.missouri.edu) is in state (Closed,None_)  (Last ticket update: Insufficient Resources: ucsd-w6.fabric-testbed.net cannot serve the requested sliver)

Resource Type: VM Notices: Reservation ebb64ccb-476f-4373-8876-17af819df8c6 (Slice cluster_gatk(9f998c48-6f4f-4af7-be33-337e94f7afb7) Graph Id:116fe360-7f45-4384-9a15-2aec138c4519 Owner:mjdbz4@health.missouri.edu) is in state (Closed,None_)  (Last ticket update: Insufficient Resources: ucsd-w9.fabric-testbed.net cannot serve the requested sliver)

Thanks,
Komal

[Komal Thareja]

The closed/dead slices do not have any resources associated with them. They get cleared periodically. Slices Dead/Closing older than 30 days are removed automatically.

I was just trying to suggest that when your slice closes, it could be helpful to see a failure reason on the portal to modify the request accordingly as in this case ucsd-w9.fabric-testbed.net cannot accomodate the VMs requested. So altering your slice request would be helpful.

Thanks,
Komal

[Komal Thareja]

Hi Manas,

Could you please look for Closed/Dead slices on the portal to check the failure reason?

I looked some of the VMs requested by your slice were in Failed state with “Insufficient resources error”. Since VMs could not be provisioned, network is closed as the VMs which are pre-requisite for it are in failed state.

Reservation ID: ebb64ccb-476f-4373-8876-17af819df8c6 Slice ID: 9f998c48-6f4f-4af7-be33-337e94f7afb7
Resource Type: VM Notices: Reservation ebb64ccb-476f-4373-8876-17af819df8c6 (Slice cluster_gatk(9f998c48-6f4f-4af7-be33-337e94f7afb7) Graph Id:116fe360-7f45-4384-9a15-2aec138c4519 Owner:mjdbz4@health.missouri.edu) is in state (Closed,None_)  (Last ticket update: Insufficient Resources: ucsd-w9.fabric-testbed.net cannot serve the requested sliver)

Thanks,
Komal

[Komal Thareja]

Maintenance is complete! Testbed is ready for use again!

[Komal Thareja]

We are still working on updating software. Apologies for the delay, we will notify once the maintenance is completed and testbed is available for use!

Thanks,
Komal

[Komal Thareja]

We have observed sometimes stale cookies in the browser result in this error. Generally opening in an incognito window clear this issue. We are looking at ways to clear stale cookies to avoid this to improve user experience.

We will keep you posted once a fix is deployed. Appreciate your feedback in helping us improve the testbed.

Thanks,
Komal

[Komal Thareja]

Hi Gregory,

Could you please try to download the tokens by opening the portal in an incognito window?

Thanks,
Komal

[Author]

Posts