[Komal Thareja]

Hi Manas,

Could you please look for Closed/Dead slices on the portal to check the failure reason?

I looked some of the VMs requested by your slice were in Failed state with “Insufficient resources error”. Since VMs could not be provisioned, network is closed as the VMs which are pre-requisite for it are in failed state.

Reservation ID: ebb64ccb-476f-4373-8876-17af819df8c6 Slice ID: 9f998c48-6f4f-4af7-be33-337e94f7afb7
Resource Type: VM Notices: Reservation ebb64ccb-476f-4373-8876-17af819df8c6 (Slice cluster_gatk(9f998c48-6f4f-4af7-be33-337e94f7afb7) Graph Id:116fe360-7f45-4384-9a15-2aec138c4519 Owner:mjdbz4@health.missouri.edu) is in state (Closed,None_)  (Last ticket update: Insufficient Resources: ucsd-w9.fabric-testbed.net cannot serve the requested sliver)

Thanks,
Komal

[Komal Thareja]

Maintenance is complete! Testbed is ready for use again!

[Komal Thareja]

We are still working on updating software. Apologies for the delay, we will notify once the maintenance is completed and testbed is available for use!

Thanks,
Komal

[Komal Thareja]

We have observed sometimes stale cookies in the browser result in this error. Generally opening in an incognito window clear this issue. We are looking at ways to clear stale cookies to avoid this to improve user experience.

We will keep you posted once a fix is deployed. Appreciate your feedback in helping us improve the testbed.

Thanks,
Komal

[Komal Thareja]

Hi Gregory,

Could you please try to download the tokens by opening the portal in an incognito window?

Thanks,
Komal

[Komal Thareja]

Maintenance is complete.

[Komal Thareja]

Maintenance is complete. Testbed is back online and ready to use.

[Komal Thareja]

Hi Nagmat,

We are currently under maintenance and will send out an update once the issue is resolved.

Thanks,
Komal

Maintenance on FABRIC-Testbed – 02/09/2022 (11:00am-1:00pm EST)

[Komal Thareja]

Hello Nagmat,

Could you please try creating your slice again?

The error you are observing is because VM creation failed before other resources could be requested to be provisioned. In this scenario, orchestrator triggers a Close for the rest of the resources and results: TicketReviewPolicy: Closing reservation due to failure in slice.

VM creation failed on UCSD because of PCI attach/detach due to a leaked VM. I have resolved that. Could you please try your slice again?

Thanks,
Komal

[Komal Thareja]

@Manas – Is it possible for you to span your slice across multiple sites instead of single site and use other network services FabNet or L2STS or L2PTP?

In the meanwhile, I will discuss this within our team and share our approach forward.

[Komal Thareja]

@Manas – It looks like you are requesting: Cores=20, RAM=128GB, Disk=2000GB. Even though we may have an overall storage available to account for such a request, some of your VMs are causing the Worker to be exhausted. Specifically Disk requested cannot be served.

I was able to create this slice with Disk usage set to 500GB instead on SALT but your storage volume doesn’t exist there. Could you please try this slice with lower disk?

[Komal Thareja]

Duplicate: https://learn.fabric-testbed.net/forums/topic/unable-to-allocate-resources-after-the-updates-maintenance/

[Komal Thareja]

Hello,

I was debugging this and noticed that VMs requested by your slice are being mapped to capacities indicated below which seem to exhaust the worker where your request lands. The capacity mapping from requested to allocated seems strange.
I am unable to reproduce this. Could you please share your notebook?

Also, could you please share the output of command: pip3 list | grep fabric from your environment?
We expect the following versions to be present.

fabric                        3.0.0
fabric-credmgr-client         1.3.2
fabric-fim                    1.4.2
fabric-fss-utils              1.4.0
fabric-orchestrator-client    1.4.3
fabrictestbed                 1.4.3
fabrictestbed-extensions      1.4.0

Capacity Allocations for your slice: cluster_gatk(d8bfce5c-c721-4d41-a7fa-e8d658a40a43)

'capacities': '{ core: 2 , ram: 8 G, disk: 10 G}',                      ===> Requested by User
'capacity_allocations': '{ core: 16 , ram: 128 G, disk: 500 G}',        ===> Allocated by Orchestrator
'capacity_hints': '{ instance_type: fabric.c16.m128.d500}'

Thanks,
Komal

• This reply was modified 2 years, 2 months ago by Komal Thareja.

[Komal Thareja]

Could you please verify from the Portal, that project for which storage volumes were created has the Tag: Component.Storage available? Also, Please make sure you are using the same project ID in JH.

If your project does not have the tag: Component.Storage, Please raise a request to enable that for your project.

[Komal Thareja]

Looks like your project doesn’t have permissions for the Persistent Storage.
You would need to request Component.Storage permissions and persistent storage from the Portal by going to the Contact Us tab.

Thanks
Komal

[Author]

Posts