[Komal Thareja]

Hello Nagmat,

Could you please try creating your slice again?

The error you are observing is because VM creation failed before other resources could be requested to be provisioned. In this scenario, orchestrator triggers a Close for the rest of the resources and results: TicketReviewPolicy: Closing reservation due to failure in slice.

VM creation failed on UCSD because of PCI attach/detach due to a leaked VM. I have resolved that. Could you please try your slice again?

Thanks,
Komal

[Komal Thareja]

@Manas – Is it possible for you to span your slice across multiple sites instead of single site and use other network services FabNet or L2STS or L2PTP?

In the meanwhile, I will discuss this within our team and share our approach forward.

[Komal Thareja]

@Manas – It looks like you are requesting: Cores=20, RAM=128GB, Disk=2000GB. Even though we may have an overall storage available to account for such a request, some of your VMs are causing the Worker to be exhausted. Specifically Disk requested cannot be served.

I was able to create this slice with Disk usage set to 500GB instead on SALT but your storage volume doesn’t exist there. Could you please try this slice with lower disk?

[Komal Thareja]

Duplicate: https://learn.fabric-testbed.net/forums/topic/unable-to-allocate-resources-after-the-updates-maintenance/

[Komal Thareja]

Hello,

I was debugging this and noticed that VMs requested by your slice are being mapped to capacities indicated below which seem to exhaust the worker where your request lands. The capacity mapping from requested to allocated seems strange.
I am unable to reproduce this. Could you please share your notebook?

Also, could you please share the output of command: pip3 list | grep fabric from your environment?
We expect the following versions to be present.

fabric                        3.0.0
fabric-credmgr-client         1.3.2
fabric-fim                    1.4.2
fabric-fss-utils              1.4.0
fabric-orchestrator-client    1.4.3
fabrictestbed                 1.4.3
fabrictestbed-extensions      1.4.0

Capacity Allocations for your slice: cluster_gatk(d8bfce5c-c721-4d41-a7fa-e8d658a40a43)

'capacities': '{ core: 2 , ram: 8 G, disk: 10 G}',                      ===> Requested by User
'capacity_allocations': '{ core: 16 , ram: 128 G, disk: 500 G}',        ===> Allocated by Orchestrator
'capacity_hints': '{ instance_type: fabric.c16.m128.d500}'

Thanks,
Komal

• This reply was modified 2 years, 7 months ago by Komal Thareja.

[Komal Thareja]

Could you please verify from the Portal, that project for which storage volumes were created has the Tag: Component.Storage available? Also, Please make sure you are using the same project ID in JH.

If your project does not have the tag: Component.Storage, Please raise a request to enable that for your project.

[Komal Thareja]

Looks like your project doesn’t have permissions for the Persistent Storage.
You would need to request Component.Storage permissions and persistent storage from the Portal by going to the Contact Us tab.

Thanks
Komal

[Komal Thareja]

Hi,

Could you please try to use the following notebook as indicated on jupyter-examples-rel1.4.1/start_here.ipynb ?

Persistent Storage: Connect to your project’s persisent storage volume.

Thanks,
Komal

[Komal Thareja]

Closing the topic

[Komal Thareja]

Maintenance is complete. UTAH and MICH sites are also available for use.

[Komal Thareja]

Hello,

Could you please run the configure notebook from ‘Jupiter-examples-rel1.4.1’ directory and then try the hello world example again?

From the screenshot it looks like environment was rolled back to Release 1.3.8.

Thanks,
Komal

[Komal Thareja]

Update is complete. All sites except MICH and UTAH are available for use. We will share another update when MICH and UTAH are available.

NOTE: Please install/upgrade to latest version of fabrictestbed-extensions if you are accessing FABRIC resources from outside the JupyterHub environment.

pip install fabrictestbed-extensions==1.4.0

[Komal Thareja]

Update is complete!

[Komal Thareja]

This looks like an issue with the Project ID setting in fabric_config/fabric_rc. Could you please re-run the notebook: Configure Environment and verify that Project ID being set is same as the Project Id in the portal?

Setup Environment
Configure Environment: Configure you Environment including creating the fabric_rc and ssh_config files.

• This reply was modified 2 years, 8 months ago by Komal Thareja.

[Komal Thareja]

Hello Nagmat,

Please Restart your Jupyter Container via File -> Hub Control Panel -> Stop My server, followed by Start My Server.
Please let us know if you still face this error.

Thanks,
Komal

[Author]

Posts